Platform: aarch64 Linux
Compiler: GNU compiler suite (gcc, ld)

I'm overhauling the build system for the firmware of a Linux embedded
device.

There are about two dozen shared libraries to build, and about a dozen executables to build.

Before I started working on this build system, there were many shared libraries being built with unresolved symbols (undefined functions) because they were missing linker dependencies e.g. "-lrt -lstdc++ -lpthread". These shared libraries would have failed to build if they'd been built with the linker option "-Wl,--no-undefined".

I went through all the Makefiles for all these shared libraries, and I added in the linker options that were needed for each library. This meant that I could then build the entire firmware (i.e. all 25 libraries and all 13 executables) with the following two options:
-Wl,--no-undefined
-Wl,--no-allow-shlib-undefined
Or alternatively:
-Wl,--unresolved-symbols=report-all

So my mission to revamp the build system went find except for one library called "libencoder.so" which was missing a function called:
void ReportError(char const *);

The implementation of this function wasn't anywhere to be found in the source code for any of the 25 shared libraries, however I found three separate implementations of this function inside three separate executable programs. It turns out that the library "libencoder.so" deliberately leaves the function "ReportError" undefined so that when an executable program links with 'libencoder.so', 'libencoder.so' will reach into the executable file and invoke the executable file's implementation of "ReportError".

But since I had now added "-Wl,--no-undefined" to the list of linker options, the library "libencoder.so" now failed to build because of an undefined reference to "ReportError".

I have tried three different solutions to get "libencoder" to build:

Idea 1) Link with the option "-Wl,-U,ReportError" to tell the linker to allow "ReportError" to be undefined. I saw this idea on a web forum for Apple macOS, and unfortunately it doesn't work on Linux.

Idea 2) Link without the option "-Wl,--no-undefined". Of course this idea works, but unfortunately it allows any function or any variable to be undefined, when all I really want is for the function "ReportError" to be allowed to be undefined. So this option does work, but it's a little too extreme.

Idea 3) In the code for "libencoder", mark the declaration of the function "ReportError" with the attribute, "__attribute((weak))__", and don't provide an implementation for it in "libencoder.so". This means that the shared object library file "libencoder.so" will contain a weak undefined symbol for "ReportError", and if you use "nm" on it as follows:
nm -D libencoder.so
then you'll see a small "w" beside "ReportError":
w ReportError

(If it were a capital 'W' then it would be a weak symbol that is defined)

Method 3 is definitely my preferred choice, however there is one problem with it: If I build "libencoder.so" with a weak undefined symbol for "ReportError", it is then unfortunately possible to successfully link an executable program that does not contain an implementation of "ReportError". In this circumstance, the executable program segfaults when it tries to call "ReportError".

So even though Method 3 is my preferred choice right now, I would only choose it if there was a way to tell the linker to refuse to build an executable file which has an undefined weak symbol. So if my routine for building the library and the exectuable files is as follows:

Line 1: gcc -o libencoder.so libencoder.c -fPIC -shared -Wl,--unresolved-symbols=report-all
Line 2: gcc -o converter converter.c -Wl,--unresolved-symbols=report-all

Then I want Line 1 to succeed in creating a shared object library file that has an undefined weak symbol, however I want Line 2 to fail if "converter.c" doesn't contain an implementation of the function "ReportError".

Unfortunately I haven't been able to find any linker option that will force Line 2 to fail to build.

Anyone got any ideas?

On Monday, October 4, 2021 at 2:50:28 PM UTC+1, Frederick Gotham wrote:

> Line 1: gcc -o libencoder.so libencoder.c -fPIC -shared -Wl,--unresolved-symbols=report-all
> Line 2: gcc -o converter converter.c -Wl,--unresolved-symbols=report-all

Line 2 should have "-lencoder" tagged onto the end

On 2021-10-04, Frederick Gotham <cauldwell.thomas@gmail.com> wrote:
> Platform: aarch64 Linux Compiler: GNU compiler suite (gcc, ld)
>
> I'm overhauling the build system for the firmware of a Linux embedded device.
>
>
> Anyone got any ideas?
Your line is too long, learn to format text first, this is usenet...

--

7-77-777
Evil Sinner!
to weak you should be meek, and you should brainfuck stronger
https://github.com/rofl0r/chaos-pp

In article <334a9621-65d5-44a1-8442-8f1a66fc5d3bn@googlegroups.com>,
Frederick Gotham <cauldwell.thomas@gmail.com> wrote:
....
>Line 1: gcc -o libencoder.so libencoder.c -fPIC -shared
>-Wl,--unresolved-symbols=report-all
>Line 2: gcc -o converter converter.c -Wl,--unresolved-symbols=report-all
>
>Then I want Line 1 to succeed in creating a shared object library
>file that has an undefined weak symbol, however I want Line 2 to fail
>if "converter.c" doesn't contain an implementation of the function
>"ReportError".
>
>Unfortunately I haven't been able to find any linker option that will
>force Line 2 to fail to build.

There probably isn't one. So, we gotta come up alternatives...

>Anyone got any ideas?

Well, assuming this is in a Makefile (or whatever modern day equivalent of
make you are using), such that any error will cause the rule to stop, there
are a couple of ways:

1) (Simple - before your Line2) grep -q ReportError converter.c

2) (More sophisticated - after your Line2) nm converter | grep -q ReportError

--
12% of Americans think that Joan of Arc was Noah's wife.

On Mon, 04 Oct 2021 06:50:18 -0700, Frederick Gotham wrote:

> Platform: aarch64 Linux
> Compiler: GNU compiler suite (gcc, ld)
>
> I'm overhauling the build system for the firmware of a Linux embedded
> device.
[snip]
> I went through all the Makefiles for all these shared libraries, and I
> added in the linker options that were needed for each library. This
> meant that I could then build the entire firmware (i.e. all 25
> libraries and all 13 executables) with the following two options:
> -Wl,--no-undefined
> -Wl,--no-allow-shlib-undefined
> Or alternatively:
> -Wl,--unresolved-symbols=report-all
>
> So my mission to revamp the build system went find except for one
library called "libencoder.so" which was missing a function called:
> void ReportError(char const *);
>
> The implementation of this function wasn't anywhere to be found in the
> source code for any of the 25 shared libraries, however I found three
> separate implementations of this function inside three separate
> executable programs. It turns out that the library "libencoder.so"
> deliberately leaves the function "ReportError" undefined so that when
> an executable program links with 'libencoder.so', 'libencoder.so' will
> reach into the executable file and invoke the executable file's
> implementation of "ReportError".
>
> But since I had now added "-Wl,--no-undefined" to the list of linker
options, the library "libencoder.so" now failed to build because of an
undefined reference to "ReportError".
>
> I have tried three different solutions to get "libencoder" to build:
[snip]
> Anyone got any ideas?

Provide a dummy implementation of the ReportError() function in it's
own library, and put that library /at the end/ of the link chain.

Should a program provide it's own ReportError(), /that/ ReportError()
will link first, and satisfy the external from libencoder.so.

However, should a program /not/ provide ReportError(), then your dummy
ReportError() will link and satisfy the external reference from
libencoder.so.

--
Lew Pitcher
"In Skills, We Trust"

In article <sjfnq9$3qg$1@dont-email.me>,
Lew Pitcher <lew.pitcher@digitalfreehold.ca> wrote:
....
>Provide a dummy implementation of the ReportError() function in it's
>own library, and put that library /at the end/ of the link chain.
>
>Should a program provide it's own ReportError(), /that/ ReportError()
>will link first, and satisfy the external from libencoder.so.
>
>However, should a program /not/ provide ReportError(), then your dummy
>ReportError() will link and satisfy the external reference from
>libencoder.so.

That's not bad, but it solves a slightly different problem.

It solves: Make sure that the resulting executable will run (*). I.e.,
that we don't get a "It compiled OK, but crashed at runtime" situation.

I think OP wanted: If there is a problem, I want the compile/link to fail.
I.e., I want to know about it at "make" time.

(*) It will run (i.e., will not crash), but it may not do what OP actually
wants, since the dummy function probably isn't what he wants to have be
running. In a way, this is the worst kind of error - "It compiles and
runs, but does not do what I want".

--
In politics and in life, ignorance is not a virtue.
-- Barack Obama --

Here's the thing...

Libraries may include external references. This does not impede their
use as libraries; those external references must be satisfied either
at link time or at runtime. If, as the OP has obliquely acknowledged
is his situation, those libraries are "shared objects", external
references that are not resolved at link time will either be resolved
at runtime, or the execution will fail.

The OP has made an "executive decision" that all libraries will either
be self-contained, or have references /only/ to other libraries. He
has enforced this decision by implementing the
-Wl,--no-undefined -Wl,--no-allow-shlib-undefined
parameters when compiling each library, and then including each
referenced library in the creation of the shared object module.

Clearly, this did not work for the one module that depends on a source
other than another library to satisfy it's external references. That
library /requires/ a user-supplied function, not one coming from
another library.

Now, the cool thing about shared object libraries is that you /dont/
have to resolve their external references at compile time. You can
leave the resolution of those external references to link time or
even execution time. You do not have to /link/ a shared object in
order to create it.

So, now the executive decision has backfired, and a single shared
object cannot be built. ISTM that the OP has a few of options:
1) revoke the executive decision for this library, so that it
can be built, or
2) rewrite (or cause to be rewritten) the source for this library
so that it no longer depends on linkage to invoke the errant
function (say, have it use dlopen(), et al, instead of simply
defining the function as
extern void ReportError(char const *); ), or
3) provide a suitable dummy ReportError() function, in it's own
library, to "link" against, so as to resolve the external
symbol.

It is regrettable that so much of the knowledge pertaining to the
process of programming has been lost. This sort of thing /used to
be/ simple knowledge.

--
Lew Pitcher
"In Skills, We Trust"

On Monday, October 4, 2021 at 9:37:53 PM UTC+1, Kenny McCormack wrote:

> I think OP wanted: If there is a problem, I want the compile/link to fail.
> I.e., I want to know about it at "make" time.

Yeah this is what I want -- I like build errors very much and I dislike runtime errors. I hate having a viable firmware tarball that malfunctions when you upload it to the device -- I'd prefer not to have a tarball at all.

Kenny I like your idea about using "nm" on the object files when building the executable file 'converter'. The Makefile for 'converter' would be something like this:

MODULES = cat.o dog.o fish.o monkey.o

%.o: %.c
$(CC) $(CFLAGS) -c $(input) -o $(output)

converter: $(MODULES)
nm "$(MODULES)" 2> /dev/null | grep ReportError | grep -q -i " t "
if [ "$?" -ne 0 ]; then
echo "Refusing to build executable file 'converter' because definition of function 'ReportError' is missing"
else
$(CC) $(inputs) -o $(output)
fi

Can anyone make it better?

On Monday, October 4, 2021 at 10:11:41 PM UTC+1, Lew Pitcher wrote:

> The OP has made an "executive decision" that all libraries will either
> be self-contained, or have references /only/ to other libraries. He
> has enforced this decision by implementing the
> -Wl,--no-undefined -Wl,--no-allow-shlib-undefined
> parameters when compiling each library, and then including each
> referenced library in the creation of the shared object module.

Yes this is what I'm going for

> Now, the cool thing about shared object libraries is that you /dont/
> have to resolve their external references at compile time.

Lots of people don't think this is cool. In my job I submitted a code review for a library I wrote in which I deliberately left a function undefined so that it could later be defined by an executable that links with the library. It didn't pass the code review.

Last I checked, MS-Windows doesn't allow this kind of Wild West linking. I was actually very surprised when I learned that Linux allows this, as well as all that "LD_PRELOAD" stuff that allows you to manipulate programs and libraries at runtime -- which I've used greatly to my advantage as I've written a Makefile for patching pre-built shared object binaries.

> So, now the executive decision has backfired, and a single shared
> object cannot be built.

I'm sticking to my decision to use "--unresolved-symbols=report-all" as a default build option.

> ISTM that the OP has a few of options:
> 1) revoke the executive decision for this library, so that it
> can be built

Yes I could do this by using "filter-out" in the individual Makefile. (That's actually how I have it working right now but I want a better, more strict solution).

> 2) rewrite (or cause to be rewritten) the source for this library
> so that it no longer depends on linkage to invoke the errant
> function (say, have it use dlopen(), et al, instead of simply
> defining the function as
> extern void ReportError(char const *); ), or
> 3) provide a suitable dummy ReportError() function, in it's own
> library, to "link" against, so as to resolve the external
> symbol.

I won't go with either of these. I like Kenny's idea of using 'nm' to check the object files.

On Monday, October 4, 2021 at 10:19:43 PM UTC+1, Frederick Gotham wrote:

> converter: $(MODULES)
> nm "$(MODULES)" 2> /dev/null | grep ReportError | grep -q -i " t "
> if [ "$?" -ne 0 ]; then
> echo "Refusing to build executable file 'converter' because definition of function 'ReportError' is missing"
> else
> $(CC) $(inputs) -o $(output)
> fi

Here's what I have now in the Makefile:

converter: $(MODULES)
@nm $(MODULES) 2> /dev/null | grep ReportError | grep -q -i " t " \
|| (echo "========== Function definition missing: 'ReportError' (required by 'libencoder.so' -- see shared/libencoder/Makefile for explanation)"; exit 1)

Lew Pitcher <lew.pitcher@digitalfreehold.ca> writes:
>Here's the thing...
>
>Libraries may include external references. This does not impede their
>use as libraries; those external references must be satisfied either
>at link time or at runtime. If, as the OP has obliquely acknowledged
>is his situation, those libraries are "shared objects", external
>references that are not resolved at link time will either be resolved
>at runtime, or the execution will fail.

Minor nit - they won't fail if the external reference is not actually
invoked, unless lazy binding is disabled.

On 2021-10-04, Frederick Gotham <cauldwell.thomas@gmail.com> wrote:
> Platform: aarch64 Linux
> Compiler: GNU compiler suite (gcc, ld)
>
> I'm overhauling the build system for the firmware of a Linux embedded
> device.
>
> There are about two dozen shared libraries to build, and about a dozen executables to build.
>
> Before I started working on this build system, there were many shared
> libraries being built with unresolved symbols (undefined functions)
> because they were missing linker dependencies e.g. "-lrt -lstdc++
> -lpthread". These shared libraries would have failed to build if
> they'd been built with the linker option "-Wl,--no-undefined".

So, like, nobody runs the code to catch the fact that it can't even
execute due to missing dynamic symbols???

.... but if we catch that at build time and fix it, ship it!

Kaz Kylheku <480-992-1380@kylheku.com> writes:
>On 2021-10-04, Frederick Gotham <cauldwell.thomas@gmail.com> wrote:
>> Platform: aarch64 Linux
>> Compiler: GNU compiler suite (gcc, ld)
>>
>> I'm overhauling the build system for the firmware of a Linux embedded
>> device.
>>
>> There are about two dozen shared libraries to build, and about a dozen executables to build.
>>
>> Before I started working on this build system, there were many shared
>> libraries being built with unresolved symbols (undefined functions)
>> because they were missing linker dependencies e.g. "-lrt -lstdc++
>> -lpthread". These shared libraries would have failed to build if
>> they'd been built with the linker option "-Wl,--no-undefined".
>
>So, like, nobody runs the code to catch the fact that it can't even
>execute due to missing dynamic symbols???

Who said anything about execute? Frederick couldn't _build_ the
library using the aforementioned build option.

Shared objects with undefined symbols work just fine unless the
application attempts to use an API in the library that requires
the external symbol, or the run-time linker BIND NOW capability
was enabled (e.g. LD_BIND_NOW is in the environment).

LD_BIND_NOW (since glibc 2.1.1)
If set to a nonempty string, causes the dynamic linker to
resolve all symbols at program startup instead of
deferring function call resolution to the point when they
are first referenced. This is useful when using a
debugger.

Another useful option:

LD_DYNAMIC_WEAK (since glibc 2.1.91)
By default, when searching shared libraries to resolve a
symbol reference, the dynamic linker will resolve to the
first definition it finds.

Old glibc versions (before 2.2), provided a different
behavior: if the linker found a symbol that was weak, it
would remember that symbol and keep searching in the
remaining shared libraries. If it subsequently found a
strong definition of the same symbol, then it would
instead use that definition. (If no further symbol was
found, then the dynamic linker would use the weak symbol
that it initially found.)

The old glibc behavior was nonstandard. (Standard
practice is that the distinction between weak and strong
symbols should have effect only at static link time.) In
glibc 2.2, the dynamic linker was modified to provide the
current behavior (which was the behavior that was provided
by most other implementations at that time).

Defining the LD_DYNAMIC_WEAK environment variable (with
any value) provides the old (nonstandard) glibc behavior,
whereby a weak symbol in one shared library may be
overridden by a strong symbol subsequently discovered in
another shared library. (Note that even when this
variable is set, a strong symbol in a shared library will
not override a weak definition of the same symbol in the
main program.)

Since glibc 2.3.4, LD_DYNAMIC_WEAK is ignored in secure-
execution mode.