On 31/07/2021 19:33, Manfred wrote:
> On 7/31/2021 3:56 PM, Bart wrote:
>> On 31/07/2021 12:42, David Brown wrote:
>>> On 31/07/2021 02:51, Bart wrote:
>>>> On 30/07/2021 22:16, David Brown wrote:
>>>>> On 30/07/2021 21:03, Bart wrote:
>>>>
>>>
>>>>>> How big would a program be if it incorporated half the Windows API as
>>>>>> statically linked code?
>>>>>>
>>>>>
>>>>> For someone who claims to have developed their own languages,
>>>>> compilers,
>>>>> linkers and other tools for decades, you are remarkably ignorant of
>>>>> how
>>>>> toolchains work.
>>>>
>>>> Why, what's wrong with my question? It is perfectly reasonable.
>>>>
>>>> I might consider bundling RAYLIB with an application for example, by
>>>> statically linking (it's some 3MB I think). But not libraries which are
>>>> guaranteed to come with the OS.
>>>>
>>>> (Some exceptions such as OpenGL where the OS one may lag behind.)
>>>>
>>>>> Or you are trolling, as some people have suggested.
>>>>> Perhaps you have just written in haste, without thinking - everyone
>>>>> makes mistakes and occasionally writes something stupid.
>>>>
>>>> OK, now I'm curious; what's the thing that is stupid? Having an opinion
>>>> different from yours?
>>>>
>>>
>>> When you use statically linked libraries, you only link in the functions
>>> you actually /use/.  So it doesn't matter if the static library supports
>>> a hundred thousand WinAPI functions and weighs in at 100 MB - if you use
>>> a hundred functions, the library will give you an extra 100 KB in your
>>> binary.  Using a static library for the WinAPI or the C standard library
>>> is only going to give you a big binary if you use a very large number of
>>> functions from the library - in which case your binary is already
>>> very big.
>>
>> I don't think this is right.
> No, it is right.
>
>  My brief experiments seem to show:
>>
>> * If you link in biglib.o, it will include everything from biglib,
>> whether or not used from outside
>>
>> * If you create biglib.a, and link in biglib.a, it will still include
>> everything in biglib, provided at least one export is used elsewhere
>>
>> But my experiments only covered a .a file made from a single .o file.
>
> That's the difference. You should test it with a final executable (and a
> decent compiler). (btw I assume you didn't mean .a as a linux library,
> so what did you mean with .a ?)
>
>>
>> It could be that it works on a granularity of individual object files.
>> But even then, if A and B each have 1000 functions, and you use one
>> function in A which calls one in B, then 2000 functions will get
>> included, not even only 1000.
>>
>> This is some way from your suggestion that the granularity is at the
>> level of individual files, and presumably individual variables.
>
> That is how static libraries work with decent compilers, the final
> executable only gets individual /functions/ (not files, but obviously
> all of the function dependencies) and individual variables. Note, stress
> is on /static/ libraries (.lib on Windows and .a on Linux, so no .dll or
> .so statically linked, in case this were a thing)
>
Traditionally - since before C existed - the granularity level for
static libraries has been the file level. If you look at libraries
where it is normal to except applications to want to use only a few
functions out of many, such as the standard C library, the source code
is typically divided into one file per function. You might have several
related functions in a single file (such as sin and cos, or all the
printf family), for slightly improved efficiency or speed of linking.

This is how static libraries have worked with good and bad compilers,
and assemblers, and compilers of different languages, since the concept
of separate compiler, assembler, librarian and linker programs was invented.

Good tools, such as gcc and clang, have features such as flags to put
each function in a separate section, allowing more efficient linking of
static libraries (and object code files) without having to split the
source up into so many small files. This is not unique to C - I first
saw it as "smart linking" in Borland Pascal for DOS. For gcc, you do
need to use a few compiler flags, which is difficult for people who have
flag allergies and curl into a fetal position at the mention of "build
systems" or "makefiles".

In more modern times - the last decade or so - link-time optimisation
has moved into the mainstream for good general-purpose compilers. That
not only cuts out code from static libraries that is not used by the
application, but can inline and otherwise optimise code across library
boundaries. (In the embedded world, such "compile and link everything
at once" toolchains have been used since the nineties at least.)

On 01/08/2021 00:54, Richard Damon wrote:
> On 7/31/21 3:10 PM, Bart wrote:
>> On 31/07/2021 20:32, Manfred wrote:
>>> On 7/31/2021 8:17 PM, Bart wrote:
>>
>>>> So it does seem that it uses granularity of a module at best. It
>>>> seems odd though that the linker doesn't by default, exclude that
>>>> symbol info which makes a very significant difference, but doesn't,
>>>> assuming it was capable, eliminate unused functions.
>>>
>>> It turns out it's not that trivial, after my earlier reply I decided
>>> to double check it, and in fact I had to ask google about it:
>>> https://stackoverflow.com/questions/6687630/how-to-remove-unused-c-c-symbols-with-gcc-and-ld
>>>
>>>
>>> In short:
>>> compile with -fdata-sections -ffunction-sections
>>> link with: --gc-sections (i.e. invoke gcc with -Wl,--gc-sections)
>>>
>>> I verified that these options do get function level granularity.
>>
>> To clarify, these options need to be applied to the library in question.
>>
>> If it's a third party library, then presumably you have to depend on the
>> people creating the .a or .lib file to use suitable options.
>>
>> (When I tried it on a small library, it more than doubled the size of
>> the size of the file.)
>>
>
> Yes, to generate the information to allow selective linking adds a lot
> of meta-data to the object files.
>

It adds section names - nothing more. Instead of the code for "foo"
being in the ".text", it is in a ".text.foo" section. You have more
sections in the object files (and therefore library, if you combine the
object files in a static library), but that's all. Object files already
have vast numbers of sections for different types of text, debug data,
initialisation data, exception information (for C++), etc.

> That is one reason it normally isn't the default for the compiler, as it
> assumes that most files being compiled don't have a lot of 'unused'
> code, so there is no need to generate all the extra meta-data to allow
> partial linking of the files.
>

That, and backwards compatibility. Running your tools without flags
generally gives the same basic unoptimised and limited functionality as
it always has, so that everything works the same - most features above
the absolute ground level need flags.

On 04/08/2021 12:32, David Brown wrote:
> On 31/07/2021 19:33, Manfred wrote:
>> On 7/31/2021 3:56 PM, Bart wrote:
>>> On 31/07/2021 12:42, David Brown wrote:
>>>> On 31/07/2021 02:51, Bart wrote:
>>>>> On 30/07/2021 22:16, David Brown wrote:
>>>>>> On 30/07/2021 21:03, Bart wrote:
>>>>>
>>>>
>>>>>>> How big would a program be if it incorporated half the Windows API as
>>>>>>> statically linked code?
>>>>>>>
>>>>>>
>>>>>> For someone who claims to have developed their own languages,
>>>>>> compilers,
>>>>>> linkers and other tools for decades, you are remarkably ignorant of
>>>>>> how
>>>>>> toolchains work.
>>>>>
>>>>> Why, what's wrong with my question? It is perfectly reasonable.
>>>>>
>>>>> I might consider bundling RAYLIB with an application for example, by
>>>>> statically linking (it's some 3MB I think). But not libraries which are
>>>>> guaranteed to come with the OS.
>>>>>
>>>>> (Some exceptions such as OpenGL where the OS one may lag behind.)
>>>>>
>>>>>> Or you are trolling, as some people have suggested.
>>>>>> Perhaps you have just written in haste, without thinking - everyone
>>>>>> makes mistakes and occasionally writes something stupid.
>>>>>
>>>>> OK, now I'm curious; what's the thing that is stupid? Having an opinion
>>>>> different from yours?
>>>>>
>>>>
>>>> When you use statically linked libraries, you only link in the functions
>>>> you actually /use/.  So it doesn't matter if the static library supports
>>>> a hundred thousand WinAPI functions and weighs in at 100 MB - if you use
>>>> a hundred functions, the library will give you an extra 100 KB in your
>>>> binary.  Using a static library for the WinAPI or the C standard library
>>>> is only going to give you a big binary if you use a very large number of
>>>> functions from the library - in which case your binary is already
>>>> very big.
>>>
>>> I don't think this is right.
>> No, it is right.
>>
>>  My brief experiments seem to show:
>>>
>>> * If you link in biglib.o, it will include everything from biglib,
>>> whether or not used from outside
>>>
>>> * If you create biglib.a, and link in biglib.a, it will still include
>>> everything in biglib, provided at least one export is used elsewhere
>>>
>>> But my experiments only covered a .a file made from a single .o file.
>>
>> That's the difference. You should test it with a final executable (and a
>> decent compiler). (btw I assume you didn't mean .a as a linux library,
>> so what did you mean with .a ?)
>>
>>>
>>> It could be that it works on a granularity of individual object files.
>>> But even then, if A and B each have 1000 functions, and you use one
>>> function in A which calls one in B, then 2000 functions will get
>>> included, not even only 1000.
>>>
>>> This is some way from your suggestion that the granularity is at the
>>> level of individual files, and presumably individual variables.
>>
>> That is how static libraries work with decent compilers, the final
>> executable only gets individual /functions/ (not files, but obviously
>> all of the function dependencies) and individual variables. Note, stress
>> is on /static/ libraries (.lib on Windows and .a on Linux, so no .dll or
>> .so statically linked, in case this were a thing)
>>
> Traditionally - since before C existed - the granularity level for
> static libraries has been the file level. If you look at libraries
> where it is normal to except applications to want to use only a few
> functions out of many, such as the standard C library, the source code
> is typically divided into one file per function. You might have several
> related functions in a single file (such as sin and cos, or all the
> printf family), for slightly improved efficiency or speed of linking.
>
> This is how static libraries have worked with good and bad compilers,
> and assemblers, and compilers of different languages, since the concept
> of separate compiler, assembler, librarian and linker programs was invented.
>
> Good tools, such as gcc and clang, have features such as flags to put
> each function in a separate section, allowing more efficient linking of
> static libraries (and object code files) without having to split the
> source up into so many small files. This is not unique to C - I first
> saw it as "smart linking" in Borland Pascal for DOS. For gcc, you do
> need to use a few compiler flags, which is difficult for people who have
> flag allergies and curl into a fetal position at the mention of "build
> systems" or "makefiles".
>
> In more modern times - the last decade or so - link-time optimisation
> has moved into the mainstream for good general-purpose compilers. That
> not only cuts out code from static libraries that is not used by the
> application, but can inline and otherwise optimise code across library
> boundaries. (In the embedded world, such "compile and link everything
> at once" toolchains have been used since the nineties at least.)
>

On some environments such as Windows, using function-at-a-time linking
instead of DLLs is pointless.

In the case of common libraries such as the C runtime, the chances are
that other applications are already using them and so they are already
in memory.

In the case of the file called 'gcc.exe' of my tdm/gcc installation, it
was presumably built with all those flags you mentioned, yet it STILL
imports msvcrt.dll. (And if not, then why not?)

If I use that gcc to build an app, and specify -lucrt to make use of
libucrt.a (and whether or not I use -ffunction-sections for my app), the
resulting executable STILL imports msvcrt.dll, but now ALSO imports:

api-ms-win-crt-environment-l1-1-0.dll
api-ms-win-crt-runtime-l1-1-0.dll

for good measure.

> This is not unique to C - I first
> saw it as "smart linking" in Borland Pascal for DOS.

For all of the eighties, I wrote all my own libraries, which didn't have
many unnecessary functions. My language's runtime was 4-8KB, which
included maths functions /and/ floating point emulation.

The only external software used was in the OS via INT calls or whatever
the mechanism was.

There were some optional modules, mainly a choice of drivers (for video,
printer etc), but once chosen, you needed the whole module. So there was
no need for traditional linking, just a means the assemble and fixup the
relevant set of modules.

The point is that typical language implementations (apart from your
Borland), had linker programs that were hopelessly inefficient,
presumably even without that smart linking.

I think I'd assumed it was because it was linking a function at a time,
now you are suggesting that it was module at a time; so what on earth
were they up to?

> In more modern times - the last decade or so - link-time optimisation
> has moved into the mainstream for good general-purpose compilers. That
> not only cuts out code from static libraries that is not used by the

Some of us have gotten even more modern and eliminated linking
completely from the task of building a set of sources into one .exe or
or .dll file.

Fixing up references to functions imported from external .dll files is
trivial (you add a reference to an import table).

For libraries that are available as source code /in the right language/
then, using whole-program compilation, there are easy ways to eliminate
unused functions, if it is important (for my stuff, it isn't, as my
programs are already tiny).

The method requires means to specify boundaries around what are
considered to be a self-contained libraries, otherwise the analysis
needed is harder.

On 04/08/2021 15:24, Bart wrote:
> On 04/08/2021 12:32, David Brown wrote:
>> On 31/07/2021 19:33, Manfred wrote:
>>> On 7/31/2021 3:56 PM, Bart wrote:
>>>> On 31/07/2021 12:42, David Brown wrote:
>>>>> On 31/07/2021 02:51, Bart wrote:
>>>>>> On 30/07/2021 22:16, David Brown wrote:
>>>>>>> On 30/07/2021 21:03, Bart wrote:
>>>>>>
>>>>>
>>>>>>>> How big would a program be if it incorporated half the Windows
>>>>>>>> API as
>>>>>>>> statically linked code?
>>>>>>>>
>>>>>>>
>>>>>>> For someone who claims to have developed their own languages,
>>>>>>> compilers,
>>>>>>> linkers and other tools for decades, you are remarkably ignorant of
>>>>>>> how
>>>>>>> toolchains work.
>>>>>>
>>>>>> Why, what's wrong with my question? It is perfectly reasonable.
>>>>>>
>>>>>> I might consider bundling RAYLIB with an application for example, by
>>>>>> statically linking (it's some 3MB I think). But not libraries
>>>>>> which are
>>>>>> guaranteed to come with the OS.
>>>>>>
>>>>>> (Some exceptions such as OpenGL where the OS one may lag behind.)
>>>>>>
>>>>>>> Or you are trolling, as some people have suggested.
>>>>>>> Perhaps you have just written in haste, without thinking - everyone
>>>>>>> makes mistakes and occasionally writes something stupid.
>>>>>>
>>>>>> OK, now I'm curious; what's the thing that is stupid? Having an
>>>>>> opinion
>>>>>> different from yours?
>>>>>>
>>>>>
>>>>> When you use statically linked libraries, you only link in the
>>>>> functions
>>>>> you actually /use/.  So it doesn't matter if the static library
>>>>> supports
>>>>> a hundred thousand WinAPI functions and weighs in at 100 MB - if
>>>>> you use
>>>>> a hundred functions, the library will give you an extra 100 KB in your
>>>>> binary.  Using a static library for the WinAPI or the C standard
>>>>> library
>>>>> is only going to give you a big binary if you use a very large
>>>>> number of
>>>>> functions from the library - in which case your binary is already
>>>>> very big.
>>>>
>>>> I don't think this is right.
>>> No, it is right.
>>>
>>>   My brief experiments seem to show:
>>>>
>>>> * If you link in biglib.o, it will include everything from biglib,
>>>> whether or not used from outside
>>>>
>>>> * If you create biglib.a, and link in biglib.a, it will still include
>>>> everything in biglib, provided at least one export is used elsewhere
>>>>
>>>> But my experiments only covered a .a file made from a single .o file.
>>>
>>> That's the difference. You should test it with a final executable (and a
>>> decent compiler). (btw I assume you didn't mean .a as a linux library,
>>> so what did you mean with .a ?)
>>>
>>>>
>>>> It could be that it works on a granularity of individual object files.
>>>> But even then, if A and B each have 1000 functions, and you use one
>>>> function in A which calls one in B, then 2000 functions will get
>>>> included, not even only 1000.
>>>>
>>>> This is some way from your suggestion that the granularity is at the
>>>> level of individual files, and presumably individual variables.
>>>
>>> That is how static libraries work with decent compilers, the final
>>> executable only gets individual /functions/ (not files, but obviously
>>> all of the function dependencies) and individual variables. Note, stress
>>> is on /static/ libraries (.lib on Windows and .a on Linux, so no .dll or
>>> .so statically linked, in case this were a thing)
>>>
>> Traditionally - since before C existed - the granularity level for
>> static libraries has been the file level.  If you look at libraries
>> where it is normal to except applications to want to use only a few
>> functions out of many, such as the standard C library, the source code
>> is typically divided into one file per function.  You might have several
>> related functions in a single file (such as sin and cos, or all the
>> printf family), for slightly improved efficiency or speed of linking.
>>
>> This is how static libraries have worked with good and bad compilers,
>> and assemblers, and compilers of different languages, since the concept
>> of separate compiler, assembler, librarian and linker programs was
>> invented.
>>
>> Good tools, such as gcc and clang, have features such as flags to put
>> each function in a separate section, allowing more efficient linking of
>> static libraries (and object code files) without having to split the
>> source up into so many small files.  This is not unique to C - I first
>> saw it as "smart linking" in Borland Pascal for DOS.  For gcc, you do
>> need to use a few compiler flags, which is difficult for people who have
>> flag allergies and curl into a fetal position at the mention of "build
>> systems" or "makefiles".
>>
>> In more modern times - the last decade or so - link-time optimisation
>> has moved into the mainstream for good general-purpose compilers.  That
>> not only cuts out code from static libraries that is not used by the
>> application, but can inline and otherwise optimise code across library
>> boundaries.  (In the embedded world, such "compile and link everything
>> at once" toolchains have been used since the nineties at least.)
>>
>
> On some environments such as Windows, using function-at-a-time linking
> instead of DLLs is pointless.

It can result in smaller and faster binaries with fewer dependencies.
Often that is a good thing.

>
> In the case of common libraries such as the C runtime, the chances are
> that other applications are already using them and so they are already
> in memory.

I would expect that a fair proportion of the functions in the C standard
library are not going to be in use by any program running on a typical
computer.

But you are right that this is one DLL that might see a good deal of
sharing on Windows (where DLL's, other than system ones, are often not
shared as each app has their own copy to avoid versioning problems).
And for bigger or slower functions at least, the extra overhead for
using a DLL function rather than a local statically linked function will
be negligible.

>
> In the case of the file called 'gcc.exe' of my tdm/gcc installation, it
> was presumably built with all those flags you mentioned, yet it STILL
> imports msvcrt.dll. (And if not, then why not?)
>
> If I use that gcc to build an app, and specify -lucrt to make use of
> libucrt.a (and whether or not I use -ffunction-sections for my app), the
> resulting executable STILL imports msvcrt.dll, but now ALSO imports:
>
>   api-ms-win-crt-environment-l1-1-0.dll
>   api-ms-win-crt-runtime-l1-1-0.dll
>
> for good measure.
>
>> This is not unique to C - I first
>> saw it as "smart linking" in Borland Pascal for DOS.
>
> For all of the eighties, I wrote all my own libraries, which didn't have
> many unnecessary functions. My language's runtime was 4-8KB, which
> included maths functions /and/ floating point emulation.
>

You can do that, for your own code. (The same applies to my code -
unused functions are very rare.) General libraries and languages for
use by many people will always have more functions than any given user
or program will need.

> The only external software used was in the OS via INT calls or whatever
> the mechanism was.
>
> There were some optional modules, mainly a choice of drivers (for video,
> printer etc), but once chosen, you needed the whole module. So there was
> no need for traditional linking, just a means the assemble and fixup the
> relevant set of modules.
>
> The point is that typical language implementations (apart from your
> Borland), had linker programs that were hopelessly inefficient,
> presumably even without that smart linking.
>
> I think I'd assumed it was because it was linking a function at a time,
> now you are suggesting that it was module at a time; so what on earth
> were they up to?

On 04/08/2021 15:08, David Brown wrote:
> On 04/08/2021 15:24, Bart wrote:

>> Some of us have gotten even more modern and eliminated linking
>> completely from the task of building a set of sources into one .exe or
>> or .dll file.
>
> If you have more than one source file or static object files or static
> libraries, you are linking.

Not with languages designed for whole-program compilation as mine now are.

For references outside the program, then using DLLs, there is no actual
'linking' either (at best, a process which has to search DLLs looking
for where functions reside, but that info can also be provided by
directives).

The physical linking process is defered until execution, and there, for
PE format files, it is mainly filling in addresses in tables. On my
largest app, this process takes no longer than it does with hello.exe
(or at least, can't be measured).

With languages like C, it's hard to get around independent compilation
of modules, so a linking stage is necessary.

But even here, I don't use a traditional linker: I process in-memory
..asm files, and fix up symbol references across modules. There are no
object files and no discrete linker.

(I may soon be simplifying that process further for languages that need
separate compilation.)

Another area where you really need to use DLLs is with interpreted
languages where static linking is meaningless, or would get extremely
hairy even if all the info was available at compile-time.

On 8/4/21 10:08 AM, David Brown wrote:
> On 04/08/2021 15:24, Bart wrote:
>> On 04/08/2021 12:32, David Brown wrote:
>>> On 31/07/2021 19:33, Manfred wrote:
>>>> On 7/31/2021 3:56 PM, Bart wrote:
>>>>> On 31/07/2021 12:42, David Brown wrote:
>>>>>> On 31/07/2021 02:51, Bart wrote:
>>>>>>> On 30/07/2021 22:16, David Brown wrote:
>>>>>>>> On 30/07/2021 21:03, Bart wrote:
....
> General-purpose linkers worked on one object file at a time, and static
> libraries are basically just a collection of object files. Borland was
> in a position to control the language, compiler, linker, object file
> format, library file format - the whole process. So they were able to
> make a more efficient system.

I think it made sense, and still makes sense, to create multi-function
modules if the multiple functions are so closely related to each other
that they are guaranteed (or nearly so) to all be used by any program
that uses any one of them. For instance, fopen() could be bundled
together with fclose().

Bart <bc@freeuk.com> writes:
>On 04/08/2021 15:08, David Brown wrote:
>> On 04/08/2021 15:24, Bart wrote:
>
>>> Some of us have gotten even more modern and eliminated linking
>>> completely from the task of building a set of sources into one .exe or
>>> or .dll file.
>>
>> If you have more than one source file or static object files or static
>> libraries, you are linking.
>
>Not with languages designed for whole-program compilation as mine now are.

I would call that a defect, not a feature.

If it takes an hour to recompile after a single change to a single file,
it's useless; although I suspect you've never tried to compile a
project with millions of lines of code.

And in any case, the software development infrastructure that has
been developed over the past five decades isn't going to change
to cater to your whims.

On 04/08/2021 17:48, James Kuyper wrote:
> On 8/4/21 10:08 AM, David Brown wrote:
>> On 04/08/2021 15:24, Bart wrote:
>>> On 04/08/2021 12:32, David Brown wrote:
>>>> On 31/07/2021 19:33, Manfred wrote:
>>>>> On 7/31/2021 3:56 PM, Bart wrote:
>>>>>> On 31/07/2021 12:42, David Brown wrote:
>>>>>>> On 31/07/2021 02:51, Bart wrote:
>>>>>>>> On 30/07/2021 22:16, David Brown wrote:
>>>>>>>>> On 30/07/2021 21:03, Bart wrote:
> ...
>> General-purpose linkers worked on one object file at a time, and static
>> libraries are basically just a collection of object files. Borland was
>> in a position to control the language, compiler, linker, object file
>> format, library file format - the whole process. So they were able to
>> make a more efficient system.
>
> I think it made sense, and still makes sense, to create multi-function
> modules if the multiple functions are so closely related to each other
> that they are guaranteed (or nearly so) to all be used by any program
> that uses any one of them. For instance, fopen() could be bundled
> together with fclose().
>

Agreed.

On 04/08/2021 17:33, Scott Lurndal wrote:
> Bart <bc@freeuk.com> writes:
>> On 04/08/2021 15:08, David Brown wrote:
>>> On 04/08/2021 15:24, Bart wrote:
>>
>>>> Some of us have gotten even more modern and eliminated linking
>>>> completely from the task of building a set of sources into one .exe or
>>>> or .dll file.
>>>
>>> If you have more than one source file or static object files or static
>>> libraries, you are linking.
>>
>> Not with languages designed for whole-program compilation as mine now are.
>
> I would call that a defect, not a feature.
>
> If it takes an hour to recompile after a single change to a single file,

If it takes a hour to compile, say, 5M lines of code, then your
throughput is 1.4Klines/second, which is very poor.

My own compiler, unoptimised, works 200-250 times faster than that on a
low-end PC, on a language that is also somewhat challenging to compile
so needs extra passes.

A compiler like tcc, at the speed it works with C, would be 2-3 times as
fast. So 5M lines might take well under 10 seconds, or 15 seconds on
mine, on my PC. Expect better figures on a high-end PC, even with one core.

That's without exploiting opportunities to use multiple cores (although
parallelising is harder), or reusing data structures from a prior build.

But I don't need to do that, because my projects are typically 99% smaller.

> it's useless; although I suspect you've never tried to compile a
> project with millions of lines of code.

The only difference is the granularity of the compilation unit moves
from a module to a program.

But the 'program' is defined as being a single executable or library
file. An application can be multiple executables and libraries, and can
comprise also script code.

A properly designed application shouldn't be too monolithic, not when
developing. Although nothing stops a built application from being tidily
installed as a single file; then it doesn't matter. It can even
masquerade as a single .exe.

> And in any case, the software development infrastructure that has
> been developed over the past five decades isn't going to change
> to cater to your whims.

The infrastructure has been getting slower, especially with tools based
around LLVM.

(On one test, Rustc was 80,000 times slower - estimated as I couldn't
wait 22 hours - than Tiny C.)

Note that many languages are run from source anyway, like most scripting
ones. Then, even if some will be compiled on demand, a 10M line
application will still mean compiling 10M lines from source.

So the same 'problem' (although bytecode compilers might be twice as fast).

On 04/08/2021 18:40, Bart wrote:
> On 04/08/2021 17:33, Scott Lurndal wrote:

>> it's useless; although I suspect you've never tried to compile a
>> project with millions of lines of code.

> But the 'program' is defined as being a single executable or library
> file. An application can be multiple executables and libraries, and can
> comprise also script code.
>
> A properly designed application shouldn't be too monolithic, not when
> developing. Although nothing stops a built application from being tidily
> installed as a single file; then it doesn't matter. It can even
> masquerade as a single .exe.

I've surveyed the EXE and DLL files on my machine. There are 50,000 in
all. The largest are DLL files belonging programs like processor, at up
to 170MB.

But 99% of them are under 10MB.

On this machine, a fast non-optimising compiler can generate binary code
at about 2MB/s second. So 99% of these files could be built in under 5
seconds, 95% in under 2 seconds, and 85% in half a second.

Even that 170MB would take under 90 seconds, nowhere near an hour.
Although it would probably need sufficient resources.

Optimised code would take a bit longer, but this is a good starting
point. It shows that whole-program compilers can be entirely viable for
a large proportion of applications.

On 8/4/2021 1:40 PM, Bart wrote:

> A compiler like tcc, at the speed it works with C, would be 2-3 times as
> fast. So 5M lines might take well under 10 seconds, or 15 seconds on
> mine, on my PC. Expect better figures on a high-end PC, even with one core.

I recently installed the WinLibs build of gcc for Windows

https://winlibs.com

I compiled a 900-line file using tcc 0.9.27 and the winlibs gcc 11.2.0
for Windows.

The gcc executable is 6.8x larger, and the printf() parts of the code
run like a dog vs tcc. Kind of a bust, as I like snappy code.

------------------------------------------------------------------------------
$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=d:/computer/apps/compilers/mingw64/bin/../libexec/gcc/x86_64-w64-mingw32/11.2.0/lto-wrapper.exe
OFFLOAD_TARGET_NAMES=nvptx-none
Target: x86_64-w64-mingw32
Configured with: ../configure
--prefix=/R/winlibs64_stage/inst_gcc-11.2.0/share/gcc
--build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32
--enable-offload-targets=nvptx-none --with-pkgversion='MinGW-W64
x86_64-posix-seh, built by Brecht Sanders' --with-tune=generic
--enable-checking=release --enable-threads=posix
--disable-sjlj-exceptions --disable-libunwind-exceptions
--disable-serial-configure --disable-bootstrap --enable-host-shared
--enable-plugin --disable-default-ssp --disable-rpath
--enable-libstdcxx-pch --enable-libstdcxx-time=yes
--disable-libstdcxx-debug --disable-version-specific-runtime-libs
--with-stabs --disable-symvers
--enable-languages=c,c++,fortran,lto,objc,obj-c++,d,jit --disable-gold
--disable-nls --disable-stage1-checking --disable-win32-registry
--disable-multilib --enable-ld --enable-libquadmath --enable-libada
--enable-libssp --enable-libstdcxx --enable-lto
--enable-fully-dynamic-string --enable-libgomp --enable-graphite
--enable-mingw-wildcard --with-mpc=/d/Prog/winlibs64_stage/custombuilt
--with-mpfr=/d/Prog/winlibs64_stage/custombuilt
--with-gmp=/d/Prog/winlibs64_stage/custombuilt
--with-isl=/d/Prog/winlibs64_stage/custombuilt
--enable-install-libiberty --enable-__cxa_atexit
--without-included-gettext --with-diagnostics-color=auto
--enable-clocale=generic --with-libiconv --with-system-zlib
--with-build-sysroot=/R/winlibs64_stage/gcc-11.2.0/build_mingw/mingw-w64
CFLAGS=-I/d/Prog/winlibs64_stage/custombuilt/include/libdl-win32
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.2.0 (MinGW-W64 x86_64-posix-seh, built by Brecht Sanders)
------------------------------------------------------------------------------

On 07/08/2021 15:03, DFS wrote:
> On 8/4/2021 1:40 PM, Bart wrote:
>
>
>> A compiler like tcc, at the speed it works with C, would be 2-3 times
>> as fast. So 5M lines might take well under 10 seconds, or 15 seconds
>> on mine, on my PC. Expect better figures on a high-end PC, even with
>> one core.
>
>
> I recently installed the WinLibs build of gcc for Windows
>
> https://winlibs.com
>
> I compiled a 900-line file using tcc 0.9.27 and the winlibs gcc 11.2.0
> for Windows.
>
> The gcc executable is 6.8x larger, and the printf() parts of the code
> run like a dog vs tcc.  Kind of a bust, as I like snappy code.

Trying compiling with -s option, which omits symbols from the executable.

I'm surprised printf (is that to the screen or directed to a file?) is
slower; that should be little to do with the generated code.

All these big compilers run like dogs, but the generated code is
supposed to be faster.

On my machine, gcc-O3 version 11 compiled one of my 43Kloc programs (an
interpreter) in 22 seconds.

tcc did it in 0.15 seconds; 150 times faster.

But when I ran the result on a test input, the gcc-O3-generated program
took nearly 17 seconds, tcc took 35 seconds, just over twice as long:

Compile-speed Runtime
gcc 1.0 1.0
tcc 146.0 2.1

Bigger=better smaller=better

On 8/7/2021 12:11 PM, Bart wrote:
> On 07/08/2021 15:03, DFS wrote:
>> On 8/4/2021 1:40 PM, Bart wrote:
>>
>>
>>> A compiler like tcc, at the speed it works with C, would be 2-3 times
>>> as fast. So 5M lines might take well under 10 seconds, or 15 seconds
>>> on mine, on my PC. Expect better figures on a high-end PC, even with
>>> one core.
>>
>>
>> I recently installed the WinLibs build of gcc for Windows
>>
>> https://winlibs.com
>>
>> I compiled a 900-line file using tcc 0.9.27 and the winlibs gcc 11.2.0
>> for Windows.
>>
>> The gcc executable is 6.8x larger, and the printf() parts of the code
>> run like a dog vs tcc.  Kind of a bust, as I like snappy code.
>
> Trying compiling with -s option, which omits symbols from the executable.

That cut the size in half. Cool. Now it's only 3x as large as the tcc
..exe.

> I'm surprised printf (is that to the screen or directed to a file?) is
> slower; that should be little to do with the generated code.

It's to the screen. Only thing slightly different than usual is using a
width wildcard:

printf("%-*s ", longword, matchwords[i]);

I replaced it with a hardcoded value but it made no difference.

printf("%-16s ", matchwords[i]);

The gcc-Windows compiled version does printf() at 1/4 the speed of tcc.

> All these big compilers run like dogs, but the generated code is
> supposed to be faster.

Most everything but the printf() is just as fast as tcc, maybe a tad faster.

My mind is still boggled by the speed of some C programs.

On my old i5-750 system this program finds N strlen() matches out of
370K words in 33/1000ths of a second:

for(i=0;i<wordcnt;i++) {
if(strlen(words[i]) == atoi(&str[0])) {
matches++;
}
}

I moved the atoi() call out of the loop and the speed increased by 135%,
to 14/1000ths of a second!

int tlen = atoi(&str[0]);
for(i=0;i<wordcnt;i++) {
if(strlen(words[i]) == tlen) {
matches++;
}
}

Incredible.

> On my machine, gcc-O3 version 11 compiled one of my 43Kloc programs (an
> interpreter) in 22 seconds.
>
> tcc did it in 0.15 seconds; 150 times faster.
>
> But when I ran the result on a test input, the gcc-O3-generated program
> took nearly 17 seconds, tcc took 35 seconds, just over twice as long:
>
>
>       Compile-speed    Runtime
> gcc     1.0            1.0
> tcc   146.0            2.1
>
>       Bigger=better    smaller=better

How about the size of the executable?

On 08/08/2021 13:51, DFS wrote:
> On 8/7/2021 12:11 PM, Bart wrote:
>> On 07/08/2021 15:03, DFS wrote:
>>> On 8/4/2021 1:40 PM, Bart wrote:
>>>
>>>
>>>> A compiler like tcc, at the speed it works with C, would be 2-3
>>>> times as fast. So 5M lines might take well under 10 seconds, or 15
>>>> seconds on mine, on my PC. Expect better figures on a high-end PC,
>>>> even with one core.
>>>
>>>
>>> I recently installed the WinLibs build of gcc for Windows
>>>
>>> https://winlibs.com
>>>
>>> I compiled a 900-line file using tcc 0.9.27 and the winlibs gcc
>>> 11.2.0 for Windows.
>>>
>>> The gcc executable is 6.8x larger, and the printf() parts of the code
>>> run like a dog vs tcc.  Kind of a bust, as I like snappy code.
>>
>> Trying compiling with -s option, which omits symbols from the executable.
>
> That cut the size in half.  Cool.  Now it's only 3x as large as the tcc
> .exe.
>

What are the actual sizes? Because tcc compiles hello.c to 2KB, gcc to
47KB, as it contains more built-in stuff (eg. printf in gcc is a wrapper
function around vprintf).

>> I'm surprised printf (is that to the screen or directed to a file?) is
>> slower; that should be little to do with the generated code.
>
> It's to the screen.  Only thing slightly different than usual is using a
> width wildcard:
>
> printf("%-*s  ", longword, matchwords[i]);
>
> I replaced it with a hardcoded value but it made no difference.
>
> printf("%-16s  ", matchwords[i]);
>
>
> The gcc-Windows compiled version does printf() at 1/4 the speed of tcc.

You're right; this program:

#include <stdio.h>
int main(void) {
for (int i=0; i<10000; ++i)
printf("%-16s","1234567890"); // no newlines
}

runs in 3 seconds with tcc. With gcc, 21 seconds, even optimised. I've
never noticed before.

My bcc compiler, which also calls printf in msvcrt, does it in 2.3
seconds for some reason. lccwin is 15 seconds, DM 2.7. MSVC is 2.7 too.
Clang is 21 seconds (which is taking being a gcc clone to extremes).

All very strange. I don't know how much this is important, or how much
text you're printing to the screen, since 500 printfs per second is
still faster than anyone can read it. The problem disappears if you
redirect to a file.

You can try bypassing printf in gcc by using sprintf to a string for a
complete line of output, then using puts() on the string. However you
shouldn't need to do this.

> I moved the atoi() call out of the loop and the speed increased by 135%,
> to 14/1000ths of a second!
>
> int tlen = atoi(&str[0]);
> for(i=0;i<wordcnt;i++) {
>   if(strlen(words[i]) == tlen) {
>     matches++;
>   }
> }
>
>
> Incredible.

Here it might be an idea to calculate strlen() on each word once, and
store the lengths in a table (this could be done while reading the text
file). Then strlen() will be outside the loop too.

>
>
>
>> On my machine, gcc-O3 version 11 compiled one of my 43Kloc programs
>> (an interpreter) in 22 seconds.
>>
>> tcc did it in 0.15 seconds; 150 times faster.
>>
>> But when I ran the result on a test input, the gcc-O3-generated
>> program took nearly 17 seconds, tcc took 35 seconds, just over twice
>> as long:
>>
>>
>>        Compile-speed    Runtime
>> gcc     1.0            1.0
>> tcc   146.0            2.1
>>
>>        Bigger=better    smaller=better
>
>
> How about the size of the executable?

About the same, comparing tcc with gcc-O0 (857/853KB). gcc-O3 made a
bigger executable (925KB). The differences are bigger than they appear
since half the size here is embedded string literals.

But still, not significant in this case. (Maybe there is a gcc option to
optimise for size, but I don't know what it is.)

On 2021-08-08, Bart <bc@freeuk.com> wrote:
> My bcc compiler, which also calls printf in msvcrt, does it in 2.3
> seconds for some reason. lccwin is 15 seconds, DM 2.7. MSVC is 2.7 too.
> Clang is 21 seconds (which is taking being a gcc clone to extremes).

The printf MSVCRT.DLL may be fast, but it's still an undocumented system
component. Microsoft could make it disappear tomorrow, or introduce
some breaking change.

A few FOSS people moaning and groaning isn't going to stop a roll-out of
a new version of Windows that breaks or removes MSVCRT.DLL.

> All very strange.

How is it strange that one printf implementation has different
performance characteristics from a completely different printf
implementation?

> I don't know how much this is important, or how much
> text you're printing to the screen, since 500 printfs per second is
> still faster than anyone can read it. The problem disappears if you
> redirect to a file.

Might it be that, similarly to the Cygwin DLL, this library
does some "cooking" of its console streams?

I know that the Cygwin DLL does.

In Cygwin, all input and output on console descriptors passes through a
software layer which:

- simulates a POSIX TTY device: tcgetattr, tcsetattr, ...

- interprets ANSI/VT100 escape sequences, converting them into Console
API calls, on both input and output.

This means that POSIX code which controls the terminal with termios,
and/or which uses hard-coded ANSI escape sequences will work.

The MSVCRT.DLL printf will not do anything like this for you.

It's pretty necessary for porting some POSIX programs to Windows,
though, without the huge hassle of rewriting their TTY related code into
Console API calls.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal

Bart <bc@freeuk.com> writes:

>> The gcc-Windows compiled version does printf() at 1/4 the speed of tcc.
>
> You're right; this program:
>
> #include <stdio.h>
> int main(void) {
> for (int i=0; i<10000; ++i)
> printf("%-16s","1234567890"); // no newlines
> }
>
> runs in 3 seconds with tcc. With gcc, 21 seconds, even optimised. I've
> never noticed before.

You often post times with no indication of what's being measured. The
compiler probably has little to do with it. What C library is being
used? How is the output handled? On my laptop, compiled with gcc
10.3.0 and linked against libc6 version 2.33, it's almost too fast to
measure:

$ time ./o >/dev/null

real 0m0.007s
user 0m0.004s
sys 0m0.004s

> My bcc compiler, which also calls printf in msvcrt, does it in 2.3
> seconds for some reason. lccwin is 15 seconds, DM 2.7. MSVC is 2.7
> too. Clang is 21 seconds (which is taking being a gcc clone to
> extremes).
>
> All very strange.

Yes, but without knowing what's being measured you may never resolve the
mystery.

> I don't know how much this is important, or how much text you're
> printing to the screen, since 500 printfs per second is still faster
> than anyone can read it. The problem disappears if you redirect to a
> file.

What problem disappears?

> You can try bypassing printf in gcc by using sprintf to a string for a
> complete line of output, then using puts() on the string. However you
> shouldn't need to do this.

Why do you think the problem (whatever it is) has got anything to do
with gcc if the problem disappears when the output is redirected?

--
Ben.

On 08/08/2021 18:12, Kaz Kylheku wrote:
> On 2021-08-08, Bart <bc@freeuk.com> wrote:
>> My bcc compiler, which also calls printf in msvcrt, does it in 2.3
>> seconds for some reason. lccwin is 15 seconds, DM 2.7. MSVC is 2.7 too.
>> Clang is 21 seconds (which is taking being a gcc clone to extremes).
>
> The printf MSVCRT.DLL may be fast, but it's still an undocumented system
> component. Microsoft could make it disappear tomorrow, or introduce
> some breaking change.
>
> A few FOSS people moaning and groaning isn't going to stop a roll-out of
> a new version of Windows that breaks or removes MSVCRT.DLL.
>
>> All very strange.
>
> How is it strange that one printf implementation has different
> performance characteristics from a completely different printf
> implementation?

But the difference is vast; what can a printf routine possibly be doing
to result in the 1300% slowdown exhibited here (79 X's printed per line):

#include <stdio.h>

int main(void) {
char str[1000];
for (int i=0; i<2000; ++i) {
printf("XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n");

}
}

bcc/tcc take 1.75 seconds; gcc/clang take 25.5 seconds, which is
ridiculous. Clearly something is seriously amiss; you can't just shrug
it off and put it down somewhat different implementations.

And what gcc/clang are up isn't even something useful like implementing
"%zu" or ANSI escape sequences.

The mystery depends because if you write it like this:
puts("XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX");

Then gcc is as fast as the others. It's not *printf formatting overheads
either, because writing it this way:
sprintf(str,"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX");
puts(str);

is also fast.

I did manage to get gcc fast using printf, by replacing #include
<stdio.h> with this line:

extern int printf(const char*, ...);

Then it directly calls the real printf instead of this version:

__mingw_ovr
__attribute__((__format__ (gnu_printf, 1, 2)))
__MINGW_ATTRIB_NONNULL(1)
int printf (const char *__format, ...)
{
int __retval;
__builtin_va_list __local_argv; __builtin_va_start( __local_argv,
__format );
__retval = __mingw_vfprintf( stdout, __format, __local_argv );
__builtin_va_end( __local_argv );
return __retval;
}

>> I don't know how much this is important, or how much
>> text you're printing to the screen, since 500 printfs per second is
>> still faster than anyone can read it. The problem disappears if you
>> redirect to a file.
>
> Might it be that, similarly to the Cygwin DLL, this library
> does some "cooking" of its console streams?
>
> I know that the Cygwin DLL does.
>
> In Cygwin, all input and output on console descriptors passes through a
> software layer which:
>
> - simulates a POSIX TTY device: tcgetattr, tcsetattr, ...
>
> - interprets ANSI/VT100 escape sequences, converting them into Console
> API calls, on both input and output.
>
> This means that POSIX code which controls the terminal with termios,
> and/or which uses hard-coded ANSI escape sequences will work.
>
> The MSVCRT.DLL printf will not do anything like this for you.

But if it did, it wouldn't slow it down at all, because the fast path
(not inside a control sequence is just as fast). And gcc (not in Cygwin
but normal Windows) doesn't support that anyway.

> It's pretty necessary for porting some POSIX programs to Windows,
> though, without the huge hassle of rewriting their TTY related code into
> Console API calls.
>

On 08/08/2021 20:21, Ben Bacarisse wrote:
> Bart <bc@freeuk.com> writes:
>
>>> The gcc-Windows compiled version does printf() at 1/4 the speed of tcc.
>>
>> You're right; this program:
>>
>> #include <stdio.h>
>> int main(void) {
>> for (int i=0; i<10000; ++i)
>> printf("%-16s","1234567890"); // no newlines
>> }
>>
>> runs in 3 seconds with tcc. With gcc, 21 seconds, even optimised. I've
>> never noticed before.
>
> You often post times with no indication of what's being measured.

Huh? The test program is shown above. A tcc-compiled program takes 7
times as long as gcc- or clang-compiled one. On Windows (I know Linux
has super-fast console output).

What I'm measuring is how long between starting the program, and
finishing writing stuff to the screen, because on Windows you can see
all this output scrolling up the screen.

> The
> compiler probably has little to do with it. What C library is being
> used? How is the output handled?

I don't know, but... should anyone really have to delve into the exact
implementation of printf to get it to do its job properly? It should
just work! Which means not working a magnitude slower than it should.

See my other post a few minutes after yours for further investigations.

>> You can try bypassing printf in gcc by using sprintf to a string for a
>> complete line of output, then using puts() on the string. However you
>> shouldn't need to do this.
>
> Why do you think the problem (whatever it is) has got anything to do
> with gcc if the problem disappears when the output is redirected?

The 'problem' is with clang/gcc calling 'printf'. All compilers are told
to use printf, yet only these two compilers end up being slow.

Maybe they end up calling some slow, character-at-a-time routine in the
C library which, for the console, means invoking some of those elaborate
WinAPI routines which have high overheads. And maybe those other
compilers manage to do it a string-at-a-time.

But if that is the case, who's fault is that, and what can one do about
it? Do you think it is likely to involve messing with the options for
these gcc or clang installations?

Because you seem keen to put the blame elsewhere.

What can you tell DFS to make fix this problem when using gcc?

Ben Bacarisse <ben.usenet@bsb.me.uk> writes:
>Bart <bc@freeuk.com> writes:
>
>>> The gcc-Windows compiled version does printf() at 1/4 the speed of tcc.
>>
>> You're right; this program:
>>
>> #include <stdio.h>
>> int main(void) {
>> for (int i=0; i<10000; ++i)
>> printf("%-16s","1234567890"); // no newlines
>> }
>>
>> runs in 3 seconds with tcc. With gcc, 21 seconds, even optimised. I've
>> never noticed before.
>
>You often post times with no indication of what's being measured. The
>compiler probably has little to do with it. What C library is being
>used? How is the output handled? On my laptop, compiled with gcc
>10.3.0 and linked against libc6 version 2.33, it's almost too fast to
>measure:
>
>$ time ./o >/dev/null
>
>real 0m0.007s
>user 0m0.004s
>sys 0m0.004s

An aggressive optimizer (speed vs space) could optimize it
into a single puts of a 160,000 character string.

On 8/8/2021 10:41 AM, Bart wrote:
> On 08/08/2021 13:51, DFS wrote:
>> On 8/7/2021 12:11 PM, Bart wrote:
>>> On 07/08/2021 15:03, DFS wrote:
>>>> On 8/4/2021 1:40 PM, Bart wrote:
>>>>
>>>>
>>>>> A compiler like tcc, at the speed it works with C, would be 2-3
>>>>> times as fast. So 5M lines might take well under 10 seconds, or 15
>>>>> seconds on mine, on my PC. Expect better figures on a high-end PC,
>>>>> even with one core.
>>>>
>>>>
>>>> I recently installed the WinLibs build of gcc for Windows
>>>>
>>>> https://winlibs.com
>>>>
>>>> I compiled a 900-line file using tcc 0.9.27 and the winlibs gcc
>>>> 11.2.0 for Windows.
>>>>
>>>> The gcc executable is 6.8x larger, and the printf() parts of the
>>>> code run like a dog vs tcc.  Kind of a bust, as I like snappy code.
>>>
>>> Trying compiling with -s option, which omits symbols from the
>>> executable.

>> That cut the size in half.  Cool.  Now it's only 3x as large as the
>> tcc .exe.
>>
>
> What are the actual sizes? Because tcc compiles hello.c to 2KB, gcc to
> 47KB, as it contains more built-in stuff (eg. printf in gcc is a wrapper
> function around vprintf).

gcc : 85,435
gcc -s : 41,984
gcc -Os -s: 38,400

tcc : 15,360
tcc -s : 15,360
tcc -Os -s: 15,360

All 6 variations run fine, so tcc apparently ignores various flags in
the compilation:

>>> I'm surprised printf (is that to the screen or directed to a file?)
>>> is slower; that should be little to do with the generated code.
>>
>> It's to the screen.  Only thing slightly different than usual is using
>> a width wildcard:
>>
>> printf("%-*s  ", longword, matchwords[i]);
>>
>> I replaced it with a hardcoded value but it made no difference.
>>
>> printf("%-16s  ", matchwords[i]);
>>
>>
>> The gcc-Windows compiled version does printf() at 1/4 the speed of tcc.
>
> You're right; this program:
>
> #include <stdio.h>
> int main(void) {
>     for (int i=0; i<10000; ++i)
>         printf("%-16s","1234567890"); // no newlines
> }
>
> runs in 3 seconds with tcc. With gcc, 21 seconds, even optimised. I've
> never noticed before.
>
> My bcc compiler, which also calls printf in msvcrt, does it in 2.3
> seconds for some reason. lccwin is 15 seconds, DM 2.7. MSVC is 2.7 too.
> Clang is 21 seconds (which is taking being a gcc clone to extremes).
>
> All very strange. I don't know how much this is important, or how much
> text you're printing to the screen, since 500 printfs per second is
> still faster than anyone can read it. The problem disappears if you
> redirect to a file.

For example, printf() 726 words split across N left-aligned columns:

Windows:
gcc: 3.07s
tcc: 0.24s

> You can try bypassing printf in gcc by using sprintf to a string for a
> complete line of output, then using puts() on the string. However you
> shouldn't need to do this.

I shouldn't. But I put your idea in place and it works a treat.

printf sprintf/puts
gcc: 3.07s 0.06s
tcc: 0.24s 0.06s

Good suggestion.

I actually had to use sprintf() and strcat() in a loop to build up the line:

for(int j = 1;j<=maxcols;j++) {
sprintf(buff,"%-*s ",longword, words[i++]);
strcat(line,buff);
}

>> I moved the atoi() call out of the loop and the speed increased by
>> 135%, to 14/1000ths of a second!
>>
>> int tlen = atoi(&str[0]);
>> for(i=0;i<wordcnt;i++) {
>>    if(strlen(words[i]) == tlen) {
>>      matches++;
>>    }
>> }
>>
>>
>> Incredible.
>
> Here it might be an idea to calculate strlen() on each word once, and
> store the lengths in a table (this could be done while reading the text
> file). Then strlen() will be outside the loop too.

I've done that in the past, but not in this code.

>>> On my machine, gcc-O3 version 11 compiled one of my 43Kloc programs
>>> (an interpreter) in 22 seconds.
>>>
>>> tcc did it in 0.15 seconds; 150 times faster.
>>>
>>> But when I ran the result on a test input, the gcc-O3-generated
>>> program took nearly 17 seconds, tcc took 35 seconds, just over twice
>>> as long:
>>>
>>>
>>>        Compile-speed    Runtime
>>> gcc     1.0            1.0
>>> tcc   146.0            2.1
>>>
>>>        Bigger=better    smaller=better
>>
>>
>> How about the size of the executable?
>
> About the same, comparing tcc with gcc-O0 (857/853KB). gcc-O3 made a
> bigger executable (925KB). The differences are bigger than they appear
> since half the size here is embedded string literals.
>
> But still, not significant in this case. (Maybe there is a gcc option to
> optimise for size, but I don't know what it is.)

-Os

"Optimize for size. -Os enables all -O2 optimizations except those that
often increase code size:

-falign-functions -falign-jumps
-falign-labels -falign-loops
-fprefetch-loop-arrays -freorder-blocks-algorithm=stc

It also enables -finline-functions, causes the compiler to tune for code
size rather than execution speed, and performs further optimizations
designed to reduce code size."

https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html

On Sunday, 8 August 2021 at 20:38:32 UTC+1, Bart wrote:
>
> The 'problem' is with clang/gcc calling 'printf'. All compilers are told
> to use printf, yet only these two compilers end up being slow.
>
> Maybe they end up calling some slow, character-at-a-time routine in the
> C library which, for the console, means invoking some of those elaborate
> WinAPI routines which have high overheads. And maybe those other
> compilers manage to do it a string-at-a-time.
>
> But if that is the case, who's fault is that, and what can one do about
> it? Do you think it is likely to involve messing with the options for
> these gcc or clang installations?
>
Why would you need a fast printf? There are a few scenarios I can
think of, like web servers that need a very fast throughput. But they are
quite specialised. And yes, you can probably get a fast printf for such
purposes by messing about with clang/gcc's settings.
However if printf either sends characters to a scrolling character-mapped
console, or is redirected to a fairly slow backing store, then really there's
no point in trying to link in a fast version.

On 08/08/2021 22:00, Malcolm McLean wrote:
> On Sunday, 8 August 2021 at 20:38:32 UTC+1, Bart wrote:
>>
>> The 'problem' is with clang/gcc calling 'printf'. All compilers are told
>> to use printf, yet only these two compilers end up being slow.
>>
>> Maybe they end up calling some slow, character-at-a-time routine in the
>> C library which, for the console, means invoking some of those elaborate
>> WinAPI routines which have high overheads. And maybe those other
>> compilers manage to do it a string-at-a-time.
>>
>> But if that is the case, who's fault is that, and what can one do about
>> it? Do you think it is likely to involve messing with the options for
>> these gcc or clang installations?
>>
> Why would you need a fast printf? There are a few scenarios I can
> think of, like web servers that need a very fast throughput. But they are
> quite specialised. And yes, you can probably get a fast printf for such
> purposes by messing about with clang/gcc's settings.
> However if printf either sends characters to a scrolling character-mapped
> console, or is redirected to a fairly slow backing store, then really there's
> no point in trying to link in a fast version.
>

Well, DFS reported 3 seconds to display output from a program instead of
the 0.25 seconds it ought to have been. That's an annoying and
unnecessary wait.

The question ought to be, why would you want a slow printf? There's
clearly something wrong here, since other programs can display output
perfectly well, even when they themselves use printf.

Bart <bc@freeuk.com> writes:

> On 08/08/2021 20:21, Ben Bacarisse wrote:
>> Bart <bc@freeuk.com> writes:
>>
>>>> The gcc-Windows compiled version does printf() at 1/4 the speed of tcc.
>>>
>>> You're right; this program:
>>>
>>> #include <stdio.h>
>>> int main(void) {
>>> for (int i=0; i<10000; ++i)
>>> printf("%-16s","1234567890"); // no newlines
>>> }
>>>
>>> runs in 3 seconds with tcc. With gcc, 21 seconds, even optimised. I've
>>> never noticed before.
>> You often post times with no indication of what's being measured.
>
> Huh? The test program is shown above. A tcc-compiled program takes 7
> times as long as gcc- or clang-compiled one. On Windows (I know Linux
> has super-fast console output).
>
> What I'm measuring is how long between starting the program, and
> finishing writing stuff to the screen, because on Windows you can see
> all this output scrolling up the screen.

You've been in the business for many years. I can't believe you don't
know what I mean.

>> The
>> compiler probably has little to do with it. What C library is being
>> used? How is the output handled?
>
> I don't know, but... should anyone really have to delve into the exact
> implementation of printf to get it to do its job properly? It should
> just work! Which means not working a magnitude slower than it should.

I my machine is three times faster than yours (just to get a simple
number) we are talking about three orders of magnitude. Do you think
that's down to the implementation of printf? You certainly haven't
posted any data to support that.

> See my other post a few minutes after yours for further investigations.
>
>>> You can try bypassing printf in gcc by using sprintf to a string for a
>>> complete line of output, then using puts() on the string. However you
>>> shouldn't need to do this.
>> Why do you think the problem (whatever it is) has got anything to do
>> with gcc if the problem disappears when the output is redirected?
>
> The 'problem' is with clang/gcc calling 'printf'. All compilers are
> told to use printf, yet only these two compilers end up being slow.

I thought you said the problem goes away when the output is redirected?
gcc and clang don't generate code that does not call printf when the
output is redirected.

> But if that is the case, who's fault is that, and what can one do
> about it? Do you think it is likely to involve messing with the
> options for these gcc or clang installations?
>
> Because you seem keen to put the blame elsewhere.

Because I think the blame must be elsewhere, but I'm not going to
install Windows (and all the other stuff one might need) just to find
out.

> What can you tell DFS to make fix this problem when using gcc?

To get 1000-fold increase in speed, ditch Windows. If he or she must
use Windows, find out what's taking the time.

--
Ben.

On 08/08/2021 22:17, Ben Bacarisse wrote:
> Bart <bc@freeuk.com> writes:

>> The 'problem' is with clang/gcc calling 'printf'. All compilers are
>> told to use printf, yet only these two compilers end up being slow.
>
> I thought you said the problem goes away when the output is redirected?
> gcc and clang don't generate code that does not call printf when the
> output is redirected.

I thought I explained what might have been happening: gcc might end up
calling a routine in msvcrt that works a character at a time, rather
than a string at a time. And the former might be slower when sent to the
console so you don't see a problem with file output.

It seems that bcc/tcc directly call msvcrt's printf, passing it a whole
string. But gcc at least implements printf as a call to __mingw_vfprintf
inside some vararg-handing code. I don't know what goes on inside
_mingw_vfprintf; as far as I can see, nothing of benefit.

>> But if that is the case, who's fault is that, and what can one do
>> about it? Do you think it is likely to involve messing with the
>> options for these gcc or clang installations?
>>
>> Because you seem keen to put the blame elsewhere.
>
> Because I think the blame must be elsewhere, but I'm not going to
> install Windows (and all the other stuff one might need) just to find
> out.
>
>> What can you tell DFS to make fix this problem when using gcc?
>
> To get 1000-fold increase in speed, ditch Windows. If he or she must
> use Windows, find out what's taking the time.

Where did the 1000-fold increase come from? The slowdown on my last test
was about 14 times, in line with what DFS reported.

The problem occurs when using gcc/clang (and to lesser extent lccwin).

You don't want displaying to a screen being a magnitude slower than
necessary.

On 08/08/2021 22:32, Bart wrote:
> On 08/08/2021 22:17, Ben Bacarisse wrote:
>> Bart <bc@freeuk.com> writes:
>
>>> The 'problem' is with clang/gcc calling 'printf'. All compilers are
>>> told to use printf, yet only these two compilers end up being slow.
>>
>> I thought you said the problem goes away when the output is redirected?
>> gcc and clang don't generate code that does not call printf when the
>> output is redirected.
>
> I thought I explained what might have been happening: gcc might end up
> calling a routine in msvcrt that works a character at a time, rather
> than a string at a time. And the former might be slower when sent to the
> console so you don't see a problem with file output.

If I tweak my program than prints lines of 79 X's, to print them one
character at a time like this: printf("X"), then, guess what, bcc/tcc
take 25 seconds too!

Further, if I redirect the output to a file, then that is fast as well.

You will probably still want to point the blame at this library, for a
having console output that is not as efficient as it could be, and away
from gcc/clang implementations.

But it is those implementations that make things worse by, somehow,
doing character at a time output at the point where it calls into the
library, so invoking 79 lots of overhead when there could have been just
one.

This explains also why using puts, or sprintf into an intermediate
buffer then puts, doesn't exhibit the slowdown.