Bart <bc@freeuk.com> writes:
>On 13/08/2023 17:48, Bart wrote:
> > On 13/08/2023 17:02, Scott Lurndal wrote:

>
>If using -O0 is viable, then look at a faster compiler. Both tcc and gcc
>(when it managed it) produced 34MB executables for my test.

Both of those compilers would choke in 10 seconds on any of this code.

On 2023-08-13, Scott Lurndal <scott@slp53.sl.home> wrote:
> Bart <bc@freeuk.com> writes:
>>On 13/08/2023 17:02, Scott Lurndal wrote:
>>> David Brown <david.brown@hesbynett.no> writes:
>>>> On 12/08/2023 13:12, Bart wrote:
>>>
>>>>
>>>>> Suppose you dispensed with all that dependency stuff; how long would it
>>>>> take to build such a project? Say compared with just compiling one module.
>>>>>
>>>>
>>>> That particular project takes a minute and a half to compile on my
>>>> laptop (it's faster on my main machine). It's 373 compilations, perhaps
>>>> 40% of it C++. Too long for convenience.
>>>
>>> My current project takes well over an hour to build on a single-core
>>> system. With 32 cores and parallel make (-j32), that's reduced to
>>> less than ten minutes
>>
>>You don't get a 32x speedup then?
>
> Compiler workloads are generally I/O bound.

Not really. E.g. if you use ccache, and everything is a hit,
it's "lightning fast" just to copy the .o files from the cache.

I just compared a build:

From ccache: 1.0s.

Actual compile: 96s.

I.e. 96x faster to pull .o from the cache after preprocessing the
translation unit and taking a hash of it and the compiler options to
look it up.

(Everything is "cache hot" in terms of VM.)

How is the makefile structured? Is it all one big dependency
tree or is there recursion?

If there is a top Makefile which has recipes that sequentially
walk through subdirectories and call $(MAKE), that can throttle
the parallel build. If a directory has fewer than 32 items,
the full parallelism won't be realized, and it has to be
done before the next subdirectory.

For high parallelization, the best thing is one large dependency tree.

The second is recursing in parallel into sub-makefiles, using
parallelizable rules rather than scripted execution.

Sub-makes communicate with the top make to keep the total jobs in the
submakes within the -j parameter. I think the top make runs a
job server that the submakes use, or something like that.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

Ben Bacarisse <ben.usenet@bsb.me.uk> writes:
> Phil Carmody <pc+usenet@asdf.org> writes:
>> What's wrong with:
>>
>> unsigned int low = 0xC0000000;
>> unsigned int high = 0xE0000000;
>> unsigned int mid = (low | high) + (low & high)/2;
>
> Did you mean (low | high)/2 + (low & high)/2 ?

Nah, I was way wronger than that.

unsigned int mid = (low & high) + (low ^ high)/2;

The logic being that every bit that's common to both needs to be doubled
and halved - so left the same - and every bit that's in one but not the
other needs to be added - without any carries, thus no overflow - and
then halved.

Phil
--
We are no longer hunters and nomads. No longer awed and frightened, as we have
gained some understanding of the world in which we live. As such, we can cast
aside childish remnants from the dawn of our civilization.
-- NotSanguine on SoylentNews, after Eugen Weber in /The Western Tradition/

Phil Carmody <pc+usenet@asdf.org> writes:

> Ben Bacarisse <ben.usenet@bsb.me.uk> writes:
>
>> Phil Carmody <pc+usenet@asdf.org> writes:
>>
>>> What's wrong with:
>>>
>>> unsigned int low = 0xC0000000;
>>> unsigned int high = 0xE0000000;
>>> unsigned int mid = (low | high) + (low & high)/2;
>>
>> Did you mean (low | high)/2 + (low & high)/2 ?
>
> Nah, I was way wronger than that.
>
> unsigned int mid = (low & high) + (low ^ high)/2;
>
> The logic being that every bit that's common to both needs to be doubled
> and halved - so left the same - and every bit that's in one but not the
> other needs to be added - without any carries, thus no overflow - and
> then halved.

A clever identity. I don't remember seeing this before.
Hat tip to you!

Kaz Kylheku <864-117-4973@kylheku.com> writes:

(a few selected replies.. too much mishmash for the rest)

> On 2023-08-05, Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>
>> Kaz Kylheku <864-117-4973@kylheku.com> writes:

[...]

>>> The term "literal constant" has a well-entrenched meaning in
>>> computer science.
>>
>> No doubt the phrase "literal constant" is used in many places
>> that discuss programming, but it is not a term with any special
>> meaning, simply an ordinary phrase put together under the usual
>> rules of normal English usage, with "literal" being an adjective
>> modifying "constant".
>
> Indeed, the phrase isn't a noncompositional compound, like
> "blackboard" or "blueberry".
>
> So what? And, nobody claimed that it is.

Your earlier statement implied that "literal constant" is a term
of art. It isn't. It's just an ordinary English phrase.

[...]

>> none of [how the phrase is used] has any bearing on the use of
>> "literal" as a noun by itself, which is a distinct and unrelated
>> construction.
>
> Sayss you. I believe it's just derived from "literal constant"
> or similar. [...]

Your belief is contradicted by historical facts.

> Constant itself is an adjective. [...]

The word "constant" is used as an adjective but it is also used
as a noun. "Constant" in computer science obviously is closely
related to "constant" in mathematics, where it has been used as
a noun for hundreds of years.

Tim Rentsch <tr.17687@z991.linuxsc.com> writes:
> Phil Carmody <pc+usenet@asdf.org> writes:
>> Ben Bacarisse <ben.usenet@bsb.me.uk> writes:
>>> Phil Carmody <pc+usenet@asdf.org> writes:
>>>
>>>> What's wrong with:
>>>>
>>>> unsigned int low = 0xC0000000;
>>>> unsigned int high = 0xE0000000;
>>>> unsigned int mid = (low | high) + (low & high)/2;
>>>
>>> Did you mean (low | high)/2 + (low & high)/2 ?
>>
>> Nah, I was way wronger than that.
>>
>> unsigned int mid = (low & high) + (low ^ high)/2;
>>
>> The logic being that every bit that's common to both needs to be doubled
>> and halved - so left the same - and every bit that's in one but not the
>> other needs to be added - without any carries, thus no overflow - and
>> then halved.
>
> A clever identity. I don't remember seeing this before.
> Hat tip to you!

Thanks, and sorry for the confusion, I last used it nearly decades ago
while bit-bashing on a DSP.

Finally - note that this rounds down, which isn't to everyone's taste.
A naive '+1' bump before the /2 can overflow, so you want to simply
add the bottom bit of low^high back into the sum.

Phil
--
We are no longer hunters and nomads. No longer awed and frightened, as we have
gained some understanding of the world in which we live. As such, we can cast
aside childish remnants from the dawn of our civilization.
-- NotSanguine on SoylentNews, after Eugen Weber in /The Western Tradition/

Kaz Kylheku <864-117-4973@kylheku.com> writes:
>On 2023-08-13, Scott Lurndal <scott@slp53.sl.home> wrote:
>> Bart <bc@freeuk.com> writes:
>>>On 13/08/2023 17:02, Scott Lurndal wrote:
>>>> David Brown <david.brown@hesbynett.no> writes:
>>>>> On 12/08/2023 13:12, Bart wrote:
>>>>
>>>>>
>>>>>> Suppose you dispensed with all that dependency stuff; how long would it
>>>>>> take to build such a project? Say compared with just compiling one module.
>>>>>>
>>>>>
>>>>> That particular project takes a minute and a half to compile on my
>>>>> laptop (it's faster on my main machine). It's 373 compilations, perhaps
>>>>> 40% of it C++. Too long for convenience.
>>>>
>>>> My current project takes well over an hour to build on a single-core
>>>> system. With 32 cores and parallel make (-j32), that's reduced to
>>>> less than ten minutes
>>>
>>>You don't get a 32x speedup then?
>>
>> Compiler workloads are generally I/O bound.
>
>Not really. E.g. if you use ccache, and everything is a hit,
>it's "lightning fast" just to copy the .o files from the cache.

We've used ccache, and had enough problems with it that we
abandoned it.

Using dependencies means that only files with changes get
rebuilt, and that's sufficient.

>
>If there is a top Makefile which has recipes that sequentially
>walk through subdirectories and call $(MAKE)

Why would one write a makefile to serialize the subdirectories?

The should be (and in our case are) fully independent.

On 2023-08-14, Scott Lurndal <scott@slp53.sl.home> wrote:
> Kaz Kylheku <864-117-4973@kylheku.com> writes:
>>On 2023-08-13, Scott Lurndal <scott@slp53.sl.home> wrote:
>>> Bart <bc@freeuk.com> writes:
>>>>On 13/08/2023 17:02, Scott Lurndal wrote:
>>>>> David Brown <david.brown@hesbynett.no> writes:
>>>>>> On 12/08/2023 13:12, Bart wrote:
>>>>>
>>>>>>
>>>>>>> Suppose you dispensed with all that dependency stuff; how long would it
>>>>>>> take to build such a project? Say compared with just compiling one module.
>>>>>>>
>>>>>>
>>>>>> That particular project takes a minute and a half to compile on my
>>>>>> laptop (it's faster on my main machine). It's 373 compilations, perhaps
>>>>>> 40% of it C++. Too long for convenience.
>>>>>
>>>>> My current project takes well over an hour to build on a single-core
>>>>> system. With 32 cores and parallel make (-j32), that's reduced to
>>>>> less than ten minutes
>>>>
>>>>You don't get a 32x speedup then?
>>>
>>> Compiler workloads are generally I/O bound.
>>
>>Not really. E.g. if you use ccache, and everything is a hit,
>>it's "lightning fast" just to copy the .o files from the cache.
>
> We've used ccache, and had enough problems with it that we
> abandoned it.

For my purposes, I'm not running into the problem; ccache
is working.

Subtly broken or not, for the present purposes, it shows that two orders
of magnitude of run-time can get saved by hashing the translation unit
and just copying canned .o files.

> Using dependencies means that only files with changes get
> rebuilt, and that's sufficient.
>
>
>>
>>If there is a top Makefile which has recipes that sequentially
>>walk through subdirectories and call $(MAKE)
>
> Why would one write a makefile to serialize the subdirectories?
>
> The should be (and in our case are) fully independent.

I wouldn't myself, but such recursive make systems are out there
in the wild.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

On 2023-08-14, Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
> Kaz Kylheku <864-117-4973@kylheku.com> writes:
>> Sayss you. I believe it's just derived from "literal constant"
>> or similar. [...]
>
> Your belief is contradicted by historical facts.
>
>> Constant itself is an adjective. [...]
>
> The word "constant" is used as an adjective but it is also used
> as a noun. "Constant" in computer science obviously is closely
> related to "constant" in mathematics, where it has been used as
> a noun for hundreds of years.

Douglas Harper's etymonline.com pegs "variable" to 1816, and "constant"
to 1832:

1832 in mathematics and physics, "a quantity which is assumed to be
invariable throughout," from constant (adj.), which is attested from
1753 in mathematics. The general sense "that which is not subject to
change" (1856) is a figurative extension from this.

It looks like an adjective-noun of convenience, like "deductible",
"convertible", "dirigible", ... formed by a process in the English
language whereby an understood noun that would be tedious to repeat
is droppped, leaving only the adjective.

The insurance adjustor doesn't have to keep repeating "deductible
portion of the claim", and just refers to "deductible".

Likwise, Hillary Clinton doesn't have to repeat mouthfuls like
"deplorable persons from a lower class holding illiberal opinions";
one word does it all.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

On 09/08/2023 14:32, Malcolm McLean wrote:
> On Wednesday, 9 August 2023 at 12:43:06 UTC+1, David Brown wrote:
>> On 09/08/2023 05:23, Malcolm McLean wrote:
>>
>>> CMake is an alternative to make.
>> It is a "meta-build" tool, aimed at a higher level than make, and which
>> generates scripts or control files for various uses - makefiles, MSVC
>> project files, and others. For some kinds of project, that might be a
>> good choice - sometimes it makes sense to separate the tasks of
>> calculating dependencies and settings, and actually controlling the
>> build. But since on many platforms CMake depends on make, it is not an
>> alternative. And CMake is not flexible enough for many uses - when I
>> have looked at it, it was not remotely flexible enough to do the kinds
>> of things I do with make. Many people use CMake, so it must have
>> merits, but it is not a replacement for make.
>>
> CMake has a "generator", which is their word for target output format.
> So you can use it to generate Xcode or Visual Studio files, or make files.
> The two big advantages are that you don't have to ship an Xcode or
> Visual Studio project file. CMake generates it for your client. And the
> Cmkae scripts are easier to read and maintain than makefiles.

It is inaccurate to say "CMake /has/ a generator" - it is better to say
it /is/ a generator. Apart from that, you have basically repeated what
I wrote.

>>> It can generate make files, in fact that's the
>>> default. Which means that if you ship a project with a CMake script, you
>>> wouldn't normally provide a make file. CMake found a niche largely because
>>> make files are too difficult to use. The other reason is that the big IDEs
>>> rejected makefiles as their format for describing how to build projects.
>> Every IDE of any quality can handle projects that are built using
>> external makefiles. An IDE that won't run "make" at the press of a
>> hotkey is as useful as an IDE that can't handle ASCII format test files.
>>
>> But if you have an external manually written makefile, that is not
>> project management - so usually you also then have to make an IDE
>> project to describe the source files, include search paths, pre-defined
>> macros, etc. That can be a duplicate of work (though good makefiles
>> find the source files and dependencies mostly automatically), and
>> duplication can lead to mistakes.
>>
> My point is that when the first IDEs came out, they could have said "the project
> description file will be a makefile, maybe with added comments to represent
> information needed by the iDE but not by the compiler". They opted not to
> do this.

IDE's need all kinds of things in the project management that are
irrelevant to the build process, and therefore unsuitable for storing in
a makefile. And makefiles can have all sorts of things that are
difficult for something like an IDE to interpret - they are far too
flexible. So "project description files" are IDE-specific, and IDE's
could not possibly use makefiles for the purpose. CMake files are less
flexible than makefiles, but are equally unsuitable for IDE project
management.

>>
>> The norm AFIK for build systems controlled by IDEs is make - they
>> generate makefiles automatically based on your project settings, dialog
>> boxes for choosing compiler flags, etc. Such systems can work
>> reasonably well, though a well-made manual makefile is often much better
>> at calculating dependency information for large projects.
>>
> There's a big difference between automatic scripts and ones written by humans.
> If something goes wrong with the automatic script, whilst a human programmer
> might have to look at it, basically the problem is almost certainly in the program
> which generated the script. So it doesn't matter too much if the script is
> unreadable. As long as the generating program is well-structured and
> debuggable.

That's all true, but I don't see the relevance.

>>
>> Some IDEs may generate CMake files, then run CMake, then make - I have
>> (unsurprisingly) not used all IDEs.
>>
>> CMake, as far as I understand it, is popular when you want a project to
>> build on Linux with gcc and Windows with MSVC - that is its niche.
>>
> Which is exactly Baby X.

CMake may be the best choice here - I don't contend that. The point of
disagreement is the myth that make is claimed to be so hard to use that
it must be avoided.

>>
>> So if you are part of a development team, and your team is rejecting
>> "make" on the basis of "it's too hard to understand", your development
>> team is in a /lot/ of trouble. New management - or at least training
>> for the management - should be number one priority until the job is run
>> by people that understand about using the best tools for the job, and
>> training people in the jobs that need done. (Of course, if CMake really
>> is a better choice for your project, that's fine - make is not the only
>> tool in the box.)
>>
> I don't think you understand how you come over sometimes.
> No one at work wants to use make for managing resources. You don't know
> better than we do how to set up our development process.

Read again what I wrote. If CMake is the right tool for your group,
great. But if something else is the right tool - make or anything else
- and your team is avoiding it because no one knows how to work it, then
you have a /big/ problem. Project management involves investing in tool
training and tool purchase (though in this case, the cost price is
zero), balanced against the usefulness of the tool, how much time it
saves in the future, how much easier it makes the work for the
development team, and - perhaps most importantly - how it affects
quality and reduces the risk of mistakes. I am not saying that "make"
is the best build tool for your needs, because I don't know your needs -
but I /am/ saying that if your team is not using the best build tool
merely "because it is hard", you development processes and management
need a good shake-up.

(I have looked at CMake for our development uses - it is not suitable
and can't do the job.)

>>
>>> It works.
>> Your evidence for that is ... what? You need users and feedback to
>> establish that. And maybe your XML is turning away potential users - it
>> would turn me away, certainly.
>>
> I've written a test program which internationises a string. So the system works
> from a narrow technical point of view. The question is about the softer
> considerations of what people actually want to use.

Yes.

>>> The question is whether it's a good approach to the problem.
>> Indeed. (My criticism here is intended as constructive, to suggest
>> better choices.)
>>> As I said, the user of Baby X will be mostly hobby programmers.
>> If your description of it is accurate, why are you limiting it like
>> that? Many professional developers would like better simple
>> cross-platform gui tools and toolkits. And if it is not better than
>> existing options, why bother with it at all? The only time it makes
>> sense to target hobby users alone is if the existing alternatives are
>> hugely complex, or hugely expensive, and that is not the case here.
>> Professionals also like "easy to use".
>>
> We use Qt for one of our products. Not one that I work on personally.
> But one of the developers told me that he had had a 12 hour build.
> Of course the Qt product is super polished, and he is being paid for
> those twelve hours. So it's worth it for us. But for hobby programming
> it's probably not. Baby X will build in a few seconds.

Qt is /big/. A 12 hour build sounds excessive (or perhaps you need new
build servers), but long build times are not uncommon for large Qt
projects. Presumably for this product, you are using a lot of Qt
features, and need that toolkit. But for many professional projects,
you don't need much - if Baby X has all that's needed, and is easy to
use, then I would expect professional developers would pick it over Qt
when a lighter toolkit is sufficient.

>>> So the
>>> emphasis is on making things easy to use. Adding alternative translations
>>> under an <international> tag is easy to do.
>>>
>> No, the translation system is /not/ easy to use, and horrendous to maintain.
>>
>> Look, imagine a developer wanted French and German translations. It's
>> just a single programmer, but he has a couple of friends who speak those
>> languages. With your solution, these two friends are mixing and editing
>> the same file, at the same time, while the programmer is simultaneously
>> modifying the file for other purposes. It is absolute madness. Do you
>> not understand that the French translations need to go in one file, and
>> the German translations need to go in a different file, and there is
>> /no/ circumstances in which it is a good idea to mix them all together
>> in one horrible XML file?
>>
> Yes but think of someone who lives inFrance, and speaks French as his native
> language and English as a second language, but no others. He'll like the idea
> that he can toggle his Baby X program between English and French. But he
> can't afford to send off the strings to a professional translator to get in
> other languages. For him, the system as it stands is ideal (I think, I'm open
> to suggestions).

On 09/08/2023 14:35, Michael S wrote:
> On Wednesday, August 9, 2023 at 2:43:06 PM UTC+3, David Brown wrote:

>> But if you have an external manually written makefile, that is not
>> project management - so usually you also then have to make an IDE
>> project to describe the source files, include search paths, pre-defined
>> macros, etc. That can be a duplicate of work (though good makefiles
>> find the source files and dependencies mostly automatically), and
>> duplication can lead to mistakes.
>>
>
> No, simply no.
> Good makefile finds "other" dependencies mostly automatically, but
> makes no attempts to figure out source files.
>

My makefiles do.

Obviously this is in combination with an appropriate layout of the
files. Basically, all my source files will be in my project's "source"
directory and subdirectories thereof, and nothing else is there. So
finding all the source files is just a recursive wildcard operation.
Any time I add or remove files to the project, the build is adjusted
automatically. It is all incredibly easy to use, and very hard to get
wrong. (Writing the makefile originally took some time.)

Of course, if you have a different way of organising your source files,
you may not have appropriate rules that can be put in your makefile to
let make find them automatically.

On 09/08/2023 17:01, Bart wrote:
>
> Where does 'nmake' come from? The cmakelists.txt file doesn't mention
> it. (And copying make.exe to nmake.exe gives other errors. Is it even
> supposed to run make? You said it only generated input for it.)
>
> Anyway I won't waste any more time with it. This is not even as bad as
> using a sledgehammer to crack a nut; it can't even find the sledgehammer
> after waiting weeks for it to arrive!
>
> Doubtless DB will stay I'm just incompetent. I don't think it's me
> that's incompetent.

CMake generates control files for other build systems, such as "make"
and MSVC project files. I have no idea how it is set up by default (I
have never tried it on Windows, and only looked at it enough to see it
can't do what I need). But since "nmake" is Microsoft's sort-of-make
utility that has shipped with every Microsoft development tool since
their first assembler for MSDOS, I'm guessing that CMake has figured out
you are running on Windows, and is trying to generate an nmake
compatible makefile and use nmake on it. Maybe CMake does not support
any other compilers or tools on Windows than MSVC, or maybe that depends
on the CMakeLists.txt file.

On 09/08/2023 19:18, Kaz Kylheku wrote:
> On 2023-08-09, David Brown <david.brown@hesbynett.no> wrote:
>> On 09/08/2023 01:24, Keith Thompson wrote:
>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>> [...]
>>>> You can't really support Unicode but not support internationalisation,
>>>> however.
>>> [...]
>>>
>>> An application could show all its application-defined messages in
>>> English, but still handle and display arbitrary text in any language.
>>> A word processor that uses English for all its messages and commands
>>> can still be used to write a novel in French.
>>>
>>
>> And some words in English need non-ASCII characters to be spelt
>> correctly, such as naïve or café. People's names might have non-ASCII
>
> Nope; the correct spelling of "naive" is exactly that, and that's
> how it appears in dictionaries.
>
> Only a few lunatic, like at the New Yorker, insist on
> spellings like "coöperation".

I am quite happy to accept "naive", but I prefer "naïve". Modern usage,
unsurprisingly, has tended towards the unaccented version since it is
much easier for most English-language keyboards. Writing it as "naïve"
is, however, still established practice.

For "cooperation", however, the diaeresis fell out of fashion long
before computer keyboards were common. (Good typewriters could handle
various accents well enough, using over-typed accents.) The spelling
"co-operation" has always been more common, though modern usage often
drops the hyphen.

The key difference between these words is that "naïve" comes from the
French (and arguably should be written in italics), where the diaeresis
is an essential aspect of the correct spelling. "Cooperation", on the
other hand, is a synthesis of two words, and the diaeresis was sometimes
added to the English spelling to clarify the pronunciation - something
that can be achieved more easily by the hyphen.

As a point of reference, the English language dictionary that I am using
for my newsreader has "naive", "naïve", "cooperate", and "co-operate",
but not "coöperate".

So in my highly biased opinion, writing "naïve" shows a care for detail,
an interest in typography bordering on the obsessive, a pedantry rarely
found outside comp.lang.c, and a penchant for showing off more flexible
keyboard setups than unlucky Windows users. Writing "coöperation", on
the other hand, is the result of trying so hard to look cultured that
you actually look foolish.

>
> I can't find clear references on this topic, but somehow, I seem to be
> carrying an old impression from somewhere that the use of the diaeresis
> for native words is just something that became fashinable in the 1800's.
>
> I.e. it's not simply archaic, but a revival of a passing Victorian
> era fad.
>

Yes, I think that is correct. But "naïve" is not a native English word
- it came from the French, complete with accent.

On 13/08/2023 09:07, Kaz Kylheku wrote:
> On 2023-08-13, David Brown <david.brown@hesbynett.no> wrote:
>> On 12/08/2023 11:58, Malcolm McLean wrote:
>>> On Saturday, 12 August 2023 at 09:36:35 UTC+1, David Brown wrote:
>>>> define compile_rule_template
>>>> # $(1) = directory
>>>> # $(2) = extra common flags
>>>> # $(3) = extra c flags
>>>> # $(4) = extra cpp flags
>>>>
>>>> # First, rules to generate target directories as needed
>>>> $(dep_dir)$(1) $(obj_dir)$(1) $(lst_dir)$(1) :
>>>> $(MKDIR) -p $$@
>>>>
>>>> # Generate dependancy files
>>>> $(dep_dir)$(1)%.d : $(1)%.c $(alldepends) | $(dep_dir)$(1)
>>>> @echo Generating depency file $$@
>>>> $(CCDEP) $(combined_cflags) $(2) $(3) -MM -MP \
>>>> -MT $(obj_dir)$(1)$$(@F:%.d=%.o) -MT \
>>>> $(dep_dir)$(1)$$(@F:%.o=%.o) \
>>>> -MF $$@ $$<
>>>>
>>>>
>>>>
>>>> I'll let you look up the gcc manual and make manual for the details of
>>>> how this all works. It generates the dependency files automatically,
>>>> and updates them automatically. Obviously you need to set the variables
>>>> referenced, so that your object files and dependency files are in
>>>> separate trees from the source files.
>>>>
>>>> I was a little reluctant to post this because Bart will get his knickers
>>>> in a twist over it, but you might find it useful or inspirational.
>>>>
>>> As professional software engineers we ought to be able to do better
>>> than this sort of thing.
>>>
>>
>> Possibly. But most of the attempts, such as CMake, have not been as
>> good. They have tried to have a less cryptic syntax, but either fail to
>> provide the features needed, or simply replace cryptic symbols with
>> cryptic words.
>
> I've learned something from your Makefile fragment there.

I'm glad to hear that - it's rare to see someone write "I've learned
something" on Usenet!

> And in turn, I have a potentially nice idea; see below.
>
> It looks like your rules might be solving the problem when header files
> disappear from dependencies.
>
> You used the -MT option twice, in order to insert the .d file itself
> as a target into the .d file, so you have this in a "whatever.d"
> file:
>
> whatever.o: whatever.c header.h ...
>
> whatever.d: whatever.c header.h ...
>
> Thus if any of the files whatever.c header.h .. change, not onlly
> is the .o out of date, but so is the .d.
>
> Now the .d is a Makefile fragment that is included.
>
> When GNU Make finishes reading the Makefile and all included fragments,
> it considers them as targets to be updated (before doing anything else).
> So at that point, any out of date .d files will get re-made by your
> dependency generating rule, and everything gets reloaded.
>
> Then the .d files are up-to-date and compiling proceeds with
> the correct dependencies.
>
> When you generate .d files as byproduct of making .o files, and don't
> have a rule to generate them separately, then a code change that removes
> a header breaks the build.

Yes.

I don't make the .d files as a byproduct of the .o files, I build them
separately. So all the dependency files get built first, before the
real compilation.

(Part of the reason I do this is because I use the same makefiles, with
a modifications, on many projects. I have some that use non-gcc
compilers for the actual compilation, but I still use gcc to generate
the dependency files. Another reason is that before I realised you
could have two -MT options I was running the dependency generation twice.)

>
> If we remove "header.h", the .d file refers to a nonexistent
> prerequisite and make complains that it has no way to build header.h,
> which is needed by whatever.o.
>
> Now here is the idea. I think we can stick with the more efficient
> system where we generate the .d as a side effect of compiling the .o
> in a single compiler invocation.
>
> We have a rule which updates .d files, but the body does this:
>
> $(dep_dir)$(1)%.d : $(1)%.c $(alldepends) | $(dep_dir)$(1)
> > $@
> touch $<
>
> The .d file is truncated to zero length, and its time stamp is touched,
> and the .c file (first prerequiste $<) is touched, to force the .o
> to rebuild.

I am not keen on touching the source file - I prefer the timestamps on
the source code to match when it was actually saved. But I appreciate
the idea.

>
> I think that, then, in the .d file, all we need is the .d dependencies:
>
> whatever.d: whatever.c whatever.h ...
>
> We don't need the .o. Whenever whatever.c or whatever.h or other files
> change, the .d file itself is out of date, and so the update rule
> will file, blowing away that file, and touching the .c file.
> So the .c file is forced to rebuild, and in so doing it will
> regenerate the .d file.
>
> I will experiment with this to see what, if any, unforseen stumbling
> blocks come up.
>

On 14/08/2023 18:06, Kaz Kylheku wrote:
> On 2023-08-14, Scott Lurndal <scott@slp53.sl.home> wrote:
>> Kaz Kylheku <864-117-4973@kylheku.com> writes:
>>> On 2023-08-13, Scott Lurndal <scott@slp53.sl.home> wrote:
>>>> Bart <bc@freeuk.com> writes:
>>>>> On 13/08/2023 17:02, Scott Lurndal wrote:
>>>>>> David Brown <david.brown@hesbynett.no> writes:
>>>>>>> On 12/08/2023 13:12, Bart wrote:
>>>>>>
>>>>>>>
>>>>>>>> Suppose you dispensed with all that dependency stuff; how long would it
>>>>>>>> take to build such a project? Say compared with just compiling one module.
>>>>>>>>
>>>>>>>
>>>>>>> That particular project takes a minute and a half to compile on my
>>>>>>> laptop (it's faster on my main machine). It's 373 compilations, perhaps
>>>>>>> 40% of it C++. Too long for convenience.
>>>>>>
>>>>>> My current project takes well over an hour to build on a single-core
>>>>>> system. With 32 cores and parallel make (-j32), that's reduced to
>>>>>> less than ten minutes
>>>>>
>>>>> You don't get a 32x speedup then?
>>>>
>>>> Compiler workloads are generally I/O bound.
>>>
>>> Not really. E.g. if you use ccache, and everything is a hit,
>>> it's "lightning fast" just to copy the .o files from the cache.
>>
>> We've used ccache, and had enough problems with it that we
>> abandoned it.
>
> For my purposes, I'm not running into the problem; ccache
> is working.
>
> Subtly broken or not, for the present purposes, it shows that two orders
> of magnitude of run-time can get saved by hashing the translation unit
> and just copying canned .o files.

I use ccache myself, and have not seen any problems with it. (I disable
it when doing timings posted here.)

>
>> Using dependencies means that only files with changes get
>> rebuilt, and that's sufficient.
>>
>>
>>>
>>> If there is a top Makefile which has recipes that sequentially
>>> walk through subdirectories and call $(MAKE)
>>
>> Why would one write a makefile to serialize the subdirectories?
>>
>> The should be (and in our case are) fully independent.
>
> I wouldn't myself, but such recursive make systems are out there
> in the wild.
>

There's a famous article somewhere titled "Recursive make considered
harmful". I just make my binary depend on the object files matching
every source file in all the subdirectories, and let make sort it all
out (in parallel, though I don't have as many cores as Scott).

On 12/08/2023 01:25, Kalevi Kolttonen wrote:

> The usages of Make are not limited to programming languages
> like C. It is a pretty general dependency tracking system
> and I have also used it for tasks like automating the
> building of LaTeX documents.
>

I do that too. Sometimes this also includes rules for converting images
from a source format to a LaTeX-friendly format, or running Python
scripts or other programs. Pretty much any task that can be described
as a directed acyclic graph of dependencies is a candidate for make.

On 11/08/2023 23:46, Bart wrote:

> K&R2 page 6 explains how to compile hello.c:
>
>     cc hello.c
>
> I want to keep that simplicity, even for much larger projects.
>

If you can keep that simplicity for your programs, fine.

Other people and other projects have more advanced needs.

My compile lines can be over 2500 characters long. Even if I omitted
the bits that I merely /want/, rather than /need/, such as warning
flags, they would be some 1500 characters. With make, I have a really
simple and clear way to keep track of all the flags, include
directories, microcontroller selection options, and everything else I
need. It's all in nice text files with comments and information - I
don't want to have to keep looking up the right choice of floating point
support flags for the particular microcontroller every time I want to do
a compilation - typing "make" is sufficient.

On 11/08/2023 18:26, Bart wrote:
> On 11/08/2023 16:39, Scott Lurndal wrote:
>> Bart <bc@freeuk.com> writes:
>>> On 11/08/2023 14:32, Scott Lurndal wrote:
>
>>>> Perhaps it is time to simply stop responding to Bart when he continues
>>>> to make such silly statements.
>>>
>>> What exactly is silly about it? Isn't that what is being said or not?
>>
>> Nobody has told you what you're allowed to have in your directories.
>>
>
> So I must have misunderstood these comments:
>
> RH: Then you have too many unrelated things in that directory.
>
> BC: That happens.
>
> SL: No, it doesn't.  Except, perhaps to you.
>
> DB: Put the files of a project in a directory.  Delete the junk.
> [following a sarcastic remark]
>
>

Unless you consider me to be your boss, then that is clearly advice in a
better way to organise your files to make things easier for yourself. I
am not "forcing" you to do anything - it's all up to /you/.

So yes, you misunderstood. As so often happens, I suspect the
misunderstanding is entirely intentional.

On 13/08/2023 18:02, Scott Lurndal wrote:
> David Brown <david.brown@hesbynett.no> writes:
>> On 11/08/2023 12:17, Bart wrote:
>>> On 11/08/2023 07:43, David Brown wrote:
>
>>> Look, just do things your way, and let others use their own approaches.
>>> I've actually been doing this stuff for longer than you.
>>>
>>
>> Longer does not mean better. Often it means set in your ways, and
>> unwilling to look at alternatives.
>
> And I seriously doubt the "longer than you" bit.

Bart /has/ been programming for longer than me - I've only been at it
for a little over 40 years.

On 10/08/2023 17:18, Ben Bacarisse wrote:
> scott@slp53.sl.home (Scott Lurndal) writes:
>
>> Ben Bacarisse <ben.usenet@bsb.me.uk> writes:
>>> Bart <bc@freeuk.com> writes:
>>
>>>> If working with Linux and you need -lm, then 'make prog' won't work. That's
>>>> just a fact.
>>>
>>> 2+2 = 4. That's just a fact.
>>
>> Indeed, and it is a fact for _any_ library containing optional functionality
>> that is needed by the application.
>
> That's only true in a circular way. Why are the math functions
> considered optional? Why not, for example, consider the IO functions
> optional so that any program that includes <stdio.h> has to link with
> -lio? (I know the history -- that's not what I'm asking.)
>
> There's even a division that could be justified by the standard. Any
> program that uses functions not required in a freestanding
> implementation could be obliged to link with -lhosted, but requiring -lm
> is just a historical barnacle.
>

I believe the "-lm" thing is, as you say, just historical. C libraries
were split to reduce their size, and since the maths functions were
often unneeded but often depend on each other, that seemed a natural
split. For many C libraries these days, libm.a is actually empty - it
exists solely so that "-lm" finds something if you use it.

On 15/08/2023 13:45, David Brown wrote:
> On 10/08/2023 17:18, Ben Bacarisse wrote:
>> scott@slp53.sl.home (Scott Lurndal) writes:
>>
>>> Ben Bacarisse <ben.usenet@bsb.me.uk> writes:
>>>> Bart <bc@freeuk.com> writes:
>>>
>>>>> If working with Linux and you need -lm, then 'make prog' won't
>>>>> work. That's
>>>>> just a fact.
>>>>
>>>> 2+2 = 4.  That's just a fact.
>>>
>>> Indeed, and it is a fact for _any_ library containing optional
>>> functionality
>>> that is needed by the application.
>>
>> That's only true in a circular way.  Why are the math functions
>> considered optional?  Why not, for example, consider the IO functions
>> optional so that any program that includes <stdio.h> has to link with
>> -lio?  (I know the history -- that's not what I'm asking.)
>>
>> There's even a division that could be justified by the standard.  Any
>> program that uses functions not required in a freestanding
>> implementation could be obliged to link with -lhosted, but requiring -lm
>> is just a historical barnacle.
>>
>
> I believe the "-lm" thing is, as you say, just historical.  C libraries
> were split to reduce their size, and since the maths functions were
> often unneeded but often depend on each other, that seemed a natural
> split.  For many C libraries these days, libm.a is actually empty - it
> exists solely so that "-lm" finds something if you use it.

A test with dlopen/dlsym managed to find "sqrt" only from "libm.so.6".
It coulldn't find it using "libc.so.6".

But I couldn't find either of those actual files on WSL Ubuntu, only
something that corresponded to the latter.

On Tuesday, 15 August 2023 at 13:45:36 UTC+1, David Brown wrote:
> On 10/08/2023 17:18, Ben Bacarisse wrote:
> > sc...@slp53.sl.home (Scott Lurndal) writes:
> >
> >> Ben Bacarisse <ben.u...@bsb.me.uk> writes:
> >>> Bart <b...@freeuk.com> writes:
> >>
> >>>> If working with Linux and you need -lm, then 'make prog' won't work. That's
> >>>> just a fact.
> >>>
> >>> 2+2 = 4. That's just a fact.
> >>
> >> Indeed, and it is a fact for _any_ library containing optional functionality
> >> that is needed by the application.
> >
> > That's only true in a circular way. Why are the math functions
> > considered optional? Why not, for example, consider the IO functions
> > optional so that any program that includes <stdio.h> has to link with
> > -lio? (I know the history -- that's not what I'm asking.)
> >
> > There's even a division that could be justified by the standard. Any
> > program that uses functions not required in a freestanding
> > implementation could be obliged to link with -lhosted, but requiring -lm
> > is just a historical barnacle.
> >
>
> I believe the "-lm" thing is, as you say, just historical. C libraries
> were split to reduce their size, and since the maths functions were
> often unneeded but often depend on each other, that seemed a natural
> split. For many C libraries these days, libm.a is actually empty - it
> exists solely so that "-lm" finds something if you use it.
>
Most big processors have a hardware square root function. So I would hope
compilers would inline calls rather than use a library.

On 15/08/2023 15:26, Malcolm McLean wrote:
> On Tuesday, 15 August 2023 at 13:45:36 UTC+1, David Brown wrote:
>> On 10/08/2023 17:18, Ben Bacarisse wrote:
>>> sc...@slp53.sl.home (Scott Lurndal) writes:
>>>
>>>> Ben Bacarisse <ben.u...@bsb.me.uk> writes:
>>>>> Bart <b...@freeuk.com> writes:
>>>>
>>>>>> If working with Linux and you need -lm, then 'make prog' won't work. That's
>>>>>> just a fact.
>>>>>
>>>>> 2+2 = 4. That's just a fact.
>>>>
>>>> Indeed, and it is a fact for _any_ library containing optional functionality
>>>> that is needed by the application.
>>>
>>> That's only true in a circular way. Why are the math functions
>>> considered optional? Why not, for example, consider the IO functions
>>> optional so that any program that includes <stdio.h> has to link with
>>> -lio? (I know the history -- that's not what I'm asking.)
>>>
>>> There's even a division that could be justified by the standard. Any
>>> program that uses functions not required in a freestanding
>>> implementation could be obliged to link with -lhosted, but requiring -lm
>>> is just a historical barnacle.
>>>
>>
>> I believe the "-lm" thing is, as you say, just historical. C libraries
>> were split to reduce their size, and since the maths functions were
>> often unneeded but often depend on each other, that seemed a natural
>> split. For many C libraries these days, libm.a is actually empty - it
>> exists solely so that "-lm" finds something if you use it.
>>
> Most big processors have a hardware square root function. So I would hope
> compilers would inline calls rather than use a library.

On many "mid level" processors these square root functions are not fully
IEEE compliant, and will not be used by default. The library function
will check for negative values, NaNs, etc., before using the hardware
instruction. If you know all your calculations are using nice real
numbers, you can use "-ffast-math" on gcc or clang to get more efficient
code.

But that all depends on the target processor, the compiler, and the
flags. Functions like "sqrt" will exist in the library regardless of
how well the compiler optimises.

David Brown <david.brown@hesbynett.no> writes:
>On 14/08/2023 18:06, Kaz Kylheku wrote:
>> On 2023-08-14, Scott Lurndal <scott@slp53.sl.home> wrote:
>>> Kaz Kylheku <864-117-4973@kylheku.com> writes:
>>>> On 2023-08-13, Scott Lurndal <scott@slp53.sl.home> wrote:
>>>>> Bart <bc@freeuk.com> writes:
>>>>>> On 13/08/2023 17:02, Scott Lurndal wrote:
>>>>>>> David Brown <david.brown@hesbynett.no> writes:
>>>>>>>> On 12/08/2023 13:12, Bart wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>> Suppose you dispensed with all that dependency stuff; how long would it
>>>>>>>>> take to build such a project? Say compared with just compiling one module.
>>>>>>>>>
>>>>>>>>
>>>>>>>> That particular project takes a minute and a half to compile on my
>>>>>>>> laptop (it's faster on my main machine). It's 373 compilations, perhaps
>>>>>>>> 40% of it C++. Too long for convenience.
>>>>>>>
>>>>>>> My current project takes well over an hour to build on a single-core
>>>>>>> system. With 32 cores and parallel make (-j32), that's reduced to
>>>>>>> less than ten minutes
>>>>>>
>>>>>> You don't get a 32x speedup then?
>>>>>
>>>>> Compiler workloads are generally I/O bound.
>>>>
>>>> Not really. E.g. if you use ccache, and everything is a hit,
>>>> it's "lightning fast" just to copy the .o files from the cache.
>>>
>>> We've used ccache, and had enough problems with it that we
>>> abandoned it.
>>
>> For my purposes, I'm not running into the problem; ccache
>> is working.
>>
>> Subtly broken or not, for the present purposes, it shows that two orders
>> of magnitude of run-time can get saved by hashing the translation unit
>> and just copying canned .o files.
>
>I use ccache myself, and have not seen any problems with it. (I disable
>it when doing timings posted here.)

The issues we saw were related to different users compiling with
different compilation flags or optimization levels (the Ccache was
shared by a hundred users). For the cases where the users were
simply compiling the code it worked fine. For the developers actually
developing the code, it caused issues when the developers modified
debugging defines or the optimization level and tried to link with
cached objects with different configurations.

In any case, we use dependency checking which limits recompilation
to the files made obsolete by changes to source or header files.

>There's a famous article somewhere titled "Recursive make considered
>harmful".

I've read it many times and found it not very compelling. Even
linux uses recursive make sucessfully, and pretty much every project
I've worked on over the last four decades (from Unixware to my
current large project) use recursive makefiles without running
into the issues in the article.

> I just make my binary depend on the object files matching
>every source file in all the subdirectories, and let make sort it all
>out (in parallel, though I don't have as many cores as Scott).

It's a really nice two socket xeon system, to be sure :-).

David Brown <david.brown@hesbynett.no> writes:
>On 13/08/2023 18:02, Scott Lurndal wrote:
>> David Brown <david.brown@hesbynett.no> writes:
>>> On 11/08/2023 12:17, Bart wrote:
>>>> On 11/08/2023 07:43, David Brown wrote:
>>
>>>> Look, just do things your way, and let others use their own approaches.
>>>> I've actually been doing this stuff for longer than you.
>>>>
>>>
>>> Longer does not mean better. Often it means set in your ways, and
>>> unwilling to look at alternatives.
>>
>> And I seriously doubt the "longer than you" bit.
>
>Bart /has/ been programming for longer than me - I've only been at it
>for a little over 40 years.
>

Ah, 47 for me. I don't recall Bart posting his history.