On 05/02/2024 23:34, Lawrence D'Oliveiro wrote:
> On Mon, 5 Feb 2024 15:17:15 +0000, Malcolm McLean wrote:
>
>> It would be easier if the CMake scripts were in
>> C. It might be bit less convenient.
>
> Aren’t “easier” and “less convenient” kind of ... opposites?

Less convenient for the whizzy suoer skilled build system programmeer to
write. Easier for the humble C programmer who has it fall over on him to
fix.

--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On Tue, 6 Feb 2024 01:41:56 +0000, Malcolm McLean wrote:

> On 05/02/2024 23:34, Lawrence D'Oliveiro wrote:
>
>> On Mon, 5 Feb 2024 15:17:15 +0000, Malcolm McLean wrote:
>>
>>> It would be easier if the CMake scripts were in C. It might be bit
>>> less convenient.
>>
>> Aren’t “easier” and “less convenient” kind of ... opposites?
>
> Less convenient for the whizzy suoer skilled build system programmeer to
> write. Easier for the humble C programmer who has it fall over on him to
> fix.

But the C code will be anything up to an order of magnitude larger than
the build rules written in the domain-specific language. So more work
required overall.

On 06/02/2024 02:27, Lawrence D'Oliveiro wrote:
> On Tue, 6 Feb 2024 01:41:56 +0000, Malcolm McLean wrote:
>
>> On 05/02/2024 23:34, Lawrence D'Oliveiro wrote:
>>
>>> On Mon, 5 Feb 2024 15:17:15 +0000, Malcolm McLean wrote:
>>>
>>>> It would be easier if the CMake scripts were in C. It might be bit
>>>> less convenient.
>>>
>>> Aren’t “easier” and “less convenient” kind of ... opposites?
>>
>> Less convenient for the whizzy suoer skilled build system programmeer to
>> write. Easier for the humble C programmer who has it fall over on him to
>> fix.
>
> But the C code will be anything up to an order of magnitude larger than
> the build rules written in the domain-specific language. So more work
> required overall.

Well yes. If the program to build the program has to be more elaborate
than the porgram itself, something is going wrong. But whilst the CMake
scripts are quite short, they aren't trivial, and there's a lot of
fussing about needed to pull in the sources, set the IDE uo in the right
way, and then make sure that the non-IDE build is exactly the same. But
if it falls over I can't work, and you don't like to call in someone
else unless it's established that the script itself actually needs
modification. So you tend to trouble shoot. But whilst I do write my own
CMkae scripts, I don't write them often enough to really know the
language, and I just forget all the details. So troubleshooting a CMake
scriot is quite hard work. Not as hard as debugging a complex C
algorithm. But that's my core job, whilst the build system is not.

--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On Tue, 6 Feb 2024 02:59:11 +0000, Malcolm McLean wrote:

> But whilst the CMake scripts are quite short, they aren't trivial ...

I know, I’ve fiddled with CMake scripts myself. The power is needed
because builds can be complex.

> But whilst I do write my own CMkae scripts, I don't write them often
> enough to really know the language ...

Same here. That’s why, every time I need to touch a CMake script, I make
sure to have the docs open in a browser window
<https://cmake.org/documentation/>. I don’t fiddle at random: I try to
understand what I am doing. And sometimes I find the person who fiddled
the script before me didn’t understand that there was a simpler way to do
something.

On 2/5/2024 6:27 PM, Lawrence D'Oliveiro wrote:
> On Tue, 6 Feb 2024 01:41:56 +0000, Malcolm McLean wrote:
>
>> On 05/02/2024 23:34, Lawrence D'Oliveiro wrote:
>>
>>> On Mon, 5 Feb 2024 15:17:15 +0000, Malcolm McLean wrote:
>>>
>>>> It would be easier if the CMake scripts were in C. It might be bit
>>>> less convenient.
>>>
>>> Aren’t “easier” and “less convenient” kind of ... opposites?
>>
>> Less convenient for the whizzy suoer skilled build system programmeer to
>> write. Easier for the humble C programmer who has it fall over on him to
>> fix.
>
> But the C code will be anything up to an order of magnitude larger than
> the build rules written in the domain-specific language. So more work
> required overall.

Putting the build rules in source code, is a bad idea, TM?

On 2/5/2024 7:09 PM, Lawrence D'Oliveiro wrote:
> On Tue, 6 Feb 2024 02:59:11 +0000, Malcolm McLean wrote:
>
>> But whilst the CMake scripts are quite short, they aren't trivial ...
>
> I know, I’ve fiddled with CMake scripts myself. The power is needed
> because builds can be complex.
>
>> But whilst I do write my own CMkae scripts, I don't write them often
>> enough to really know the language ...
>
> Same here. That’s why, every time I need to touch a CMake script, I make
> sure to have the docs open in a browser window
> <https://cmake.org/documentation/>.

Nothing wrong with that! :^)

I don’t fiddle at random: I try to
> understand what I am doing. And sometimes I find the person who fiddled
> the script before me didn’t understand that there was a simpler way to do
> something.

On Mon, 5 Feb 2024 22:32:40 -0800, Chris M. Thomasson wrote:

> On 2/5/2024 6:27 PM, Lawrence D'Oliveiro wrote:
>
>> But the C code will be anything up to an order of magnitude larger than
>> the build rules written in the domain-specific language. So more work
>> required overall.
>
> Putting the build rules in source code, is a bad idea, TM?

Build rules *are* source code.

On 2/5/2024 11:25 PM, Lawrence D'Oliveiro wrote:
> On Mon, 5 Feb 2024 22:32:40 -0800, Chris M. Thomasson wrote:
>
>> On 2/5/2024 6:27 PM, Lawrence D'Oliveiro wrote:
>>
>>> But the C code will be anything up to an order of magnitude larger than
>>> the build rules written in the domain-specific language. So more work
>>> required overall.
>>
>> Putting the build rules in source code, is a bad idea, TM?
>
> Build rules *are* source code.

Well, trying to put something that is akin to a makefile into source
code is a bad idea? Any better?

On 06/02/2024 07:33, Chris M. Thomasson wrote:
> On 2/5/2024 11:25 PM, Lawrence D'Oliveiro wrote:
>> On Mon, 5 Feb 2024 22:32:40 -0800, Chris M. Thomasson wrote:
>>
>>> On 2/5/2024 6:27 PM, Lawrence D'Oliveiro wrote:
>>>
>>>> But the C code will be anything up to an order of magnitude larger than
>>>> the build rules written in the domain-specific language. So more work
>>>> required overall.
>>>
>>> Putting the build rules in source code, is a bad idea, TM?
>>
>> Build rules *are* source code.
>
> Well, trying to put something that is akin to a makefile into source
> code is a bad idea? Any better?

CMake is a programming language in its own right and there's no
ambiguity about that, though it is special purpose one unsuitable to
write anything except build scripts. make is syntax for instructions to
a program which builds programs. There might be something in it which is
technically Turing compatible and thus can be used to create arbitrary
behaviour, but if true it will be difficult to do and abuse of the
intended purpose.

C is a general purpose programming language. However if you provide a
library of functions designed to help with building programs then it
becomes effectively a special purpose programming language, but still
with the general purpose grammar. It's still suitable for general
purpose programming and so less suitable for wriitng build scripts than
make, but the language is familar.

CMake and make both gained wide traction. Whilst apparently build
scripts using C as the scripting language have been tried before and
it's not a totally new idea, it's been tried out rather than seriously
tried.
--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On 05/02/2024 22:19, Malcolm McLean wrote:
> On 05/02/2024 19:47, David Brown wrote:
>> On 05/02/2024 16:17, Malcolm McLean wrote:
>>> On 05/02/2024 14:42, Ben Bacarisse wrote:
>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>>
>>
>>>> The term "scripting language" is so vague as to be almost useless, but
>>>> in this context the suggestion seems to be to use C as a language to
>>>> invoke the commands required to compile and link a C program.  But C is
>>>> very poorly suited to this task.  One would end up writing a library of
>>>> functions to do string manipulation and (unless 'system' was deemed
>>>> sufficient) program execution.  C itself would bring very little to the
>>>> party and the resulting scripts would be hard to read.
>>>>
>>> I'd dispute "very poorly". Of course you can devise a language which
>>> is better suited to building programs specifically. CMake takes
>>> exactly that approach. So what happens? It's effectively another
>>> programming language to learn. Someone else wrote elaborate CMakes
>>> scripts which I use at work. Sometimes thing go wrong. And then I'm
>>> messing about with a language I hardly ever use and does things in
>>> ways I am unfamilar with, trying to troubleshoot. It would be easier
>>> if the CMake scripts were in C. It might be bit less convenient. But
>>> it's just looping and branching and calling subroutines at the end of
>>> the day. As you know full well, that's all computers can do.
>>>
>>> The library would have facilities for working with lists of strings,
>>> of course. But C can do that perfectly well, and I don't see it as
>>> big problem.
>>>
>>
>> Have you ever tried /any/ other programming language?  Apart from
>> assembly and Forth, I have not seen a language that is more cumbersome
>> for string handling than C.  Yes, you /can/ do string handling in C,
>> but no one would /choose/ C for that.
>>
>> It's not unreasonable to suggest that you'd rather base your build
>> system on an existing mainstream language than a domain-specific
>> language (though DSL's have the big advantage of having a syntax and
>> semantics tuned to the task in question).  But why C?
>>
> The newgroup is comp.lang.c. So if we propose using a generally accepted
> and widely understood programming language as our build scripting
> language, the choice of language has to be C.

No, it does not. It is perfectly reasonable to say "not C".

>
>> Here's a quick challenge for you.
>>
>> I have a directory "src".  I want to find all the .c files in "src".
>> I want to make a list of all these files, and a list of matching
>> object files to make in the "build/obj/src" directory.  For each file,
>> I want to call "gcc -c src/file.c -o build/obj/src/file.o".  I want to
>> do so in parallel, up to 8 commands at a time.  (Ignore any possible
>> runtime errors.)
>>
>>
>> With make, that would be (this is untested) :
>>
>>      default : all
>>      .PHONY all
>>      srcdir = "src"
>>      objdir = "build/obj"
>>
>>      srcfiles = $(wildcard $(srcdir)/*.c)
>>      objfiles_src = $(srcfiles:.c=.o)
>>      objfiles = $(addprefix $(objdir)/, \
>>          $(patsubst ../%,%,$(objfiles_src)))
>>
>>      all : objfiles
>>
>>      $(objdir)/src :
>>              mkdir -p $@
>>
>>      $(objdir)/%.o : %.c | $(objdir)/src
>>              gcc -c $< -o $@
>>
>> Run with "make -j 8".
>>
>>
>> With Python, you could have (untested) :
>>
>>      import glob
>>      import os
>>      import multiprocessing
>>      import subprocess
>>
>>      srcdir = "src"
>>      objdir = "build/obj"
>>
>>      srcfiles = glob.glob(srcdir + "/*.c")
>>      objfiles_src = [fn[:-2] + ".o" for fn in srcfiles]
>>      objfiles = [objdir + "/" + fn for fn in srcfiles]
>>
>>      os.mkdirs(objdir + "/" + srcdir, exist_ok = True)
>>
>>      with multiprocessing.Pool(5) as pool :
>>          pool.map(lambda src, obj : subprocess.call("gcc -c " +
>>                   src + " -o " + obj, shell = True),
>>                  zip(srcfiles, objfiles))
>>
>> Obviously much of this could be put in a reusable library, so that end
>> users don't have to know about process pools.  (And there are many
>> other ways to structure such code.)
>>
>>
>> Your task is to duplicate this in C.
>>
>>
>
> #include "builduils.h"
>
> int main(void)
> {
>    STRINGLIST *sources;
>    STRINGLIST *objfiles;
>    int i;
>
>
>    sources = glob("src/*.c");
>    objfiles = stringlist();
>    for (i = 0; i < stringlist_Nstrings(sources); i++)
>    {
>       char objname[1024];
>       snprintf(objname, 1024, "%s/%s", objdir,
> replaceextension(stringlist_get(sources, i), ".o"));
>       stringlist_add(objfiles, objname);
>    }
>    char outputdir[1024];
>    snprinf("%s/%s", 1024, objdir, srcdir);
>    mkdir(outputdir);
>
>    callparallel("gcc -c %s %s", srcfiles, objfiles);
> }
>
> OK now I've cheated a little bit by inventing buildutils library
> functions ad hoc.

Yes. Of course a practical solution would split the common code into a
library, and not expect people to re-write that all the time. But a key
point here is that in make and in Python - both well-established
languages and tools - this is already in place. But you are skipping
the hard bits. (Good luck implementing "callparallel" with a sane
interface that includes the call you have made here!)

We know it is /possible/ to do this kind of thing in C. We also know it
is ugly and fiddly compared to other languages that handle strings and
lists as first-class types (or at least have them as classes). Even
when you assume the existence of a library with every type and function
you want, you still need a lot of extra boilerplate code, you have
artificial limits, memory leaks, etc.

> But I haven't done so outrageously. And we're leaking
> memory. But likely we have many gigabytres, so who are about maybe 2K
> for a few strings. And yes, maybe we could reoplace snprintf() with
> something that creates an arbitrary length output.
>
> The Python is a bit shorter and more concise. But there's not that much
> in it.

On 05/02/2024 22:28, bart wrote:
> On 05/02/2024 19:47, David Brown wrote:
>> On 05/02/2024 16:17, Malcolm McLean wrote:
>
>>> The library would have facilities for working with lists of strings,
>>> of course. But C can do that perfectly well, and I don't see it as
>>> big problem.
>>>
>>
>> Have you ever tried /any/ other programming language?  Apart from
>> assembly and Forth, I have not seen a language that is more cumbersome
>> for string handling than C.  Yes, you /can/ do string handling in C,
>> but no one would /choose/ C for that.
>
> See the build.c demo elsewhere in the thread.
>
>
>> I have a directory "src".  I want to find all the .c files in "src".
>> I want to make a list of all these files, and a list of matching
>> object files to make in the "build/obj/src" directory.  For each file,
>> I want to call "gcc -c src/file.c -o build/obj/src/file.o".  I want to
>> do so in parallel, up to 8 commands at a time.  (Ignore any possible
>> runtime errors.)
>
>> With Python, you could have (untested) :
>>
>>      import glob
>>      import os
>>      import multiprocessing
>>      import subprocess
>>
>>      srcdir = "src"
>>      objdir = "build/obj"
>>
>>      srcfiles = glob.glob(srcdir + "/*.c")
>>      objfiles_src = [fn[:-2] + ".o" for fn in srcfiles]
>>      objfiles = [objdir + "/" + fn for fn in srcfiles]
>>
>>      os.mkdirs(objdir + "/" + srcdir, exist_ok = True)
>>
>>      with multiprocessing.Pool(5) as pool :
>>          pool.map(lambda src, obj : subprocess.call("gcc -c " +
>>                   src + " -o " + obj, shell = True),
>>                  zip(srcfiles, objfiles))
>>
>> Obviously much of this could be put in a reusable library, so that end
>> users don't have to know about process pools.  (And there are many
>> other ways to structure such code.)
>
> I had a go at expressing the Python version in my scripting language:
>
>     srcdir := "src/"
>     objdir := "build/obj/"+srcdir
>
>     srcfiles := dirlist(srcdir + "*.c")
>
>     objfiles_src := mapvs(changeext, srcfiles, "o")
>     objfiles := mapsv(+, objdir, objfiles_src)
>
>     createdir(objdir)
>
>     for i, sfile in srcfiles do
>         execcmd(sfprint("gcc -c # -o #", srcdir+sfile, objfiles[i]))
>     od
>
> A couple of issues: 'createdir' can only create one directory at a time;
> for a chain of them like a/b/c, I'd need to split it up and do it one by
> one. (To test this, I created it manually.)

Your scripting language is definitely an improvement over C for this
kind of thing. That is as I would expect - scripting languages tend to
be much better at working with strings and external commands.

>
> Also, I don't have any features for parallel executions. However
> 'execcmd' starts a process but then doesn't wait for it to complete. If
> I instead use 'system' (or my 'execwait'), then compiling all Lua .c
> files in ./src takes 6.5 seconds instead of 2.5 seconds.
>
> But this isn't the point of posting this. My interpreter can be
> expressed as a single C file. That allows you to use this kind of
> scripting, without adding any extra dependencies. You still only need a
> C compiler.
>
> (I tried your Python, but os.mkdir(dir) didn't work, even after fixing
> the name typo and removing the named argument. Bypassing that, there
> were all sorts of errors to do with pickle.py and 'unbounded' methods.
>

"os.mkdirs()" was not a typo. "mkdirs()" is a different function from
"mkdir()" - it can make multiple directories as needed. But I may have
had other errors.

> I believe your glob.glob routine returns filenames with path prepended
> (my dirlist doesn't).

Yes.

> I suspect that's why you chose a dest path as
> build/obj/src rather than build/obj.

No. I used that because it is the structure I use for bigger projects.
I generally have multiple source directories, and in my build
directories I have "obj", "deps" (for dependency files), perhaps also
directories for listings files and other bits and pieces, and inside
these the directory structure mirrors the source tree. (I actually
often have another layer between "build" and "obj" because there can be
multiple different builds for the same project.)

> Anyway, I wasn't able to compare
> peformance.)

There was no intention of that. I merely wanted to show that something
that is key to software building can be done relatively easily in a
higher-level language suitable for scripting, in a way that is simpler
and easier than in C.

It was not a full build in any sense, and did not cover dependency graphs.

>
> BTW this scripting is still hard work. For building you want to express
> the requires as data, not code. If you can do that (see my build.c
> example), then C may be adequate.
>

Eventually, when you add all the features and flexibility people want,
you end up with a domain-specific language as you data, interpreted by
code written in some other language (C or whatever - though again I
expect other languages to be more suitable). You've just re-invented
"make".

Of course it is possible to re-invent make in a better way than the
original (at least for some people's preferences and needs - "better" is
always subjective). If I were doing so, I'd want an embedded high-level
scripting language of some sort that was not a variant of Lisp. (Some
people like Lisp, I don't.) Possible options from existing languages
would be Lua, TCL and Python. And I'd want it neatly integrated, not
just a way to make extensions, as that would simplify the main DSL.
(There's plenty of other things I'd change, but I don't want to go
through a list here.)

On 06/02/2024 00:38, Lawrence D'Oliveiro wrote:
> On Mon, 5 Feb 2024 21:28:20 +0000, bart wrote:
>
>> (I tried your Python, but os.mkdir(dir) didn't work, even after fixing
>> the name typo and removing the named argument.
>
> “mkdirs” should have been “makedirs”
> <https://docs.python.org/3/library/os.html#os.makedirs>.
>

Oh, so it was a typo - the "s" was right, but not the "mk" :-(

The code was just for reference, not tested or complete.