How shall we write a better build system?

Current systems use scripts with ad hoc, weird syntax. They don't work
consistently across platforms, they break and are hard to fix, and are
complicated to set up. You even get get one scripting system as a front
end to another, a sort of admission of failure.

But can we do better?

A scripting language provides loops, conditional branches, and
subroutine calls. And we've already got a perfecty good language for
describing that. It is C itself.

So here's the proposal. We implement a C interpreter. Then we provide a
library of facilities - obviously globbing and system calls, maybe also
installed program detection, as C functions called in the normal way.
And the resulting C program is the build script. Because the C languague
is well known and standardised, there's no unfamiliar syntax or
unexpected behaviour.

We've got the skills to do this. Bart writes compilers, and so can
easily modify one to run as an interpreter. And we can develop the
library using normal C compilers. When we're happy with it, it is folded
into the interpreter. A clean, closed, achieveable, but very flexible
design.

It will need to run on Linux, Mac and Windows at a minimum. And
hopefully build on all three with the same script and no platform
conditional code.

--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On 04/02/2024 15:39, Malcolm McLean wrote:
> How shall we write a better build system?
>
> Current systems use scripts with ad hoc, weird syntax. They don't work
> consistently across platforms, they break and are hard to fix, and are
> complicated to set up. You even get get one scripting system as a front
> end to another, a sort of admission of failure.
>
> But can we do better?
>
> A scripting language provides loops, conditional branches, and
> subroutine calls. And we've already got a perfecty good language for
> describing that. It is C itself.
>
> So here's the proposal. We implement a C interpreter.

Do we need an interpreter? We need to assume there is a C compiler
otherwise we won't get very far with the build!

If the concern is the time it takes to compile the script, it should be
a small proportion of that needed to build the app. I assume the script
is only built once.

C for scripting builds has been done before. I think Thiago Adams did so.

The Seed7 build process had its own version of an auto-config script
which was written in C, not Bash (but building the app still uses
makefiles, nearly 20 of them for assorted compilers).

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>How shall we write a better build system?
>
>Current systems use scripts with ad hoc, weird syntax.

Fact not in evidence. Your opinion thereof does not make it a fact.

bart <bc@freeuk.com> writes:
>On 04/02/2024 15:39, Malcolm McLean wrote:
>> How shall we write a better build system?
>>
>> Current systems use scripts with ad hoc, weird syntax. They don't work
>> consistently across platforms, they break and are hard to fix, and are
>> complicated to set up. You even get get one scripting system as a front
>> end to another, a sort of admission of failure.
>>
>> But can we do better?
>>
>> A scripting language provides loops, conditional branches, and
>> subroutine calls. And we've already got a perfecty good language for
>> describing that. It is C itself.
>>
>> So here's the proposal. We implement a C interpreter.

This was tried. It's called the c-shell.

On 04/02/2024 16:58, Scott Lurndal wrote:
> bart <bc@freeuk.com> writes:
>> On 04/02/2024 15:39, Malcolm McLean wrote:
>>> How shall we write a better build system?
>>>
>>> Current systems use scripts with ad hoc, weird syntax. They don't work
>>> consistently across platforms, they break and are hard to fix, and are
>>> complicated to set up. You even get get one scripting system as a front
>>> end to another, a sort of admission of failure.
>>>
>>> But can we do better?
>>>
>>> A scripting language provides loops, conditional branches, and
>>> subroutine calls. And we've already got a perfecty good language for
>>> describing that. It is C itself.
>>>
>>> So here's the proposal. We implement a C interpreter.
>
> This was tried. It's called the c-shell.

This is not a shell program. And it's not proposing using a language
that vaguely looks like C, but /is/ C. I don't think the proposal is to
ever type it live via a REPL, but always from a file.

Although as I understand it, what is proposed could be done right now:
anyone can provide a small C program to orchestrate the building of the
main program.

So perhaps it will be more of a library.

But it will take some care that it doesn't end up doing exactly what
makefiles do (with lots of assumptions about C compilers and the same
failure points), but written as a sequence of C function calls.

I think the main objection will be that it is a two step process. This
may have been the motivation to use an interpreter, but that is too huge
of a task, and too big a dependency, to solve a problem that could be
achieved with shebang lines.

On 04/02/2024 17:15, bart wrote:
> On 04/02/2024 16:58, Scott Lurndal wrote:
>> bart <bc@freeuk.com> writes:
>>> On 04/02/2024 15:39, Malcolm McLean wrote:
>>>> How shall we write a better build system?
>>>>
>>>> Current systems use scripts with ad hoc, weird syntax. They don't work
>>>> consistently across platforms, they break and are hard to fix, and are
>>>> complicated to set up. You even get get one scripting system as a front
>>>> end to another, a sort of admission of failure.
>>>>
>>>> But can we do better?
>>>>
>>>> A scripting language provides loops, conditional branches, and
>>>> subroutine calls. And we've already got a perfecty good language for
>>>> describing that. It is C itself.
>>>>
>>>> So here's the proposal. We implement a C interpreter.
>>
>> This was tried.  It's called the c-shell.
>
> This is not a shell program. And it's not proposing using a language
> that vaguely looks like C, but /is/ C. I don't think the proposal is to
> ever type it live via a REPL, but always from a file.
>
> Although as I understand it, what is proposed could be done right now:
> anyone can provide a small C program to orchestrate the building of the
> main program.
>
> So perhaps it will be more of a library.
>
> But it will take some care that it doesn't end up doing exactly what
> makefiles do (with lots of assumptions about C compilers and the same
> failure points), but written as a sequence of C function calls.
>
> I think the main objection will be that it is a two step process. This
> may have been the motivation to use an interpreter, but that is too huge
> of a task, and too big a dependency, to solve a problem that could be
> achieved with shebang lines.
>
tcc has an interpeter mode. So yes, you could put a tcc shebang and
achieve the same thing. But then the library wouldn't be integrated.
--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

scott@slp53.sl.home (Scott Lurndal) writes:

> bart <bc@freeuk.com> writes:
>>On 04/02/2024 15:39, Malcolm McLean wrote:
>>> How shall we write a better build system?
>>>
>>> Current systems use scripts with ad hoc, weird syntax. They don't work
>>> consistently across platforms, they break and are hard to fix, and are
>>> complicated to set up. You even get get one scripting system as a front
>>> end to another, a sort of admission of failure.
>>>
>>> But can we do better?
>>>
>>> A scripting language provides loops, conditional branches, and
>>> subroutine calls. And we've already got a perfecty good language for
>>> describing that. It is C itself.
>>>
>>> So here's the proposal. We implement a C interpreter.
>
> This was tried. It's called the c-shell.

csh isn't a C interpreter by any stretch of the imagination!

--
Ben.

On 04/02/2024 23:32, Ben Bacarisse wrote:
> scott@slp53.sl.home (Scott Lurndal) writes:
>
>> bart <bc@freeuk.com> writes:
>>> On 04/02/2024 15:39, Malcolm McLean wrote:
>>>> How shall we write a better build system?
>>>>
>>>> Current systems use scripts with ad hoc, weird syntax. They don't work
>>>> consistently across platforms, they break and are hard to fix, and are
>>>> complicated to set up. You even get get one scripting system as a front
>>>> end to another, a sort of admission of failure.
>>>>
>>>> But can we do better?
>>>>
>>>> A scripting language provides loops, conditional branches, and
>>>> subroutine calls. And we've already got a perfecty good language for
>>>> describing that. It is C itself.
>>>>
>>>> So here's the proposal. We implement a C interpreter.
>>
>> This was tried. It's called the c-shell.
>
> csh isn't a C interpreter by any stretch of the imagination!
>

No. But the concept of a C interpreter isn't new. There's one called
pico C which is open source and available to be incorporated. And a
scripting language is an obvious application for an intepreter.
--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

Ben Bacarisse <ben.usenet@bsb.me.uk> writes:

>> [...]
>
> csh isn't a C interpreter by any stretch of the imagination!

IT ISN'T??? No wonder I've been having trouble writing csh
scripts...

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:

> On 04/02/2024 23:32, Ben Bacarisse wrote:
>> scott@slp53.sl.home (Scott Lurndal) writes:
>>
>>> bart <bc@freeuk.com> writes:
>>>> On 04/02/2024 15:39, Malcolm McLean wrote:
>>>>> How shall we write a better build system?
>>>>>
>>>>> Current systems use scripts with ad hoc, weird syntax. They don't work
>>>>> consistently across platforms, they break and are hard to fix, and are
>>>>> complicated to set up. You even get get one scripting system as a front
>>>>> end to another, a sort of admission of failure.
>>>>>
>>>>> But can we do better?
>>>>>
>>>>> A scripting language provides loops, conditional branches, and
>>>>> subroutine calls. And we've already got a perfecty good language for
>>>>> describing that. It is C itself.
>>>>>
>>>>> So here's the proposal. We implement a C interpreter.
>>>
>>> This was tried. It's called the c-shell.
>>
>> csh isn't a C interpreter by any stretch of the imagination!
>
> No. But the concept of a C interpreter isn't new.

Indeed. But C is not a good fit for interpretation, so there are often
compromises.

> There's one called pico C

Yes, and there's tcc's -run option.

> which is open source and available to be incorporated.

Likewise tcc. So what's wrong (in your opinion) with these two that
caused you to suggest implementing another C interpreter?

> And a scripting
> language is an obvious application for an intepreter.

The term "scripting language" is so vague as to be almost useless, but
in this context the suggestion seems to be to use C as a language to
invoke the commands required to compile and link a C program. But C is
very poorly suited to this task. One would end up writing a library of
functions to do string manipulation and (unless 'system' was deemed
sufficient) program execution. C itself would bring very little to the
party and the resulting scripts would be hard to read.

--
Ben.

On 05/02/2024 14:42, Ben Bacarisse wrote:
> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>
>> On 04/02/2024 23:32, Ben Bacarisse wrote:
>>> scott@slp53.sl.home (Scott Lurndal) writes:
>>>
>>>> bart <bc@freeuk.com> writes:
>>>>> On 04/02/2024 15:39, Malcolm McLean wrote:
>>>>>> How shall we write a better build system?
>>>>>>
>>>>>> Current systems use scripts with ad hoc, weird syntax. They don't work
>>>>>> consistently across platforms, they break and are hard to fix, and are
>>>>>> complicated to set up. You even get get one scripting system as a front
>>>>>> end to another, a sort of admission of failure.
>>>>>>
>>>>>> But can we do better?
>>>>>>
>>>>>> A scripting language provides loops, conditional branches, and
>>>>>> subroutine calls. And we've already got a perfecty good language for
>>>>>> describing that. It is C itself.
>>>>>>
>>>>>> So here's the proposal. We implement a C interpreter.
>>>>
>>>> This was tried. It's called the c-shell.
>>>
>>> csh isn't a C interpreter by any stretch of the imagination!
>>
>> No. But the concept of a C interpreter isn't new.
>
> Indeed. But C is not a good fit for interpretation, so there are often
> compromises.
>
It's less of a problem now. The interpreter can just spin up a virtual
machine and then create a memory space. That was less of a practical
approach when machines were more limited.
>> There's one called pico C
>
> Yes, and there's tcc's -run option.
>
>> which is open source and available to be incorporated.
>
> Likewise tcc. So what's wrong (in your opinion) with these two that
> caused you to suggest implementing another C interpreter?
>
Bart writes compilers. So I assumed, wrongly as it turned out, that it
would be very simple and attractive to him to modify one to be an
interpreter. The system would be core C language, plus a library
callable as C functions with facilities such as "glob". Now the obvious
way to implement that is to write the library in C or some other
language, the fold it into the interpreter. tcc might provide an
alternative way of achieving the same thing, but I haven't looked into
it. To fold the library into the interpreter you need the interpreter
source and to be comfortable with it.
>> And a scripting
>> language is an obvious application for an intepreter.
>
> The term "scripting language" is so vague as to be almost useless, but
> in this context the suggestion seems to be to use C as a language to
> invoke the commands required to compile and link a C program. But C is
> very poorly suited to this task. One would end up writing a library of
> functions to do string manipulation and (unless 'system' was deemed
> sufficient) program execution. C itself would bring very little to the
> party and the resulting scripts would be hard to read.
>
I'd dispute "very poorly". Of course you can devise a language which is
better suited to building programs specifically. CMake takes exactly
that approach. So what happens? It's effectively another programming
language to learn. Someone else wrote elaborate CMakes scripts which I
use at work. Sometimes thing go wrong. And then I'm messing about with a
language I hardly ever use and does things in ways I am unfamilar with,
trying to troubleshoot. It would be easier if the CMake scripts were in
C. It might be bit less convenient. But it's just looping and branching
and calling subroutines at the end of the day. As you know full well,
that's all computers can do.

The library would have facilities for working with lists of strings, of
course. But C can do that perfectly well, and I don't see it as big problem.

--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On 05/02/2024 15:17, Malcolm McLean wrote:
> On 05/02/2024 14:42, Ben Bacarisse wrote:
>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>
>>> On 04/02/2024 23:32, Ben Bacarisse wrote:
>>>> scott@slp53.sl.home (Scott Lurndal) writes:
>>>>
>>>>> bart <bc@freeuk.com> writes:
>>>>>> On 04/02/2024 15:39, Malcolm McLean wrote:
>>>>>>> How shall we write a better build system?
>>>>>>>
>>>>>>> Current systems use scripts with ad hoc, weird syntax. They don't
>>>>>>> work
>>>>>>> consistently across platforms, they break and are hard to fix,
>>>>>>> and are
>>>>>>> complicated to set up. You even get get one scripting system as a
>>>>>>> front
>>>>>>> end to another, a sort of admission of failure.
>>>>>>>
>>>>>>> But can we do better?
>>>>>>>
>>>>>>> A scripting language provides loops, conditional branches, and
>>>>>>> subroutine calls. And we've already got a perfecty good language for
>>>>>>> describing that. It is C itself.
>>>>>>>
>>>>>>> So here's the proposal. We implement a C interpreter.
>>>>>
>>>>> This was tried.  It's called the c-shell.
>>>>
>>>> csh isn't a C interpreter by any stretch of the imagination!
>>>
>>> No. But the concept of a C interpreter isn't new.
>>
>> Indeed.  But C is not a good fit for interpretation, so there are often
>> compromises.
>>
> It's less of a problem now. The interpreter can just spin up a virtual
> machine and then create a memory space. That was less of a practical
> approach when machines were more limited.
>>> There's one called pico C
>>
>> Yes, and there's tcc's -run option.
>>
>>> which is open source and available to be incorporated.
>>
>> Likewise tcc.  So what's wrong (in your opinion) with these two that
>> caused you to suggest implementing another C interpreter?
>>
> Bart writes compilers. So I assumed, wrongly as it turned out, that it
> would be very simple and attractive to him to modify one to be an
> interpreter.

It's actually quite hard to write an interpreter for a statically typed
language, which has to call FFI routines written in native code, which
include call-back functions that will call back into your program; they
will expect a native code function!

It's extra hard when the language is C, because then it is impossible to
draw a line between routines in language X which is being interpreted,
and language Y which is native code on the other side of the FFI.

Since X and Y are both C.

However, I do have an interpreter for a statically typed IL. To
interpret C code, I need to change the C compiler backend to generate
that IL. (And also tweak that IL because it uses a 64-bit execution
core, but C requires a mixed 32/64-bit core.)

So it's still quite a bit of work.

But I'm still unclear as to why you need a C interpreter, rather than
just using whatever C compiler is provided.

If you provide an interpreter, what form would it take; a binary? If it
can be binary, then why not a binary of the app?

And if you provide source code for the interpreter, theb building the
interpreter might be as much of a job as building the main app!

Ben Bacarisse <ben.usenet@bsb.me.uk> writes:
> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>> On 04/02/2024 23:32, Ben Bacarisse wrote:
[...]
>> No. But the concept of a C interpreter isn't new.
>
> Indeed. But C is not a good fit for interpretation, so there are often
> compromises.
>
>> There's one called pico C
>
> Yes, and there's tcc's -run option.

From the tcc man page:

"tcc -run a.c"
Compile a.c and execute it directly

So it's not exactly an interpreter, though it might be useful in some of
the same contexts where an interpreter would be useful. It would be
straightforward to implement a wrapper for any C compiler that does the
same thing.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

On 05/02/2024 16:00, bart wrote:
> On 05/02/2024 15:17, Malcolm McLean wrote:
>> On 05/02/2024 14:42, Ben Bacarisse wrote:
>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>
>>>> On 04/02/2024 23:32, Ben Bacarisse wrote:
>>>>> scott@slp53.sl.home (Scott Lurndal) writes:
>>>>>
>>>>>> bart <bc@freeuk.com> writes:
>>>>>>> On 04/02/2024 15:39, Malcolm McLean wrote:
>>>>>>>> How shall we write a better build system?
>>>>>>>>
>>>>>>>> Current systems use scripts with ad hoc, weird syntax. They
>>>>>>>> don't work
>>>>>>>> consistently across platforms, they break and are hard to fix,
>>>>>>>> and are
>>>>>>>> complicated to set up. You even get get one scripting system as
>>>>>>>> a front
>>>>>>>> end to another, a sort of admission of failure.
>>>>>>>>
>>>>>>>> But can we do better?
>>>>>>>>
>>>>>>>> A scripting language provides loops, conditional branches, and
>>>>>>>> subroutine calls. And we've already got a perfecty good language
>>>>>>>> for
>>>>>>>> describing that. It is C itself.
>>>>>>>>
>>>>>>>> So here's the proposal. We implement a C interpreter.
>>>>>>
>>>>>> This was tried.  It's called the c-shell.
>>>>>
>>>>> csh isn't a C interpreter by any stretch of the imagination!
>>>>
>>>> No. But the concept of a C interpreter isn't new.
>>>
>>> Indeed.  But C is not a good fit for interpretation, so there are often
>>> compromises.
>>>
>> It's less of a problem now. The interpreter can just spin up a virtual
>> machine and then create a memory space. That was less of a practical
>> approach when machines were more limited.
>>>> There's one called pico C
>>>
>>> Yes, and there's tcc's -run option.
>>>
>>>> which is open source and available to be incorporated.
>>>
>>> Likewise tcc.  So what's wrong (in your opinion) with these two that
>>> caused you to suggest implementing another C interpreter?
>>>
>> Bart writes compilers. So I assumed, wrongly as it turned out, that it
>> would be very simple and attractive to him to modify one to be an
>> interpreter.
>
> It's actually quite hard to write an interpreter for a statically typed
> language, which has to call FFI routines written in native code, which
> include call-back functions that will call back into your program; they
> will expect a native code function!
>
> It's extra hard when the language is C, because then it is impossible to
> draw a line between routines in language X which is being interpreted,
> and language Y which is native code on the other side of the FFI.
>
> Since X and Y are both C.
>
> However, I do have an interpreter for a statically typed IL. To
> interpret C code, I need to change the C compiler backend to generate
> that IL. (And also tweak that IL because it uses a 64-bit execution
> core, but C requires a mixed 32/64-bit core.)
>
> So it's still quite a bit of work.
>
> But I'm still unclear as to why you need a C interpreter, rather than
> just using whatever C compiler is provided.
>
> If you provide an interpreter, what form would it take; a binary? If it
> can be binary, then why not a binary of the app?
>
> And if you provide source code for the interpreter, theb building the
> interpreter might be as much of a job as building the main app!
>
>
The way if would work is that there would be a program called
"fixedmake", or whatever, which was distributed as an executable exactly
like make. The someone writes program source, packages it up in a zip
file, and instead of a makefile, he puts a "fixedmake" file in the
source directory. However the fixedmake file is formally a C program.
Maybe we could have the convention that it is called buildme.c.

The user types fixedmake buildme.c. fixedmake is effectively a C
intepreter. buildme.c executes, calls C compilers and other things as
externals, and the program is built.

Now because buildme.c is a C program, we could use other approaches. You
could go gcc buildme.c -lfixedmake.lib, prodcue an a.out, and run a.out.
But the snag is that it's meant to be a build system. And launching gcc
is in and of itself a build. So it's not really automated any more.

Or we could say that since fixedmake is really just a C intepreter, we
don't have to be ones to write that. Just use tcc. But the problem is
that whilst buildme.c will uusally be fairly short and simple, to keep
it short and simple there will need to a rich library with facilites for
getting lisrs of files, launching compilers, and so on. So if we use
tcc, we've got to package this library as well as buildme.c and the
actual sources in the distribution.

However tcc is open source, so we might be able to modify tcc.

Before the system has any sort of traction, you can't assume that the
fixedmake executable will be available, however. So people are going to
have to provide it with the distribution. And if you provide as binary,
why not just provide the program as binary? And if you provide as source
and build with make, why not build the program with make? But I don't
see any way round that for any new build system.

--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On Mon, 5 Feb 2024 16:00:43 +0000
bart <bc@freeuk.com> wrote:
>
> But I'm still unclear as to why you need a C interpreter, rather than
> just using whatever C compiler is provided.
>

One reason could be that when build system's "meta" code (as opposed to
bulk of project's code) has a bug, especially out of bound array
access, I'd rather want it to fail gracefully, with meaningful error
indication, something that typical compiled C environment unable to
provide.

On 05/02/2024 16:51, Malcolm McLean wrote:
> On 05/02/2024 16:00, bart wrote:

>> And if you provide source code for the interpreter, theb building the
>> interpreter might be as much of a job as building the main app!
>>
>>
> The way if would work is that there would be a program called
> "fixedmake", or whatever, which was distributed as an executable exactly
> like make. The someone writes program source, packages it up in a zip
> file, and instead of a makefile, he puts a "fixedmake" file in the
> source directory. However the fixedmake file is formally a C program.
> Maybe we could have the convention that it is called buildme.c.
>
> The user types fixedmake buildme.c. fixedmake is effectively a C
> intepreter. buildme.c executes, calls C compilers and other things as
> externals, and the program is built.
>
> Now because buildme.c is a C program, we could use other approaches. You
> could go gcc buildme.c -lfixedmake.lib, prodcue an a.out, and run a.out.
> But the snag is that it's meant to be a build system. And launching gcc
> is in and of itself a build. So it's not really automated any more.
>
> Or we could say that since fixedmake is really just a C intepreter, we
> don't have to be ones to write that. Just use tcc. But the problem is
> that whilst buildme.c will uusally be fairly short and simple, to keep
> it short and simple there will need to a rich library with facilites for
> getting lisrs of files, launching compilers, and so on. So if we use
> tcc, we've got to package this library as well as buildme.c and the
> actual sources in the distribution.
>
> However tcc is open source, so we might be able to modify tcc.
>
> Before the system has any sort of traction, you can't assume that the
> fixedmake executable will be available, however. So people are going to
> have to provide it with the distribution. And if you provide as binary,
> why not just provide the program as binary? And if you provide as source
> and build with make, why not build the program with make? But I don't
> see any way round that for any new build system.

One problem as I see is that the build system doesn't know which C
compiler is available, or which one the client wishes to use.

The client will know that, so can simply supply that information.
However it is needed in two places:

(1) The compiler used to build build.c (as it's called in the demo
below).

(2) The compiler used by build.exe to build the app.

(1) is easy, the client is just told to use their prefered compiler here:

tcc build.c

But the next step is harder: build.exe won't know what was typed in step
(1). Even if they do 'tcc -run build.c', 'tcc' does not appear as
'args[0]', it will be 'build.c' (not even build.exe).

Here is a mock-up anyway of a build system:

1) The demo app used has three C files cipher.c, hmac.c,sha2.c

2) There is a program called build.c

3) The data describing the project is in a file called here
filelist.txt, although it contains C source code.

Files (2) and (3) are shown below. It runs like this, using tcc's run
option:

c:\c>tcc -run build.c
Compiler = tcc
Invoking compiler:tcc -o cipher.exe cipher.c hmac.c sha2.c
Finished building: cipher.exe

build.c defaults to using tcc. For use with another compiler, the
process looks like this:

c:\c>gcc build.c -o build

c:\c>build gcc
Compiler = gcc
Invoking compiler:gcc -o cipher.exe cipher.c hmac.c sha2.c
Finished building: cipher.exe

It's a litle untidier. However you can always provide a makefile to shut
people up. The difference here is:

* The makefile only contains the handful of lines needed to implement
the above

* The process can be easily be done manually if needed

* The critical build info, the list of files, is inside the file
filelist.txt. While it is C, it can still be presented in a readable
manner. It can even be incorporated into the user's own programs.

The info in filelist.txt is simplistic. It also needs the locations of
any special headers, the libraries to use and and so on. This is all
info added to the final invocation.

Ideally build.c, knowing the name of the compiler, can select the right
forms of any options needed.

--------------------------------------------------------------
filelist.txt
--------------------------------------------------------------
//#define exeext = ".exe"

char* files[] = {
"cipher.c",
"hmac.c",
"sha2.c"
};

int nfiles = sizeof(files)/sizeof(files[0]);

char* projectname = "cipher";
char* exefilename = "cipher.exe";
--------------------------------------------------------------

--------------------------------------------------------------
build.c
--------------------------------------------------------------
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include "filelist.txt"

char* compiler;

void error(char* mess, char* param){
printf("Build error: %s %s\n",mess, param);
exit(1);
}

void compileproject(void) {
int n, length, i;
char* cmdstr;

length=0;
for (i=0; i<nfiles; ++i) length+=strlen(files[i])+1;
cmdstr=malloc(length+100);

strcpy(cmdstr, compiler);
strcat(cmdstr, " ");

strcat(cmdstr, "-o ");
strcat(cmdstr, exefilename);
strcat(cmdstr, " ");

for (i=0; i<nfiles; ++i) {
strcat(cmdstr, files[i]);
strcat(cmdstr, " ");
}

printf("Invoking compiler:");
puts(cmdstr);
if (system(cmdstr)!=0)
error("Error building:", exefilename);
else
printf("Finished building: %s\n", exefilename);
}

int main(int n, char** args) {

if (n>=2)
compiler=strdup(args[1]);
else
compiler="tcc";

printf("Compiler = %s\n", compiler);

compileproject();
} --------------------------------------------------------------

On 2/5/2024 9:39 AM, bart wrote:
> On 05/02/2024 16:51, Malcolm McLean wrote:
>> On 05/02/2024 16:00, bart wrote:
[...]
>     printf("Invoking compiler:");
>     puts(cmdstr);
>     if (system(cmdstr)!=0)
>         error("Error building:", exefilename);
>     else
>         printf("Finished building: %s\n", exefilename);
> }
[...]

For some reason this is reminding me of how one compiles and links GLSL
shader programs on the fly wrt OpenGL.

On 05/02/2024 16:17, Malcolm McLean wrote:
> On 05/02/2024 14:42, Ben Bacarisse wrote:
>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>

>> The term "scripting language" is so vague as to be almost useless, but
>> in this context the suggestion seems to be to use C as a language to
>> invoke the commands required to compile and link a C program.  But C is
>> very poorly suited to this task.  One would end up writing a library of
>> functions to do string manipulation and (unless 'system' was deemed
>> sufficient) program execution.  C itself would bring very little to the
>> party and the resulting scripts would be hard to read.
>>
> I'd dispute "very poorly". Of course you can devise a language which is
> better suited to building programs specifically. CMake takes exactly
> that approach. So what happens? It's effectively another programming
> language to learn. Someone else wrote elaborate CMakes scripts which I
> use at work. Sometimes thing go wrong. And then I'm messing about with a
> language I hardly ever use and does things in ways I am unfamilar with,
> trying to troubleshoot. It would be easier if the CMake scripts were in
> C. It might be bit less convenient. But it's just looping and branching
> and calling subroutines at the end of the day. As you know full well,
> that's all computers can do.
>
> The library would have facilities for working with lists of strings, of
> course. But C can do that perfectly well, and I don't see it as big
> problem.
>

Have you ever tried /any/ other programming language? Apart from
assembly and Forth, I have not seen a language that is more cumbersome
for string handling than C. Yes, you /can/ do string handling in C, but
no one would /choose/ C for that.

It's not unreasonable to suggest that you'd rather base your build
system on an existing mainstream language than a domain-specific
language (though DSL's have the big advantage of having a syntax and
semantics tuned to the task in question). But why C?

Here's a quick challenge for you.

I have a directory "src". I want to find all the .c files in "src". I
want to make a list of all these files, and a list of matching object
files to make in the "build/obj/src" directory. For each file, I want
to call "gcc -c src/file.c -o build/obj/src/file.o". I want to do so in
parallel, up to 8 commands at a time. (Ignore any possible runtime errors.)

With make, that would be (this is untested) :

default : all
.PHONY all
srcdir = "src"
objdir = "build/obj"

srcfiles = $(wildcard $(srcdir)/*.c)
objfiles_src = $(srcfiles:.c=.o)
objfiles = $(addprefix $(objdir)/, \
$(patsubst ../%,%,$(objfiles_src)))

all : objfiles
$(objdir)/src :
mkdir -p $@
$(objdir)/%.o : %.c | $(objdir)/src
gcc -c $< -o $@

Run with "make -j 8".

With Python, you could have (untested) :

import glob
import os
import multiprocessing
import subprocess

srcdir = "src"
objdir = "build/obj"

srcfiles = glob.glob(srcdir + "/*.c")
objfiles_src = [fn[:-2] + ".o" for fn in srcfiles]
objfiles = [objdir + "/" + fn for fn in srcfiles]
os.mkdirs(objdir + "/" + srcdir, exist_ok = True)

with multiprocessing.Pool(5) as pool :
pool.map(lambda src, obj : subprocess.call("gcc -c " +
src + " -o " + obj, shell = True),
zip(srcfiles, objfiles))

Obviously much of this could be put in a reusable library, so that end
users don't have to know about process pools. (And there are many other
ways to structure such code.)

Your task is to duplicate this in C.

On 05/02/2024 19:47, David Brown wrote:
> On 05/02/2024 16:17, Malcolm McLean wrote:
>> On 05/02/2024 14:42, Ben Bacarisse wrote:
>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>
>
>>> The term "scripting language" is so vague as to be almost useless, but
>>> in this context the suggestion seems to be to use C as a language to
>>> invoke the commands required to compile and link a C program.  But C is
>>> very poorly suited to this task.  One would end up writing a library of
>>> functions to do string manipulation and (unless 'system' was deemed
>>> sufficient) program execution.  C itself would bring very little to the
>>> party and the resulting scripts would be hard to read.
>>>
>> I'd dispute "very poorly". Of course you can devise a language which
>> is better suited to building programs specifically. CMake takes
>> exactly that approach. So what happens? It's effectively another
>> programming language to learn. Someone else wrote elaborate CMakes
>> scripts which I use at work. Sometimes thing go wrong. And then I'm
>> messing about with a language I hardly ever use and does things in
>> ways I am unfamilar with, trying to troubleshoot. It would be easier
>> if the CMake scripts were in C. It might be bit less convenient. But
>> it's just looping and branching and calling subroutines at the end of
>> the day. As you know full well, that's all computers can do.
>>
>> The library would have facilities for working with lists of strings,
>> of course. But C can do that perfectly well, and I don't see it as big
>> problem.
>>
>
> Have you ever tried /any/ other programming language?  Apart from
> assembly and Forth, I have not seen a language that is more cumbersome
> for string handling than C.  Yes, you /can/ do string handling in C, but
> no one would /choose/ C for that.
>
> It's not unreasonable to suggest that you'd rather base your build
> system on an existing mainstream language than a domain-specific
> language (though DSL's have the big advantage of having a syntax and
> semantics tuned to the task in question).  But why C?
>
The newgroup is comp.lang.c. So if we propose using a generally accepted
and widely understood programming language as our build scripting
language, the choice of language has to be C.

> Here's a quick challenge for you.
>
> I have a directory "src".  I want to find all the .c files in "src".  I
> want to make a list of all these files, and a list of matching object
> files to make in the "build/obj/src" directory.  For each file, I want
> to call "gcc -c src/file.c -o build/obj/src/file.o".  I want to do so in
> parallel, up to 8 commands at a time.  (Ignore any possible runtime
> errors.)
>
>
> With make, that would be (this is untested) :
>
>     default : all
>     .PHONY all
>     srcdir = "src"
>     objdir = "build/obj"
>
>     srcfiles = $(wildcard $(srcdir)/*.c)
>     objfiles_src = $(srcfiles:.c=.o)
>     objfiles = $(addprefix $(objdir)/, \
>         $(patsubst ../%,%,$(objfiles_src)))
>
>     all : objfiles
>
>     $(objdir)/src :
>             mkdir -p $@
>
>     $(objdir)/%.o : %.c | $(objdir)/src
>             gcc -c $< -o $@
>
> Run with "make -j 8".
>
>
> With Python, you could have (untested) :
>
>     import glob
>     import os
>     import multiprocessing
>     import subprocess
>
>     srcdir = "src"
>     objdir = "build/obj"
>
>     srcfiles = glob.glob(srcdir + "/*.c")
>     objfiles_src = [fn[:-2] + ".o" for fn in srcfiles]
>     objfiles = [objdir + "/" + fn for fn in srcfiles]
>
>     os.mkdirs(objdir + "/" + srcdir, exist_ok = True)
>
>     with multiprocessing.Pool(5) as pool :
>         pool.map(lambda src, obj : subprocess.call("gcc -c " +
>                  src + " -o " + obj, shell = True),
>                 zip(srcfiles, objfiles))
>
> Obviously much of this could be put in a reusable library, so that end
> users don't have to know about process pools.  (And there are many other
> ways to structure such code.)
>
>
> Your task is to duplicate this in C.
>
>

#include "builduils.h"

int main(void)
{ STRINGLIST *sources;
STRINGLIST *objfiles;
int i;

sources = glob("src/*.c");
objfiles = stringlist();
for (i = 0; i < stringlist_Nstrings(sources); i++)
{
char objname[1024];
snprintf(objname, 1024, "%s/%s", objdir,
replaceextension(stringlist_get(sources, i), ".o"));
stringlist_add(objfiles, objname);
}
char outputdir[1024];
snprinf("%s/%s", 1024, objdir, srcdir);
mkdir(outputdir);

callparallel("gcc -c %s %s", srcfiles, objfiles);
}

OK now I've cheated a little bit by inventing buildutils library
functions ad hoc. But I haven't done so outrageously. And we're leaking
memory. But likely we have many gigabytres, so who are about maybe 2K
for a few strings. And yes, maybe we could reoplace snprintf() with
something that creates an arbitrary length output.

The Python is a bit shorter and more concise. But there's not that much
in it.
--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On Mon, 5 Feb 2024 20:47:53 +0100
David Brown <david.brown@hesbynett.no> wrote:
>
> Have you ever tried /any/ other programming language? Apart from
> assembly and Forth, I have not seen a language that is more
> cumbersome for string handling than C.
>

In order of my exposure to them: Pascal, Fortran, Ada.

On 05/02/2024 19:47, David Brown wrote:
> On 05/02/2024 16:17, Malcolm McLean wrote:

>> The library would have facilities for working with lists of strings,
>> of course. But C can do that perfectly well, and I don't see it as big
>> problem.
>>
>
> Have you ever tried /any/ other programming language?  Apart from
> assembly and Forth, I have not seen a language that is more cumbersome
> for string handling than C.  Yes, you /can/ do string handling in C, but
> no one would /choose/ C for that.

See the build.c demo elsewhere in the thread.

> I have a directory "src".  I want to find all the .c files in "src".  I
> want to make a list of all these files, and a list of matching object
> files to make in the "build/obj/src" directory.  For each file, I want
> to call "gcc -c src/file.c -o build/obj/src/file.o".  I want to do so in
> parallel, up to 8 commands at a time.  (Ignore any possible runtime
> errors.)

> With Python, you could have (untested) :
>
>     import glob
>     import os
>     import multiprocessing
>     import subprocess
>
>     srcdir = "src"
>     objdir = "build/obj"
>
>     srcfiles = glob.glob(srcdir + "/*.c")
>     objfiles_src = [fn[:-2] + ".o" for fn in srcfiles]
>     objfiles = [objdir + "/" + fn for fn in srcfiles]
>
>     os.mkdirs(objdir + "/" + srcdir, exist_ok = True)
>
>     with multiprocessing.Pool(5) as pool :
>         pool.map(lambda src, obj : subprocess.call("gcc -c " +
>                  src + " -o " + obj, shell = True),
>                 zip(srcfiles, objfiles))
>
> Obviously much of this could be put in a reusable library, so that end
> users don't have to know about process pools.  (And there are many other
> ways to structure such code.)

I had a go at expressing the Python version in my scripting language:

srcdir := "src/"
objdir := "build/obj/"+srcdir

srcfiles := dirlist(srcdir + "*.c")

objfiles_src := mapvs(changeext, srcfiles, "o")
objfiles := mapsv(+, objdir, objfiles_src)

createdir(objdir)

for i, sfile in srcfiles do
execcmd(sfprint("gcc -c # -o #", srcdir+sfile, objfiles[i]))
od

A couple of issues: 'createdir' can only create one directory at a time;
for a chain of them like a/b/c, I'd need to split it up and do it one by
one. (To test this, I created it manually.)

Also, I don't have any features for parallel executions. However
'execcmd' starts a process but then doesn't wait for it to complete. If
I instead use 'system' (or my 'execwait'), then compiling all Lua .c
files in ./src takes 6.5 seconds instead of 2.5 seconds.

But this isn't the point of posting this. My interpreter can be
expressed as a single C file. That allows you to use this kind of
scripting, without adding any extra dependencies. You still only need a
C compiler.

(I tried your Python, but os.mkdir(dir) didn't work, even after fixing
the name typo and removing the named argument. Bypassing that, there
were all sorts of errors to do with pickle.py and 'unbounded' methods.

I believe your glob.glob routine returns filenames with path prepended
(my dirlist doesn't). I suspect that's why you chose a dest path as
build/obj/src rather than build/obj. Anyway, I wasn't able to compare
peformance.)

BTW this scripting is still hard work. For building you want to express
the requires as data, not code. If you can do that (see my build.c
example), then C may be adequate.

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
> On 05/02/2024 19:47, David Brown wrote:
[...]
>> Have you ever tried /any/ other programming language?  Apart from
>> assembly and Forth, I have not seen a language that is more
>> cumbersome for string handling than C.  Yes, you /can/ do string
>> handling in C, but no one would /choose/ C for that.
>> It's not unreasonable to suggest that you'd rather base your build
>> system on an existing mainstream language than a domain-specific
>> language (though DSL's have the big advantage of having a syntax and
>> semantics tuned to the task in question).  But why C?
>>
> The newgroup is comp.lang.c. So if we propose using a generally
> accepted and widely understood programming language as our build
> scripting language, the choice of language has to be C.

Right -- *if* we want to discuss it here in comp.lang.c.

If C is not a suitable language for a new build systems, the solution is
to discuss it somewhere else, not to discuss it here and pretend C is
the best answer.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

On Mon, 5 Feb 2024 15:17:15 +0000, Malcolm McLean wrote:

> It would be easier if the CMake scripts were in
> C. It might be bit less convenient.

Aren’t “easier” and “less convenient” kind of ... opposites?

On Mon, 5 Feb 2024 21:28:20 +0000, bart wrote:

> (I tried your Python, but os.mkdir(dir) didn't work, even after fixing
> the name typo and removing the named argument.

“mkdirs” should have been “makedirs”
<https://docs.python.org/3/library/os.html#os.makedirs>.

> Bypassing that, there
> were all sorts of errors to do with pickle.py and 'unbounded' methods.

Trouble with your Python install? In other words, Windows trouble?

On Mon, 5 Feb 2024 17:39:25 +0000, bart wrote:

> strcat(cmdstr, "-o ");
> strcat(cmdstr, exefilename);
> strcat(cmdstr, " ");
>
> for (i=0; i<nfiles; ++i) {
> strcat(cmdstr, files[i]);
> strcat(cmdstr, " ");
> }
>
> ...
> if (system(cmdstr)!=0)

What is wrong with this code?