On 28/02/2024 02:09, Kaz Kylheku wrote:
> On 2024-02-27, Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
>> On Tue, 27 Feb 2024 23:21:28 +0100, David Brown wrote:
>>
>>> The #embed pre-processor directive turns the file into a list of integer
>>> constants, one per byte (unless an implementation offers other options).
>>
>> What a waste of time.
>
> Plus easily doable in 1970's Lisp.
>

That would be useful, if we were living in the 1970's or if anyone had
wanted to learn Lisp this side of the millennium bug.

As I mentioned before, it's not particularly difficult to do this kind
of manipulation, and people write utilities for them in a variety of
languages, or download a variety of free tools for the job.

But it will often be more convenient to have it built into the language
and compiler. And for those interested in speed, the test
implementations have handled this far more efficiently than other
techniques. Logically, #embed turns the file into a list of numbers. In
practice, if you use it for the common case of initialising a const
array of unsigned char, the compiler simply copies and pastes the file
into the output as a binary blob.

It would, IMHO, have been useful also to have had an "embed operator" in
the manner of the "pragma operator", so that it could be used in a macro
definition.

On 27/02/2024 23:12, bart wrote:
> On 27/02/2024 20:25, Lawrence D'Oliveiro wrote:
>> On Tue, 27 Feb 2024 09:36:38 +0100, David Brown wrote:
>>
>>> And with C23, we will get #embed, though it is not yet supported by
>>> major tools.
>>
>> More and more hacks on the preprocessor. Why not just get rid of it and
>> replace it with something like m4?
>>
>> Because then you will discover that string-based macros are inherently an
>> unmanageable problem.
>
> I hadn't notice that #embed was a preprocessor directive. But that is
> not the problem here, it is this:
>
> "The expansion of a #embed directive is a token sequence formed from the
> list of integer constant expressions described below."
>
> If a string like "ABC" really is converted to the five tokens 'A' comma
> 'B' comma 'C', then it's going to make long strings and binary files
> inefficient.
>
> Embedding a 100KB file will result in a 100KB bigger executable, but
> along the way it may have to generate 200,000 tokens within the
> compiler, half of them commas. Which in turn will need to be turned into
> 100,000 integer expressions.
>
> I would hope that implementations find some way of streamlining that
> process, perhaps by turning that 100KB of data directly into a 100KB
> string.

They won't use strings, they will use data blobs - binary data. Then
there is no issue with null bytes. And yes, implementations will skip
the token generation (unless you are doing something weird, such as
using #embed to read the parameters to a function call).

Tests with prototype implementations gave extremely fast results.

On 28/02/2024 11:54, David Brown wrote:
> On 27/02/2024 23:12, bart wrote:
>> On 27/02/2024 20:25, Lawrence D'Oliveiro wrote:
>>> On Tue, 27 Feb 2024 09:36:38 +0100, David Brown wrote:
>>>
>>>> And with C23, we will get #embed, though it is not yet supported by
>>>> major tools.
>>>
>>> More and more hacks on the preprocessor. Why not just get rid of it and
>>> replace it with something like m4?
>>>
>>> Because then you will discover that string-based macros are
>>> inherently an
>>> unmanageable problem.
>>
>> I hadn't notice that #embed was a preprocessor directive. But that is
>> not the problem here, it is this:
>>
>> "The expansion of a #embed directive is a token sequence formed from
>> the list of integer constant expressions described below."
>>
>> If a string like "ABC" really is converted to the five tokens 'A'
>> comma 'B' comma 'C', then it's going to make long strings and binary
>> files inefficient.
>>
>> Embedding a 100KB file will result in a 100KB bigger executable, but
>> along the way it may have to generate 200,000 tokens within the
>> compiler, half of them commas. Which in turn will need to be turned
>> into 100,000 integer expressions.
>>
>> I would hope that implementations find some way of streamlining that
>> process, perhaps by turning that 100KB of data directly into a 100KB
>> string.
>
> They won't use strings, they will use data blobs - binary data.  Then
> there is no issue with null bytes.

AFAIK strings in C can have embedded zeros when not assumed to be
zero-terminated. So here:

char s[]={1,2,3,0,4,5,6};

s will have a length of 7.

>  And yes, implementations will skip
> the token generation (unless you are doing something weird, such as
> using #embed to read the parameters to a function call).

What happens if you do -E to preprocess only?

On 28/02/2024 14:13, bart wrote:
> On 28/02/2024 11:54, David Brown wrote:
>> On 27/02/2024 23:12, bart wrote:
>>> On 27/02/2024 20:25, Lawrence D'Oliveiro wrote:
>>>> On Tue, 27 Feb 2024 09:36:38 +0100, David Brown wrote:
>>>>
>>>>> And with C23, we will get #embed, though it is not yet supported by
>>>>> major tools.
>>>>
>>>> More and more hacks on the preprocessor. Why not just get rid of it and
>>>> replace it with something like m4?
>>>>
>>>> Because then you will discover that string-based macros are
>>>> inherently an
>>>> unmanageable problem.
>>>
>>> I hadn't notice that #embed was a preprocessor directive. But that is
>>> not the problem here, it is this:
>>>
>>> "The expansion of a #embed directive is a token sequence formed from
>>> the list of integer constant expressions described below."
>>>
>>> If a string like "ABC" really is converted to the five tokens 'A'
>>> comma 'B' comma 'C', then it's going to make long strings and binary
>>> files inefficient.
>>>
>>> Embedding a 100KB file will result in a 100KB bigger executable, but
>>> along the way it may have to generate 200,000 tokens within the
>>> compiler, half of them commas. Which in turn will need to be turned
>>> into 100,000 integer expressions.
>>>
>>> I would hope that implementations find some way of streamlining that
>>> process, perhaps by turning that 100KB of data directly into a 100KB
>>> string.
>>
>> They won't use strings, they will use data blobs - binary data.  Then
>> there is no issue with null bytes.
>
> AFAIK strings in C can have embedded zeros when not assumed to be
> zero-terminated. So here:
>
>     char s[]={1,2,3,0,4,5,6};
>
> s will have a length of 7.

That's not a string, it's an array of char. A "string" in C is "a
contiguous sequence of characters terminated by and including the first
null character". The difference is crucial in respect to the handling
of null bytes. And it is the main reason for #embed generating a
comma-separated sequence of integer constants rather than a string. (It
also avoids messy hex character sequences if you show the output of
#embed somewhere.)

>
>>   And yes, implementations will skip the token generation (unless you
>> are doing something weird, such as using #embed to read the parameters
>> to a function call).
>
> What happens if you do -E to preprocess only?
>

That's something weird :-)

I guess you get the integer list.

On Wed, 28 Feb 2024 12:50:10 +0100, David Brown wrote:

> ... people write utilities for them in a variety of languages ...
>
> But it will often be more convenient to have it built into the language
> and compiler.

What can be built into the language can only ever be a small subset of
the many and varied ways that people have incorporated data blobs into
their programs. Often these will need to have custom structures with
computed header fields, that kind of thing. So you will need custom
build tools to construct these structures, and then you might as well
include those blobs directly into the final build, rather than go
through some extra step of pretending to turn them back into some
source form.

For example, here’s an old Android project of mine (OK, so the app is
Java code, but the same principle applies)
<https://bitbucket.org/ldo17/unicode_browser_android/src/master/>
where I wrote a custom Python script to read a Nameslist.txt file
downloaded from unicode.org to generate a table which could be loaded
into memory quickly for easy searching.

On 28/02/2024 20:56, Lawrence D'Oliveiro wrote:
> On Wed, 28 Feb 2024 12:50:10 +0100, David Brown wrote:
>
>> ... people write utilities for them in a variety of languages ...
>>
>> But it will often be more convenient to have it built into the language
>> and compiler.
>
> What can be built into the language can only ever be a small subset of
> the many and varied ways that people have incorporated data blobs into
> their programs. Often these will need to have custom structures with
> computed header fields, that kind of thing. So you will need custom
> build tools to construct these structures, and then you might as well
> include those blobs directly into the final build, rather than go
> through some extra step of pretending to turn them back into some
> source form.
>
> For example, here’s an old Android project of mine (OK, so the app is
> Java code, but the same principle applies)
> <https://bitbucket.org/ldo17/unicode_browser_android/src/master/>
> where I wrote a custom Python script to read a Nameslist.txt file
> downloaded from unicode.org to generate a table which could be loaded
> into memory quickly for easy searching.

I can see now where you get your coding style from. You seem to like
stretching things out vertically as much as possible:

public void Add
(
int CategoryCode,
ItemType Item
)
/* Use this instead of add to populate CodeToIndex table. */
{
CodeToIndex.put(CategoryCode, getCount());
add(Item);
} /*Add*/

In C:

void Add(int CategoryCode, ItemType Item) {
CodeToIndex_put(CategoryCode, getCount());
add(Item);
}

4 non-comment lines versus 9. I know Java needs tons of boilerplate, but
but it is not all the language's fault.

bart <bc@freeuk.com> writes:
[...]
> AFAIK strings in C can have embedded zeros when not assumed to be
> zero-terminated. So here:
>
> char s[]={1,2,3,0,4,5,6};
>
> s will have a length of 7.

Strings *by definition* cannot have embedded zeros. A null character
terminates a string.

A string literal can have embedded \0 characters, but if you're
suggesting that #embed should expand to a string literal, I can see
several disadvantages and no significant advantages. For one thing, the
data may or may not end with a null character; string literals always
do.

[...]

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

bart <bc@freeuk.com> writes:
[...]
> AFAIK strings in C can have embedded zeros when not assumed to be
> zero-terminated. So here:
>
> char s[]={1,2,3,0,4,5,6};
>
> s will have a length of 7.

s will have a *size* of 7. Its length, as a string, is 3. The
distinction between "length" and "size" is particularly important in
this case.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

David Brown <david.brown@hesbynett.no> writes:
[...]
> They won't use strings, they will use data blobs - binary data. Then
> there is no issue with null bytes. And yes, implementations will skip
> the token generation (unless you are doing something weird, such as
> using #embed to read the parameters to a function call).
>
> Tests with prototype implementations gave extremely fast results.

I'm not sure how that would work. #embed is a preprocessor directive,
and at least in the abstract model it has to expand to valid C code.

I would have expected that it would simply generate the list of
comma-separated integer constants described in the standard; later
phases would simply parse that list and generate code as if that
sequence had been written in the original source file. Do you know of
an implementation that does something else?

For example, say you have a file "foo.dat" containing 4 bytes with
values 0, 1, 2, and 3. This would be perfectly valid:

struct foo {
unsigned char a;
unsigned short b;
unsigned int c;
double d;
};

struct foo obj = {
#embed "foo.dat"
};

#embed isn't defined to translate an input file to a sequence of bytes.
It's defined to translate an input file to a sequence of integer
constant expressions.

*Maybe* a compiler could optimize for the case where it knows that it's
being used to initialize an array of unsigned char, but (a) that would
require the preprocessor to have information that normally doesn't exist
until later phases, and (b) I'm not convinced it would be worth the
effort.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

On 28/02/2024 21:57, Keith Thompson wrote:
> David Brown <david.brown@hesbynett.no> writes:
> [...]
>> They won't use strings, they will use data blobs - binary data. Then
>> there is no issue with null bytes. And yes, implementations will skip
>> the token generation (unless you are doing something weird, such as
>> using #embed to read the parameters to a function call).
>>
>> Tests with prototype implementations gave extremely fast results.
>
> I'm not sure how that would work. #embed is a preprocessor directive,
> and at least in the abstract model it has to expand to valid C code.
>
> I would have expected that it would simply generate the list of
> comma-separated integer constants described in the standard; later
> phases would simply parse that list and generate code as if that
> sequence had been written in the original source file. Do you know of
> an implementation that does something else?
>
> For example, say you have a file "foo.dat" containing 4 bytes with
> values 0, 1, 2, and 3. This would be perfectly valid:
>
> struct foo {
> unsigned char a;
> unsigned short b;
> unsigned int c;
> double d;
> };
>
> struct foo obj = {
> #embed "foo.dat"
> };

It would be unfortunate if your example was allowed. Clearly a binary
representation of an instance of your struct would probably require 16
bytes rather than 4, of which one may be padding.

Certainly if you were to write it out to disk as binary, it would need
more than 4.

> #embed isn't defined to translate an input file to a sequence of bytes.
> It's defined to translate an input file to a sequence of integer
> constant expressions.

Maybe it should be defined exactly like that, because that is what
people might expect. You example is better off using a normal text file
which contains an actual comma-delimited list (and which can mix ints
and floats), and a regular #include.

bart <bc@freeuk.com> writes:
> On 28/02/2024 21:57, Keith Thompson wrote:
>> David Brown <david.brown@hesbynett.no> writes:
>> [...]
>>> They won't use strings, they will use data blobs - binary data. Then
>>> there is no issue with null bytes. And yes, implementations will skip
>>> the token generation (unless you are doing something weird, such as
>>> using #embed to read the parameters to a function call).
>>>
>>> Tests with prototype implementations gave extremely fast results.
>> I'm not sure how that would work. #embed is a preprocessor
>> directive,
>> and at least in the abstract model it has to expand to valid C code.
>> I would have expected that it would simply generate the list of
>> comma-separated integer constants described in the standard; later
>> phases would simply parse that list and generate code as if that
>> sequence had been written in the original source file. Do you know of
>> an implementation that does something else?
>> For example, say you have a file "foo.dat" containing 4 bytes with
>> values 0, 1, 2, and 3. This would be perfectly valid:
>> struct foo {
>> unsigned char a;
>> unsigned short b;
>> unsigned int c;
>> double d;
>> };
>> struct foo obj = {
>> #embed "foo.dat"
>> };
>
> It would be unfortunate if your example was allowed. Clearly a binary
> representation of an instance of your struct would probably require 16
> bytes rather than 4, of which one may be padding.

Depending on the sizes and alignments of the various types, sure.
So what?

N3096 is the latest public C23 draft.

https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3096.pdf

#embed is defined in section 6.10.3.

The expansion of a #embed directive is a token sequence
formed from the list of integer constant expressions described
below. The group of tokens for each integer constant expression
in the list is separated in the token sequence from the group
of tokens for the previous integer constant expression in the
list by a comma. The sequence neither begins nor ends in a
comma. If the list of integer constant expressions is empty,
the token sequence is empty. The directive is replaced by its
expansion and, with the presence of certain embed parameters,
additional or replacement token sequences.

It's a preprocessor directive. The preprocessor operates on text and
proprocessing tokens, not on raw data. There is no way to directly
represent raw data in C source code. (I suppose string literals do so
to some extent, but they can't represent generalized raw data.)

The usage I described above is allowed. I see nothing unfortunate about
it. If you only want to use #embed with arrays of unsigned char, then
do that.

Its primary intended use is to read binary file contents at compile time
and allow a program to treat those contents as a raw representation,
particularly as the initialization for an array of unsigned char. There
was no reason to impose arbitrary restrictions to make it impossible to
use for any other purposes.

I suppose it would have been possible for #embed to expand to the raw
data itself, a binary copy of the input file. That would require C
source code, which currently is plain text, to be able to support
delimited chunks of binary data. It would require changes to portions
of the compiler after the preprocessor. Presumably you'd be able to
write the same representation directly in a C source file, which means
that C source files would no longer necessarily be representable as
text. I can see that causing all kinds of problems.

Fortunately, none of that was necessary, since the authors came up with
a way to define #embed in the preprocessor without making any other
changes to how C source code is represented. The fact that it can be
used in other odd ways doesn't bother me. The code I wrote above is
valid; I never said it was acceptable style.

> Certainly if you were to write it out to disk as binary, it would need
> more than 4.

Yes. So what?

>> #embed isn't defined to translate an input file to a sequence of bytes.
>> It's defined to translate an input file to a sequence of integer
>> constant expressions.
>
> Maybe it should be defined exactly like that, because that is what
> people might expect. You example is better off using a normal text
> file which contains an actual comma-delimited list (and which can mix
> ints and floats), and a regular #include.

I certainly wouldn't advocate writing code like the above. My point is
that, given the definition of #embed in the C23 standard, it's valid and
has well defined semantics.

If you have suggestions for alternate ways to define #embed, they might
be interesting, but it's too late to change the existing specification.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

On Wed, 28 Feb 2024 21:34:14 +0000, bart wrote:

> In C:
>
> void Add(int CategoryCode, ItemType Item) {
> CodeToIndex_put(CategoryCode, getCount());
> add(Item);
> }
>
> 4 non-comment lines versus 9. I know Java needs tons of boilerplate, but
> but it is not all the language's fault.

Or how about

void Add(int CategoryCode, ItemType Item) {CodeToIndex_put(CategoryCode, getCount());add(Item);}

Wow! I never realized you could do that in C!! I thought it was an
error to put stuff after column 72 or something. Thanks for the tip!!!

On 28/02/2024 23:52, Lawrence D'Oliveiro wrote:
> On Wed, 28 Feb 2024 21:34:14 +0000, bart wrote:
>
>> In C:
>>
>> void Add(int CategoryCode, ItemType Item) {
>> CodeToIndex_put(CategoryCode, getCount());
>> add(Item);
>> }
>>
>> 4 non-comment lines versus 9. I know Java needs tons of boilerplate, but
>> but it is not all the language's fault.
>
> Or how about
>
> void Add(int CategoryCode, ItemType Item) {CodeToIndex_put(CategoryCode, getCount());add(Item);}
>
> Wow! I never realized you could do that in C!! I thought it was an
> error to put stuff after column 72 or something. Thanks for the tip!!!

Well, you could write an entire program on one line.

Or you can write an entire program in one thin column:

v\
o\
i\
d\
....

Or you can use common sense and avoiding writing code which is either
too compact or so spread out vertically that you have to hunt for the
actual code. Like trying to find the bits of meat in a thin soup.

That's what I took away from your Java code, which looks remarkably like
the spaced-out examples you posted recently.

On 28/02/2024 23:31, Keith Thompson wrote:
> bart <bc@freeuk.com> writes:

>> It would be unfortunate if your example was allowed. Clearly a binary
>> representation of an instance of your struct would probably require 16
>> bytes rather than 4, of which one may be padding.
>
> Depending on the sizes and alignments of the various types, sure.
> So what?
>

> If you have suggestions for alternate ways to define #embed, they might
> be interesting, but it's too late to change the existing specification.
>

My early comments on this were about compiler performance. I suggested
there might be a way to turn 100,000 byte values in a file, directly
into a 100KB string or data block, without needing to first convert
100,000 values into 100,000 integer expressions representated as tokens,
and to then parse those 100,000 expressions into AST nodes etc.

DB suggested something like that was actually done. But you can't do
that if those 100,000 numbers represent from 100KB to 800KB of memory
depending on the data type of the strucure they're initialising.

They might even be mixed type. Or it might be an example like this:

A binary file contains 8 bytes representing one IEEE754 float value. It
is desired to use that to initialise a double array of one element.

However #embed will that into 8 integer values of 0 to 255 each (I assume).

It's not clear either what happens when one of the integers has the
value 150, say, but it is used to initialise an element of type (signed)
char. It sounds like it would make it hard to inialise a char[] array,
when char is signed, from a file of UTF8 text.

Basically, #embed is dumb.

For flexibility, I wouldn't use #embed at all. Just have an actual
comma-separated set of values in a text file, and use #include instead.

bart <bc@freeuk.com> writes:
> On 28/02/2024 23:31, Keith Thompson wrote:
>> bart <bc@freeuk.com> writes:
>>> It would be unfortunate if your example was allowed. Clearly a binary
>>> representation of an instance of your struct would probably require 16
>>> bytes rather than 4, of which one may be padding.
>> Depending on the sizes and alignments of the various types, sure.
>> So what?
>
>> If you have suggestions for alternate ways to define #embed, they might
>> be interesting, but it's too late to change the existing specification.
>
> My early comments on this were about compiler performance. I suggested
> there might be a way to turn 100,000 byte values in a file, directly
> into a 100KB string or data block, without needing to first convert
> 100,000 values into 100,000 integer expressions representated as
> tokens, and to then parse those 100,000 expressions into AST nodes
> etc.

I suggest that (a) parsing thoser 100,000 byte values isn't likely to be
a huge deal (if you have actual performance figures that contradict
that, feel free to present them), and (b) any solution that doesn't
involve expanding to C source code would require a lot more work to
implement for very little benefit.

> DB suggested something like that was actually done. But you can't do
> that if those 100,000 numbers represent from 100KB to 800KB of memory
> depending on the data type of the strucure they're initialising.

Neither gcc nor clang implements #embed yet. DB mentioned prototype
implementations. I've asked him for more information.

> They might even be mixed type. Or it might be an example like this:
>
> A binary file contains 8 byes representing one IEEE754 float
> value. It is desired to use that to initialise a double array of one
> element.
>
> However #embed will that into 8 integer values of 0 to 255 each (I assume).

Assuming CHAR_BIT==8, yes. You can use it to initialize a union, or use
memcpy() to copy from an array of unsigned char into a double object.
(Storing double values in binary files is uncommon, but it's certainly
possible.)

> It's not clear either what happens when one of the integers has the
> value 150, say, but it is used to initialise an element of type
> (signed) char. It sounds like it would make it hard to inialise a
> char[] array, when char is signed, from a file of UTF8 text.

Say you have a binary file containing a single byte with the value 150
(when interpreted as an 8-bit unsigned char). Then
#embed "file.dat"
will expand to something like
150
or
0x96

So if you write:

char array[] = {
#embed file.dat
};

then it's treated exactly the same as
char array[] = { 150 };

If plain char is signed, then the result of the conversion is
implementation-defined, but is very very likely to result in a value of
-106.

I expect that 99% of the uses of #embed will be to initialize arrays of
unsigned char (or uint8_t). For that purpose, it should work just fine.

> Basically, #embed is dumb.

Do you object to the fact that the authors didn't add additional
arbitrary restrictions to forbid uses that you don't like?

> For flexibility, I wouldn't use #embed at all. Just have an actual
> comma-separated set of values in a text file, and use #include
> instead.

And you can still do that.

If you have a png image file and you want to include its contents in
your C program, you can use a separate program to translate the file to
C source and #include the result, or you can use `#embed "foo.png"`.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

On Thu, 29 Feb 2024 00:15:17 +0000, bart wrote:

> Or you can use common sense and avoiding writing code which is either
> too compact or so spread out vertically that you have to hunt for the
> actual code. Like trying to find the bits of meat in a thin soup.

Terribly sorry about that. I wonder if you could look at this part of the
same code file:

final android.util.SparseArray<Integer> CodeToIndex =
new android.util.SparseArray<Integer>();

and show me how to thicken that part of my humble, tasteless gruel? Maybe
using that same “_” trick you used to do OO in C in your previous example?

On 28/02/2024 21:56, Lawrence D'Oliveiro wrote:
> On Wed, 28 Feb 2024 12:50:10 +0100, David Brown wrote:
>
>> ... people write utilities for them in a variety of languages ...
>>
>> But it will often be more convenient to have it built into the language
>> and compiler.
>
> What can be built into the language can only ever be a small subset of
> the many and varied ways that people have incorporated data blobs into
> their programs.

Of course. But that doesn't mean that a language should not include a
feature that makes it easy for a lot of people to get some data blobs
into their code. Maybe /you/ won't find it useful, but other people will.

On 28/02/2024 22:57, Keith Thompson wrote:
> David Brown <david.brown@hesbynett.no> writes:
> [...]
>> They won't use strings, they will use data blobs - binary data. Then
>> there is no issue with null bytes. And yes, implementations will skip
>> the token generation (unless you are doing something weird, such as
>> using #embed to read the parameters to a function call).
>>
>> Tests with prototype implementations gave extremely fast results.
>
> I'm not sure how that would work. #embed is a preprocessor directive,
> and at least in the abstract model it has to expand to valid C code.
>
> I would have expected that it would simply generate the list of
> comma-separated integer constants described in the standard; later
> phases would simply parse that list and generate code as if that
> sequence had been written in the original source file. Do you know of
> an implementation that does something else?
>

The key thing, as I understand it, is that the compiler gets to know
that the integers in the list are all "nice". And since the
preprocessor and the compiler are part of the same implementation (even
if they are separate programs communicating with pipes or temporary
files), the preprocessor could pass on the binary blob in a pre-parsed form.

Think about what a preprocessor and compiler does with the initialisers
in an array, written in normal text (such as by using "xxd -i" or
another external script). For each integer, it has to divide up the
tokens, identify the comma, parse the integer, check that it is a valid
integer, figure out its type based on the size (and suffix, if any). It
needs to record the line number and column number for possible later
reference in error or warning messages. It has to check the value of
the integer against the type for the array elements, and possibly change
the value to suit, or issue warnings for out-of-range values. It has to
allocate all the space to store this information as it goes along,
without knowing the size of the array - so it will be lots of small
mallocs and/or wasted space. It's a /lot/. (Simpler compilers can get
away with a bit less effort, especially if they have more limited warnings.)

With #embed, the preprocessor can generate a compiler-specific "start of
embed" informational directive (much like "#line" directives and such
things generated by preprocessors today), then the data in a very
specific format, then an "end of embed" directive. It could, for
example, generate all the integers in the format "0xAB, " with 16
elements per line. The compiler wouldn't need to parse the data
normally - it knows exactly how many elements there are (from the "start
of embed" directive), it knows exactly where to find each entry (as each
is 6 characters long), it only needs to look at two of these characters,
there's never any errors, the source line number is fixed (at the #embed
line), and so on.

A more tightly coupled preprocessor and compiler can do even better -
for array initialisation, the binary blob could be used directly without
ever generating integer constants or parsing them.

The results of testing are that #embed is /massively/ faster and lower
memory compared to external generators, especially for larger files.
And it gives you the data on-hand for optimisation purposes, unlike
external direct linking of binary blobs. (So you can get the size of
the array, or use values from it as compile-time known values.)

> For example, say you have a file "foo.dat" containing 4 bytes with
> values 0, 1, 2, and 3. This would be perfectly valid:
>
> struct foo {
> unsigned char a;
> unsigned short b;
> unsigned int c;
> double d;
> };
>
> struct foo obj = {
> #embed "foo.dat"
> };
>
> #embed isn't defined to translate an input file to a sequence of bytes.
> It's defined to translate an input file to a sequence of integer
> constant expressions.
>

Yes. But the prime speed (and memory usage) gains come in, are for
large files, and that means array initialisers. That does not conflict
with using it for cases like yours.

> *Maybe* a compiler could optimize for the case where it knows that it's
> being used to initialize an array of unsigned char, but (a) that would
> require the preprocessor to have information that normally doesn't exist
> until later phases, and (b) I'm not convinced it would be worth the
> effort.
>

Look at
<https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1040r6.html#design-practice-speed>.

In those tests, for a 40 MB file gcc #embed is 200 times faster than
"xxd -i" generated files, and takes about 2.5% of the memory. It scales
to 1 GB files. And that's just a proof-of-concept implementation.

On 29/02/2024 02:53, Lawrence D'Oliveiro wrote:
> On Thu, 29 Feb 2024 00:15:17 +0000, bart wrote:
>
>> Or you can use common sense and avoiding writing code which is either
>> too compact or so spread out vertically that you have to hunt for the
>> actual code. Like trying to find the bits of meat in a thin soup.
>
> Terribly sorry about that. I wonder if you could look at this part of the
> same code file:
>
> final android.util.SparseArray<Integer> CodeToIndex =
> new android.util.SparseArray<Integer>();
>
> and show me how to thicken that part of my humble, tasteless gruel? Maybe
> using that same “_” trick you used to do OO in C in your previous example?

You've shown an example of a piece of meat. In main.java, 70% of the
lines are either blanks or contain only an opening or closing bracket.

On 29/02/2024 09:10, David Brown wrote:
> On 28/02/2024 22:57, Keith Thompson wrote:

>> *Maybe* a compiler could optimize for the case where it knows that it's
>> being used to initialize an array of unsigned char, but (a) that would
>> require the preprocessor to have information that normally doesn't exist
>> until later phases, and (b) I'm not convinced it would be worth the
>> effort.
>>
>
> Look at
> <https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1040r6.html#design-practice-speed>.
>
> In those tests, for a 40 MB file gcc #embed is 200 times faster than
> "xxd -i" generated files, and takes about 2.5% of the memory.  It scales
> to 1 GB files.  And that's just a proof-of-concept implementation.

I've just down my own tests, with a 40MB data file containing random
A..Z letters (so can be processed as a text file).

This was converted also to a 120MB text file contain a list of numbers
("65,66,73,...", 3 characters for each data byte).

Using 'strinclude' in my old C compiler, it took about 1 second to build
this program:

#include <stdio.h>
#include <string.h>

char* s=strinclude("data");

int main(void) {
printf("%zu\n", strlen(s));
}

(Running it shows '40000000'.) The same test in my language (which has
no intermediate ASM stage) took 0.3 seconds.

Next I tried instead that 120MB text file containing the same data but
as text, initialising a char[] array using #include.

Tcc took 12 seconds. Bcc took 56 seconds (via ASM etc).

gcc got up to about 3GB memory usage then 'cc1' failed trying to
allocate 0.5GB, after about a minute.

Processing long list of numbers DOES use considerable resources. Bear in
mind that #embed also needs to take binary data and generate tokens,
possibly converting each binary number to text.

On 28/02/2024 21:36, Keith Thompson wrote:
> bart <bc@freeuk.com> writes:
> [...]
>> AFAIK strings in C can have embedded zeros when not assumed to be
>> zero-terminated. So here:
>>
>> char s[]={1,2,3,0,4,5,6};
>>
>> s will have a length of 7.
>
> Strings *by definition* cannot have embedded zeros. A null character
> terminates a string.
>
C strings. Not strings in other programming languages. And only if you
define "C strings" in a rather restrictive but, to be fair, totally
legitimate way. So I wouldn't have put in the asterisks.

--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On 28/02/2024 21:36, Keith Thompson wrote:
> bart <bc@freeuk.com> writes:
> [...]
>> AFAIK strings in C can have embedded zeros when not assumed to be
>> zero-terminated. So here:
>>
>> char s[]={1,2,3,0,4,5,6};
>>
>> s will have a length of 7.
>
> Strings *by definition* cannot have embedded zeros. A null character
> terminates a string.
>
> A string literal can have embedded \0 characters, but if you're
> suggesting that #embed should expand to a string literal, I can see
> several disadvantages and no significant advantages. For one thing, the
> data may or may not end with a null character; string literals always
> do.

Not here:

char s[] = "ABC";
char t[3] = "DEF";

The "DEF" string doesn't end with a zero.

Is 'string' given a special meaning in the standard?

/That/ would seem to me to be too restrictive. Does this:

char *s;

define a pointer to a such string, or can it be any kind of data? For
example, `char*` is used by the GetOpenFileName WinAPI function for a
/series/ of zero-terminated strings which itself is terminated with two
zero bytes.

So it is some property that is attributed to the data that will be stored.

I normally use `cstring` or `stringz` outside the language when refering
to a zero-terminated sequences of characters, which implies that
embedded zeros aren't allowed.

On 29/02/2024 12:56, Malcolm McLean wrote:
> On 28/02/2024 21:36, Keith Thompson wrote:
>> bart <bc@freeuk.com> writes:
>> [...]
>>> AFAIK strings in C can have embedded zeros when not assumed to be
>>> zero-terminated. So here:
>>>
>>>      char s[]={1,2,3,0,4,5,6};
>>>
>>> s will have a length of 7.
>>
>> Strings *by definition* cannot have embedded zeros.  A null character
>> terminates a string.
>>
> C strings. Not strings in other programming languages.

Let me point you to the name of this Usenet group.

And strings in any programming language have either :

1. A string of characters and a terminating null, thus no embedded null
characters.
2. A starting length (such as Pascal strings).
3. A fixed size.
4. A more advanced structure.

An array of bytes is not a "string".

> And only if you
> define "C strings" in a rather restrictive but, to be fair, totally
> legitimate way. So I wouldn't have put in the asterisks.
>

The definition of "C string" is given in section 7.1.1p1 of the C
standards. There is only one definition of "C string".

On 29/02/2024 14:31, bart wrote:
> On 28/02/2024 21:36, Keith Thompson wrote:
>> bart <bc@freeuk.com> writes:
>> [...]
>>> AFAIK strings in C can have embedded zeros when not assumed to be
>>> zero-terminated. So here:
>>>
>>>      char s[]={1,2,3,0,4,5,6};
>>>
>>> s will have a length of 7.
>>
>> Strings *by definition* cannot have embedded zeros.  A null character
>> terminates a string.
>>
>> A string literal can have embedded \0 characters, but if you're
>> suggesting that #embed should expand to a string literal, I can see
>> several disadvantages and no significant advantages.  For one thing, the
>> data may or may not end with a null character; string literals always
>> do.
>
> Not here:
>
>     char s[]  = "ABC";
>     char t[3] = "DEF";
>
> The "DEF" string doesn't end with a zero.

And is, therefore, not a string.

>
> Is 'string' given a special meaning in the standard?

Yes. Things that work with the strX functions. Which means they are
'\0' terminated.

>
> /That/ would seem to me to be too restrictive. Does this:
>
>    char *s;
>
> define a pointer to a such string, or can it be any kind of data? For

That points to a char. That could be followed by more chars and it one
of those is a '\0', then it's a string. You know this.

> example, `char*` is used by the GetOpenFileName WinAPI function for a
> /series/ of zero-terminated strings which itself is terminated with two
> zero bytes.

That is a windowsism, then.

Why didn't they use the NULL terminated char **argv kind of thing?

>
> So it is some property that is attributed to the data that will be stored.
>
> I normally use `cstring` or `stringz` outside the language when refering
> to a zero-terminated sequences of characters, which implies that
> embedded zeros aren't allowed.
>
>
>
>

On 2/29/24 01:47, bart wrote:

> My early comments on this were about compiler performance. I suggested
> there might be a way to turn 100,000 byte values in a file, directly
> into a 100KB string or data block, without needing to first convert
> 100,000 values into 100,000 integer expressions representated as tokens,
> and to then parse those 100,000 expressions into AST nodes etc.

But you HAVE to do that il #embed is in the preprocessor,
because his job is to give compilable text to the real
compiler. No other way is possible.

> Basically, #embed is dumb.

No.

--
+---------------------------------------------------------------------+
| https://tube.interhacker.space/a/tth/video-channels |
+---------------------------------------------------------------------+