On 29/02/2024 15:31, bart wrote:
> On 28/02/2024 21:36, Keith Thompson wrote:
>> bart <bc@freeuk.com> writes:
>> [...]
>>> AFAIK strings in C can have embedded zeros when not assumed to be
>>> zero-terminated. So here:
>>>
>>>      char s[]={1,2,3,0,4,5,6};
>>>
>>> s will have a length of 7.
>>
>> Strings *by definition* cannot have embedded zeros.  A null character
>> terminates a string.
>>
>> A string literal can have embedded \0 characters, but if you're
>> suggesting that #embed should expand to a string literal, I can see
>> several disadvantages and no significant advantages.  For one thing, the
>> data may or may not end with a null character; string literals always
>> do.
>
> Not here:
>
>     char s[]  = "ABC";

"ABC" is a "string literal". Once things like concatenation of adjacent
strings, macro expansion, etc., is complete, a null character is
appended to it. Then it is used as an initialiser for the array "s".
After initialisation, "s" is an array of 4 chars and contains a string.

(Note - a "string literal" might not be a "string", because string
literals can contain embedded nulls. This is a footnote in 6.4.5
describing string literals.)

>     char t[3] = "DEF";
>
> The "DEF" string doesn't end with a zero.

"DEF" is not a "string" - it is a "string literal". It does get a null
character appended during translation phase 7. But only the first three
characters - therefore not including the null character - get copied to
"t" during the initialisation of "t". "t" is an array of 3 chars, and
it does not contain a string.

>
> Is 'string' given a special meaning in the standard?
>

Yes. See 7.1.1p1.

> /That/ would seem to me to be too restrictive. Does this:
>
>    char *s;
>
> define a pointer to a such string, or can it be any kind of data? For
> example, `char*` is used by the GetOpenFileName WinAPI function for a
> /series/ of zero-terminated strings which itself is terminated with two
> zero bytes.
>

"char *s;" declares a pointer to a char, or a pointer to the start of an
array of char. It is /not/ a string, or a pointer to a string. C
strings are values, and exist at run-time - they are not types. So "s"
can point to a string, or a char (which will be a string if and only if
it is a null character), or an array of chars (which may or may not
contain a string), or it could point to anything else.

> So it is some property that is attributed to the data that will be stored.

Yes.

>
> I normally use `cstring` or `stringz` outside the language when refering
> to a zero-terminated sequences of characters, which implies that
> embedded zeros aren't allowed.
>

That makes sense. Different languages have different ways of holding
sequences of characters (generic "string" data), so you need to qualify
the term if it is not clear from the context.

But when we are discussing C, and there is no other qualification,
"string" means "C string" - the definition of "string" given in the C
standards.

On 2/29/24 11:18, bart wrote:
> Using 'strinclude' in my old C compiler, it took about 1 second to build
> this program:
>
>   #include <stdio.h>
>   #include <string.h>
>
>   char* s=strinclude("data");
>
>   int main(void) {
>      printf("%zu\n", strlen(s));
>  }

tth@redlady:~/Desktop$ man strinclude
No manual entry for strinclude
tth@redlady:~/Desktop$

--
+---------------------------------------------------------------------+
| https://tube.interhacker.space/a/tth/video-channels |
+---------------------------------------------------------------------+

bart <bc@freeuk.com> writes:
>On 28/02/2024 23:52, Lawrence D'Oliveiro wrote:
>> On Wed, 28 Feb 2024 21:34:14 +0000, bart wrote:
>>
>>> In C:
>>>
>>> void Add(int CategoryCode, ItemType Item) {
>>> CodeToIndex_put(CategoryCode, getCount());
>>> add(Item);
>>> }
>>>
>>> 4 non-comment lines versus 9. I know Java needs tons of boilerplate, but
>>> but it is not all the language's fault.
>>
>> Or how about
>>
>> void Add(int CategoryCode, ItemType Item) {CodeToIndex_put(CategoryCode, getCount());add(Item);}
>>
>> Wow! I never realized you could do that in C!! I thought it was an
>> error to put stuff after column 72 or something. Thanks for the tip!!!
>
>Well, you could write an entire program on one line.

int main(int b,char**i){long long n=B,a=I^n,r=(a/b&a)>>4,y=atoi(*++i),_=(((a^n/b)*(y>>T)|y>>S)&r)|(a^r);printf("%.8s\n",(char*)&_);}

(A winner from the obfuscated C contest).

bart <bc@freeuk.com> writes:
>On 28/02/2024 23:31, Keith Thompson wrote:
>> bart <bc@freeuk.com> writes:
>
>>> It would be unfortunate if your example was allowed. Clearly a binary
>>> representation of an instance of your struct would probably require 16
>>> bytes rather than 4, of which one may be padding.
>>
>> Depending on the sizes and alignments of the various types, sure.
>> So what?
>>
>
>> If you have suggestions for alternate ways to define #embed, they might
>> be interesting, but it's too late to change the existing specification.
>>
>
>My early comments on this were about compiler performance. I suggested
>there might be a way to turn 100,000 byte values in a file, directly
>into a 100KB string or data block, without needing to first convert
>100,000 values into 100,000 integer expressions representated as tokens,
>and to then parse those 100,000 expressions into AST nodes etc.
>
>DB suggested something like that was actually done. But you can't do
>that if those 100,000 numbers represent from 100KB to 800KB of memory
>depending on the data type of the strucure they're initialising.

An implementation is free to simply pass a variant (or the directive
itself) of #embed from the pre-processor to the compiler if the programmer
isn't using -E, and the compiler could simply copy the embedded file
into the object file directly, without processing it as a series of
integer values. Much like the #file and #line directives passed by
the pre-processor to the compiler.

On 29.02.2024 16:48, Scott Lurndal wrote:
>
> int main(int b,char**i){long long n=B,a=I^n,r=(a/b&a)>>4,y=atoi(*++i),_=(((a^n/b)*(y>>T)|y>>S)&r)|(a^r);printf("%.8s\n",(char*)&_);}

What does it do?

What preconditions must be fulfilled or what additions
does it need to compile?

>
> (A winner from the obfuscated C contest).

(Are non-compiling C sources allowed in the contest?)

Janis

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
> On 28/02/2024 21:36, Keith Thompson wrote:
>> bart <bc@freeuk.com> writes:
>> [...]
>>> AFAIK strings in C can have embedded zeros when not assumed to be
>>> zero-terminated. So here:
>>>
>>> char s[]={1,2,3,0,4,5,6};
>>>
>>> s will have a length of 7.
>> Strings *by definition* cannot have embedded zeros. A null
>> character
>> terminates a string.
>>
> C strings. Not strings in other programming languages. And only if you
> define "C strings" in a rather restrictive but, to be fair, totally
> legitimate way. So I wouldn't have put in the asterisks.

Yes, C strings. Or has we call them here in comp.lang.c, strings.

Sheesh.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

tTh <tth@none.invalid> writes:
>On 2/29/24 01:47, bart wrote:
>
>> My early comments on this were about compiler performance. I suggested
>> there might be a way to turn 100,000 byte values in a file, directly
>> into a 100KB string or data block, without needing to first convert
>> 100,000 values into 100,000 integer expressions representated as tokens,
>> and to then parse those 100,000 expressions into AST nodes etc.
>
> But you HAVE to do that il #embed is in the preprocessor,
> because his job is to give compilable text to the real
> compiler. No other way is possible.

The standard does not require the preprocessor to be
separate from a 'real' compiler. It's acceptable for an implementation
to implement both in a single executable. Absent -E, the
preprocessor and compiler can cooperate to efficiently handle
#embed without generating parseable C code.

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>On 29.02.2024 16:48, Scott Lurndal wrote:
>>
>> int main(int b,char**i){long long n=B,a=I^n,r=(a/b&a)>>4,y=atoi(*++i),_=(((a^n/b)*(y>>T)|y>>S)&r)|(a^r);printf("%.8s\n",(char*)&_);}
>
>What does it do?
>
>What preconditions must be fulfilled or what additions
>does it need to compile?
>
>>
>> (A winner from the obfuscated C contest).
>
>(Are non-compiling C sources allowed in the contest?)

https://www.ioccc.org/years.html

The above is from 'burton'.

bart <bc@freeuk.com> writes:
> On 28/02/2024 21:36, Keith Thompson wrote:
>> bart <bc@freeuk.com> writes:
>> [...]
>>> AFAIK strings in C can have embedded zeros when not assumed to be
>>> zero-terminated. So here:
>>>
>>> char s[]={1,2,3,0,4,5,6};
>>>
>>> s will have a length of 7.
>> Strings *by definition* cannot have embedded zeros. A null
>> character
>> terminates a string.
>> A string literal can have embedded \0 characters, but if you're
>> suggesting that #embed should expand to a string literal, I can see
>> several disadvantages and no significant advantages. For one thing, the
>> data may or may not end with a null character; string literals always
>> do.
>
> Not here:
>
> char s[] = "ABC";
> char t[3] = "DEF";
>
> The "DEF" string doesn't end with a zero.

That's not strictly true, nor is it relevant.

For #embed, you generally don't know the length of the resulting
sequence of values, so you can't usually write something like:

const unsigned char data[N] = {
#embed "foo.dat"
};

And the way it's described in the standard is that "DEF" refers to an
array object that does include a trailing null byte, but only the first
3 characters are used to initialize t. Of course a compiler can
optimize that one byte away.

> Is 'string' given a special meaning in the standard?

Of course it is. Surely you know that. 7.1.1 paragraph 1.

"abc\0def" is a valid string literal, but its value is not a string.
(No, the standard doesn't say that the value of a string literal is a
string.)

[...]

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

David Brown <david.brown@hesbynett.no> writes:
[...]
> "char *s;" declares a pointer to a char, or a pointer to the start of
> an array of char.

"char *s;" defines a pointer to char. At runtime, it may or may not
point to the initial element of an array object. (And for most
purposes, a char object is treated as 1-element array of char.)

> It is /not/ a string, or a pointer to a string.

It may be a pointer to a string at run time, depending on what it
currently points to.

> C strings are values, and exist at run-time - they are not types. So
> "s" can point to a string, or a char (which will be a string if and
> only if it is a null character), or an array of chars (which may or
> may not contain a string), or it could point to anything else.

"s" can point to *the initial element of* an array of chars. if that
array contains a string, then (the value of) s is by definition a
pointer to a string.

[...]

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

David Brown <david.brown@hesbynett.no> writes:
> On 28/02/2024 22:57, Keith Thompson wrote:
>> David Brown <david.brown@hesbynett.no> writes:
>> [...]
>>> They won't use strings, they will use data blobs - binary data. Then
>>> there is no issue with null bytes. And yes, implementations will skip
>>> the token generation (unless you are doing something weird, such as
>>> using #embed to read the parameters to a function call).
>>>
>>> Tests with prototype implementations gave extremely fast results.
>> I'm not sure how that would work. #embed is a preprocessor
>> directive,
>> and at least in the abstract model it has to expand to valid C code.
>> I would have expected that it would simply generate the list of
>> comma-separated integer constants described in the standard; later
>> phases would simply parse that list and generate code as if that
>> sequence had been written in the original source file. Do you know of
>> an implementation that does something else?
>
> The key thing, as I understand it, is that the compiler gets to know
> that the integers in the list are all "nice". And since the
> preprocessor and the compiler are part of the same implementation
> (even if they are separate programs communicating with pipes or
> temporary files), the preprocessor could pass on the binary blob in a
> pre-parsed form.
[...]

Sure, an implementation *could* optimize #embed so it expands to some
implementation-defined nonstandard form that later phases can treat as
raw data. But since it's defined as a preprocessor directive, it's
difficult to see how it could do so while covering all cases.

[...]

> The results of testing are that #embed is /massively/ faster and lower
> memory compared to external generators, especially for larger files.
> And it gives you the data on-hand for optimisation purposes, unlike
> external direct linking of binary blobs. (So you can get the size of
> the array, or use values from it as compile-time known values.)

What testing? The very latest versions of gcc and clang (I checked both
their git repos yesterday) do not yet implement #embed.

>> For example, say you have a file "foo.dat" containing 4 bytes with
>> values 0, 1, 2, and 3. This would be perfectly valid:
>> struct foo {
>> unsigned char a;
>> unsigned short b;
>> unsigned int c;
>> double d;
>> };
>> struct foo obj = {
>> #embed "foo.dat"
>> };
>> #embed isn't defined to translate an input file to a sequence of
>> bytes.
>> It's defined to translate an input file to a sequence of integer
>> constant expressions.
>
> Yes. But the prime speed (and memory usage) gains come in, are for
> large files, and that means array initialisers. That does not
> conflict with using it for cases like yours.

So a compiler that does this would have to be able to handle

struct foo obj = {
#blob
<binary data>
#endblob>
};

and initialize a, b, c, and d to 0, 1, 2, and 3.0, respectively from
successive bytes of the binary data. Either that, or the preprocessor
would have to use information it doesn't have to determine how to expand
#embed.

>> *Maybe* a compiler could optimize for the case where it knows that it's
>> being used to initialize an array of unsigned char, but (a) that would
>> require the preprocessor to have information that normally doesn't exist
>> until later phases, and (b) I'm not convinced it would be worth the
>> effort.
>
> Look at
> <https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1040r6.html#design-practice-speed>.
>
> In those tests, for a 40 MB file gcc #embed is 200 times faster than
> "xxd -i" generated files, and takes about 2.5% of the memory. It
> scales to 1 GB files. And that's just a proof-of-concept
> implementation.

That's for std::embed, a proposed C++ feature that's *not* defined as a
preprocessor directive. Sample usage from the paper:

constexpr std::span<const std::byte> fxaa_binary =
std::embed( "fxaa.spirv" );

So the compiler knows the type of the object being initialized.

(Note that the author of that C++ paper is also the editor for the C
standard.)

I'm still skeptical that C's #embed will actually be implemented other
than as expanding to a sequence of integer constants.

On the other hand, C23 allows for additional implementation-defined
parameters to #embed (as well as the standard embed parameters limit,
prefix, suffix, and is_empty). Such a parameter could specify how it's
expanded, perhaps to some implementation-defined blob format. *If*
compilers optimize #embed to something other than a sequence of integer
constant expressions, that's probably how it would be done. But since
neither gcc nor clang implements #embed at all, it may be too early to
speculate.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

scott@slp53.sl.home (Scott Lurndal) writes:
> bart <bc@freeuk.com> writes:
>>On 28/02/2024 23:31, Keith Thompson wrote:
>>> bart <bc@freeuk.com> writes:
>>
>>>> It would be unfortunate if your example was allowed. Clearly a binary
>>>> representation of an instance of your struct would probably require 16
>>>> bytes rather than 4, of which one may be padding.
>>>
>>> Depending on the sizes and alignments of the various types, sure.
>>> So what?
>>>
>>
>>> If you have suggestions for alternate ways to define #embed, they might
>>> be interesting, but it's too late to change the existing specification.
>>>
>>
>>My early comments on this were about compiler performance. I suggested
>>there might be a way to turn 100,000 byte values in a file, directly
>>into a 100KB string or data block, without needing to first convert
>>100,000 values into 100,000 integer expressions representated as tokens,
>>and to then parse those 100,000 expressions into AST nodes etc.
>>
>>DB suggested something like that was actually done. But you can't do
>>that if those 100,000 numbers represent from 100KB to 800KB of memory
>>depending on the data type of the strucure they're initialising.
>
> An implementation is free to simply pass a variant (or the directive
> itself) of #embed from the pre-processor to the compiler if the programmer
> isn't using -E, and the compiler could simply copy the embedded file
> into the object file directly, without processing it as a series of
> integer values. Much like the #file and #line directives passed by
> the pre-processor to the compiler.

Sure, an implementation has to operate *as if* it implemented the 8
translation phases separately. But given a structure initialized with
#embed, it would have to generate additional code to initialize the
structure members from the bytes of the binary blob.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

On 29.02.2024 17:17, Scott Lurndal wrote:
> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>> On 29.02.2024 16:48, Scott Lurndal wrote:
>>>
>>> int main(int b,char**i){long long n=B,a=I^n,r=(a/b&a)>>4,y=atoi(*++i),_=(((a^n/b)*(y>>T)|y>>S)&r)|(a^r);printf("%.8s\n",(char*)&_);}
>>
>> What does it do?
>>
>> What preconditions must be fulfilled or what additions
>> does it need to compile?

With the link below I see it "needs" a 600+ lines long Makefile.

And I see there's some variable definitions with magic numbers
passed.

>>
>>>
>>> (A winner from the obfuscated C contest).
>>
>> (Are non-compiling C sources allowed in the contest?)
>
> https://www.ioccc.org/years.html
>
> The above is from 'burton'.

Thanks.

Janis

On 29.02.2024 17:18, Keith Thompson wrote:
>
> "abc\0def" is a valid string literal, but its value is not a string.
> (No, the standard doesn't say that the value of a string literal is a
> string.)

This sounds somewhat strange in my ears. Usually a literal for a type
will constitute an instance of the type. - I suppose the irregularity
stems from the fact that there's no explicit string object type in C.

Janis

scott@slp53.sl.home (Scott Lurndal) writes:
> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>>On 29.02.2024 16:48, Scott Lurndal wrote:
>>>
>>> int main(int b,char**i){long long n=B,a=I^n,r=(a/b&a)>>4,y=atoi(*++i),_=(((a^n/b)*(y>>T)|y>>S)&r)|(a^r);printf("%.8s\n",(char*)&_);}
>>
>>What does it do?
>>
>>What preconditions must be fulfilled or what additions
>>does it need to compile?
>>
>>>
>>> (A winner from the obfuscated C contest).
>>
>>(Are non-compiling C sources allowed in the contest?)
>
> https://www.ioccc.org/years.html
>
> The above is from 'burton'.

"burton" has submitted a number of entries over the years. This one is
https://www.ioccc.org/2020/burton/index.html

The program by itself does not compile. The provided Makefile passes
additional arguments, with B and I being defined differently in
big-endian vs. little-endian machines.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
> On 29.02.2024 17:18, Keith Thompson wrote:
>> "abc\0def" is a valid string literal, but its value is not a string.
>> (No, the standard doesn't say that the value of a string literal is a
>> string.)
>
> This sounds somewhat strange in my ears. Usually a literal for a type
> will constitute an instance of the type. - I suppose the irregularity
> stems from the fact that there's no explicit string object type in C.

Exactly, "string" is not a type.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:
>scott@slp53.sl.home (Scott Lurndal) writes:
>> bart <bc@freeuk.com> writes:
>>>On 28/02/2024 23:31, Keith Thompson wrote:
>>>> bart <bc@freeuk.com> writes:
>>>
>>>>> It would be unfortunate if your example was allowed. Clearly a binary
>>>>> representation of an instance of your struct would probably require 16
>>>>> bytes rather than 4, of which one may be padding.
>>>>
>>>> Depending on the sizes and alignments of the various types, sure.
>>>> So what?
>>>>
>>>
>>>> If you have suggestions for alternate ways to define #embed, they might
>>>> be interesting, but it's too late to change the existing specification.
>>>>
>>>
>>>My early comments on this were about compiler performance. I suggested
>>>there might be a way to turn 100,000 byte values in a file, directly
>>>into a 100KB string or data block, without needing to first convert
>>>100,000 values into 100,000 integer expressions representated as tokens,
>>>and to then parse those 100,000 expressions into AST nodes etc.
>>>
>>>DB suggested something like that was actually done. But you can't do
>>>that if those 100,000 numbers represent from 100KB to 800KB of memory
>>>depending on the data type of the strucure they're initialising.
>>
>> An implementation is free to simply pass a variant (or the directive
>> itself) of #embed from the pre-processor to the compiler if the programmer
>> isn't using -E, and the compiler could simply copy the embedded file
>> into the object file directly, without processing it as a series of
>> integer values. Much like the #file and #line directives passed by
>> the pre-processor to the compiler.
>
>Sure, an implementation has to operate *as if* it implemented the 8
>translation phases separately. But given a structure initialized with
>#embed, it would have to generate additional code to initialize the
>structure members from the bytes of the binary blob.

Would it? Or could it simply assume that the binary blob
is already in the same binary format that writing an instance
of the structure from a C application on the same host would have created?

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>On 29.02.2024 17:17, Scott Lurndal wrote:
>> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>>> On 29.02.2024 16:48, Scott Lurndal wrote:
>>>>
>>>> int main(int b,char**i){long long n=B,a=I^n,r=(a/b&a)>>4,y=atoi(*++i),_=(((a^n/b)*(y>>T)|y>>S)&r)|(a^r);printf("%.8s\n",(char*)&_);}
>>>
>>> What does it do?
>>>
>>> What preconditions must be fulfilled or what additions
>>> does it need to compile?
>
>With the link below I see it "needs" a 600+ lines long Makefile.

The readme simply says compile it and run it
as ./prog <value between 1 and 512>.

On 29/02/2024 18:28, Scott Lurndal wrote:
> Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:
>> scott@slp53.sl.home (Scott Lurndal) writes:
>>> bart <bc@freeuk.com> writes:
>>>> On 28/02/2024 23:31, Keith Thompson wrote:
>>>>> bart <bc@freeuk.com> writes:
>>>>
>>>>>> It would be unfortunate if your example was allowed. Clearly a binary
>>>>>> representation of an instance of your struct would probably require 16
>>>>>> bytes rather than 4, of which one may be padding.
>>>>>
>>>>> Depending on the sizes and alignments of the various types, sure.
>>>>> So what?
>>>>>
>>>>
>>>>> If you have suggestions for alternate ways to define #embed, they might
>>>>> be interesting, but it's too late to change the existing specification.
>>>>>
>>>>
>>>> My early comments on this were about compiler performance. I suggested
>>>> there might be a way to turn 100,000 byte values in a file, directly
>>>> into a 100KB string or data block, without needing to first convert
>>>> 100,000 values into 100,000 integer expressions representated as tokens,
>>>> and to then parse those 100,000 expressions into AST nodes etc.
>>>>
>>>> DB suggested something like that was actually done. But you can't do
>>>> that if those 100,000 numbers represent from 100KB to 800KB of memory
>>>> depending on the data type of the strucure they're initialising.
>>>
>>> An implementation is free to simply pass a variant (or the directive
>>> itself) of #embed from the pre-processor to the compiler if the programmer
>>> isn't using -E, and the compiler could simply copy the embedded file
>>> into the object file directly, without processing it as a series of
>>> integer values. Much like the #file and #line directives passed by
>>> the pre-processor to the compiler.
>>
>> Sure, an implementation has to operate *as if* it implemented the 8
>> translation phases separately. But given a structure initialized with
>> #embed, it would have to generate additional code to initialize the
>> structure members from the bytes of the binary blob.
>
> Would it? Or could it simply assume that the binary blob
> is already in the same binary format that writing an instance
> of the structure from a C application on the same host would have created?

That would depend on the sizes of the fields in the struct, and the size
of the integer constants in the #embed. The norm for #embed will be
unsigned char integer constants, so it will only be a direct fit for the
binary representation of the struct if all the struct fields are
compatible with that. But a compiler could have vendor parameters on
the #embed to change those sizes.

David Brown <david.brown@hesbynett.no> writes:
>On 29/02/2024 18:28, Scott Lurndal wrote:
>> Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:
>>> scott@slp53.sl.home (Scott Lurndal) writes:
>>>> bart <bc@freeuk.com> writes:
>>>>> On 28/02/2024 23:31, Keith Thompson wrote:
>>>>>> bart <bc@freeuk.com> writes:
>>>>>
>>>>>>> It would be unfortunate if your example was allowed. Clearly a binary
>>>>>>> representation of an instance of your struct would probably require 16
>>>>>>> bytes rather than 4, of which one may be padding.
>>>>>>
>>>>>> Depending on the sizes and alignments of the various types, sure.
>>>>>> So what?
>>>>>>
>>>>>
>>>>>> If you have suggestions for alternate ways to define #embed, they might
>>>>>> be interesting, but it's too late to change the existing specification.
>>>>>>
>>>>>
>>>>> My early comments on this were about compiler performance. I suggested
>>>>> there might be a way to turn 100,000 byte values in a file, directly
>>>>> into a 100KB string or data block, without needing to first convert
>>>>> 100,000 values into 100,000 integer expressions representated as tokens,
>>>>> and to then parse those 100,000 expressions into AST nodes etc.
>>>>>
>>>>> DB suggested something like that was actually done. But you can't do
>>>>> that if those 100,000 numbers represent from 100KB to 800KB of memory
>>>>> depending on the data type of the strucure they're initialising.
>>>>
>>>> An implementation is free to simply pass a variant (or the directive
>>>> itself) of #embed from the pre-processor to the compiler if the programmer
>>>> isn't using -E, and the compiler could simply copy the embedded file
>>>> into the object file directly, without processing it as a series of
>>>> integer values. Much like the #file and #line directives passed by
>>>> the pre-processor to the compiler.
>>>
>>> Sure, an implementation has to operate *as if* it implemented the 8
>>> translation phases separately. But given a structure initialized with
>>> #embed, it would have to generate additional code to initialize the
>>> structure members from the bytes of the binary blob.
>>
>> Would it? Or could it simply assume that the binary blob
>> is already in the same binary format that writing an instance
>> of the structure from a C application on the same host would have created?
>
>That would depend on the sizes of the fields in the struct, and the size
>of the integer constants in the #embed.

I'm embedding a binary file. I want the representation in memory
to be _exactly_ the same as in the file, regardless of how it is
defined in the C code (array of char, array of int, array of long, struct whatever).

On 29/02/2024 18:03, Keith Thompson wrote:
> David Brown <david.brown@hesbynett.no> writes:
>> On 28/02/2024 22:57, Keith Thompson wrote:
>>> David Brown <david.brown@hesbynett.no> writes:
>>> [...]
>>>> They won't use strings, they will use data blobs - binary data. Then
>>>> there is no issue with null bytes. And yes, implementations will skip
>>>> the token generation (unless you are doing something weird, such as
>>>> using #embed to read the parameters to a function call).
>>>>
>>>> Tests with prototype implementations gave extremely fast results.
>>> I'm not sure how that would work. #embed is a preprocessor
>>> directive,
>>> and at least in the abstract model it has to expand to valid C code.
>>> I would have expected that it would simply generate the list of
>>> comma-separated integer constants described in the standard; later
>>> phases would simply parse that list and generate code as if that
>>> sequence had been written in the original source file. Do you know of
>>> an implementation that does something else?
>>
>> The key thing, as I understand it, is that the compiler gets to know
>> that the integers in the list are all "nice". And since the
>> preprocessor and the compiler are part of the same implementation
>> (even if they are separate programs communicating with pipes or
>> temporary files), the preprocessor could pass on the binary blob in a
>> pre-parsed form.
> [...]
>
> Sure, an implementation *could* optimize #embed so it expands to some
> implementation-defined nonstandard form that later phases can treat as
> raw data. But since it's defined as a preprocessor directive, it's
> difficult to see how it could do so while covering all cases.
>

It would require a strong link between the compiler and the preprocessor
- as you know, these don't have to be separate programs. In a more
weakly coupled system, there could still be a method for passing a
binary blob to the compiler in addition to the integer data, and let the
compiler use whichever form it preferred (based on what your code does
with the data).

> [...]
>
>> The results of testing are that #embed is /massively/ faster and lower
>> memory compared to external generators, especially for larger files.
>> And it gives you the data on-hand for optimisation purposes, unlike
>> external direct linking of binary blobs. (So you can get the size of
>> the array, or use values from it as compile-time known values.)
>
> What testing? The very latest versions of gcc and clang (I checked both
> their git repos yesterday) do not yet implement #embed.
>

I believe prototypes, tests, or proofs of concept have been made for
gcc, clang and perhaps other tools. I posted a link to some results -
more are floating around the internet if you want to look for them.

>>> For example, say you have a file "foo.dat" containing 4 bytes with
>>> values 0, 1, 2, and 3. This would be perfectly valid:
>>> struct foo {
>>> unsigned char a;
>>> unsigned short b;
>>> unsigned int c;
>>> double d;
>>> };
>>> struct foo obj = {
>>> #embed "foo.dat"
>>> };
>>> #embed isn't defined to translate an input file to a sequence of
>>> bytes.
>>> It's defined to translate an input file to a sequence of integer
>>> constant expressions.
>>
>> Yes. But the prime speed (and memory usage) gains come in, are for
>> large files, and that means array initialisers. That does not
>> conflict with using it for cases like yours.
>
> So a compiler that does this would have to be able to handle
>
> struct foo obj = {
> #blob
> <binary data>
> #endblob>
> };
>
> and initialize a, b, c, and d to 0, 1, 2, and 3.0, respectively from
> successive bytes of the binary data. Either that, or the preprocessor
> would have to use information it doesn't have to determine how to expand
> #embed.
>

I think I've covered how that could be handled. (And I don't know how
it /will/ be handled. But I am sure compiler implementers will figure a
way to make it work correctly for any use of the integer constant list,
while also making it as efficient as they reasonably can for the common
case of initialising an unsigned char array.)

>>> *Maybe* a compiler could optimize for the case where it knows that it's
>>> being used to initialize an array of unsigned char, but (a) that would
>>> require the preprocessor to have information that normally doesn't exist
>>> until later phases, and (b) I'm not convinced it would be worth the
>>> effort.
>>
>> Look at
>> <https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1040r6.html#design-practice-speed>.
>>
>> In those tests, for a 40 MB file gcc #embed is 200 times faster than
>> "xxd -i" generated files, and takes about 2.5% of the memory. It
>> scales to 1 GB files. And that's just a proof-of-concept
>> implementation.
>
> That's for std::embed, a proposed C++ feature that's *not* defined as a
> preprocessor directive. Sample usage from the paper:
>
> constexpr std::span<const std::byte> fxaa_binary =
> std::embed( "fxaa.spirv" );
>
> So the compiler knows the type of the object being initialized.
>
> (Note that the author of that C++ paper is also the editor for the C
> standard.)

The work on #embed is being done simultaneously for C and C++.
std::embed() gives you slightly different way to write it, but the
implementation is the same. (Not unlike _Pragma and #pragma in C.)

Other pages I have seen with speed tests show the same pattern while
referring explicitly to #embed.

>
> I'm still skeptical that C's #embed will actually be implemented other
> than as expanding to a sequence of integer constants.
>

We'll see when it all hits the mainline compilers!

> On the other hand, C23 allows for additional implementation-defined
> parameters to #embed (as well as the standard embed parameters limit,
> prefix, suffix, and is_empty). Such a parameter could specify how it's
> expanded, perhaps to some implementation-defined blob format. *If*
> compilers optimize #embed to something other than a sequence of integer
> constant expressions, that's probably how it would be done. But since
> neither gcc nor clang implements #embed at all, it may be too early to
> speculate.
>

scott@slp53.sl.home (Scott Lurndal) writes:
>David Brown <david.brown@hesbynett.no> writes:
>>On 29/02/2024 18:28, Scott Lurndal wrote:
>>> Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:
>>>> scott@slp53.sl.home (Scott Lurndal) writes:

>>>>> An implementation is free to simply pass a variant (or the directive
>>>>> itself) of #embed from the pre-processor to the compiler if the programmer
>>>>> isn't using -E, and the compiler could simply copy the embedded file
>>>>> into the object file directly, without processing it as a series of
>>>>> integer values. Much like the #file and #line directives passed by
>>>>> the pre-processor to the compiler.
>>>>
>>>> Sure, an implementation has to operate *as if* it implemented the 8
>>>> translation phases separately. But given a structure initialized with
>>>> #embed, it would have to generate additional code to initialize the
>>>> structure members from the bytes of the binary blob.
>>>
>>> Would it? Or could it simply assume that the binary blob
>>> is already in the same binary format that writing an instance
>>> of the structure from a C application on the same host would have created?
>>
>>That would depend on the sizes of the fields in the struct, and the size
>>of the integer constants in the #embed.
>
>I'm embedding a binary file. I want the representation in memory
>to be _exactly_ the same as in the file, regardless of how it is
>defined in the C code (array of char, array of int, array of long, struct whatever).
>

I have an actual use case today where #embed of a (C++) std::map binary
object created by separate tool would be very useful. I'm
planning on using mmap to load it at runtime at the moment.

On 2024-02-29, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>> On 29.02.2024 17:18, Keith Thompson wrote:
>>> "abc\0def" is a valid string literal, but its value is not a string.
>>> (No, the standard doesn't say that the value of a string literal is a
>>> string.)
>>
>> This sounds somewhat strange in my ears. Usually a literal for a type
>> will constitute an instance of the type. - I suppose the irregularity
>> stems from the fact that there's no explicit string object type in C.
>
> Exactly, "string" is not a type.

It is a type in the broader sense, in that is a logical proposition
about the attributes of an object that is true or false.

It's just not a type in the C static type system.

What that means is that there does not exist a constraint rule in
standard C requiring some expression or object to conform to the string
type. The concept "string" is not represented in the constraint system.
But it is a type concept.

(There are rules that require a string, but they are not constraint
rules. E.g. if strlen is given an argument which isn't a string, the
behavior is undefined.)

Consider:
char a[3] = "abc";
size_t l = strlen(a);

In the unlikely event that this example would capture the attention of a
computer scientist who researches type systems, he or she would identify
that as having a type error. (One that the C type system is too weak to
model.)

"Upper case letter" is also a type; that's why the header is called
<ctype.h>.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

On 2/29/24 14:26, Kaz Kylheku wrote:
> On 2024-02-29, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
....>> Exactly, "string" is not a type.
>
> It is a type in the broader sense, in that is a logical proposition
> about the attributes of an object that is true or false.

If I defined something to be a sequence of floating point numbers
terminated by a NaN, would that thing also qualify as a type, according
to the definition you're using?

Could you give a source for the definition of "type" that you're using?
Can you use the word "type" in a statement whose truth relies upon the
difference between that definition and the way that "type" is defined by
the C standard? Preferably it would be a useful statement that applies to C.

On 29/02/2024 19:05, Scott Lurndal wrote:
> David Brown <david.brown@hesbynett.no> writes:
>> On 29/02/2024 18:28, Scott Lurndal wrote:
>>> Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:
>>>> scott@slp53.sl.home (Scott Lurndal) writes:
>>>>> bart <bc@freeuk.com> writes:
>>>>>> On 28/02/2024 23:31, Keith Thompson wrote:
>>>>>>> bart <bc@freeuk.com> writes:
>>>>>>
>>>>>>>> It would be unfortunate if your example was allowed. Clearly a binary
>>>>>>>> representation of an instance of your struct would probably require 16
>>>>>>>> bytes rather than 4, of which one may be padding.
>>>>>>>
>>>>>>> Depending on the sizes and alignments of the various types, sure.
>>>>>>> So what?
>>>>>>>
>>>>>>
>>>>>>> If you have suggestions for alternate ways to define #embed, they might
>>>>>>> be interesting, but it's too late to change the existing specification.
>>>>>>>
>>>>>>
>>>>>> My early comments on this were about compiler performance. I suggested
>>>>>> there might be a way to turn 100,000 byte values in a file, directly
>>>>>> into a 100KB string or data block, without needing to first convert
>>>>>> 100,000 values into 100,000 integer expressions representated as tokens,
>>>>>> and to then parse those 100,000 expressions into AST nodes etc.
>>>>>>
>>>>>> DB suggested something like that was actually done. But you can't do
>>>>>> that if those 100,000 numbers represent from 100KB to 800KB of memory
>>>>>> depending on the data type of the strucure they're initialising.
>>>>>
>>>>> An implementation is free to simply pass a variant (or the directive
>>>>> itself) of #embed from the pre-processor to the compiler if the programmer
>>>>> isn't using -E, and the compiler could simply copy the embedded file
>>>>> into the object file directly, without processing it as a series of
>>>>> integer values. Much like the #file and #line directives passed by
>>>>> the pre-processor to the compiler.
>>>>
>>>> Sure, an implementation has to operate *as if* it implemented the 8
>>>> translation phases separately. But given a structure initialized with
>>>> #embed, it would have to generate additional code to initialize the
>>>> structure members from the bytes of the binary blob.
>>>
>>> Would it? Or could it simply assume that the binary blob
>>> is already in the same binary format that writing an instance
>>> of the structure from a C application on the same host would have created?
>>
>> That would depend on the sizes of the fields in the struct, and the size
>> of the integer constants in the #embed.
>
> I'm embedding a binary file. I want the representation in memory
> to be _exactly_ the same as in the file, regardless of how it is
> defined in the C code (array of char, array of int, array of long, struct whatever).
>

Then you would want a union of the struct type and an appropriately
sized unsigned char array, and initialise the unsigned char area with
the bytes of the file using #embed.