I have implemented the preprocessor 'on demand' in my c compiler
front end and I was kind of regretting it because the code become
much more complex compared with preprocess in everything in
advance and separately.

I was planning to change it to simplify the code but I realized that
this makes it possible the integration with the compiler.

My parser asks 'give me the next token’ for a preprocessor state machine
and it returns only tokens already preprocessed.

For instance:

void F() {
char ch =
#if 0
'a'
#else
'b'
#endif
;
} after '=' the parser pulls the next token and the preprocessor move
its internal state consuming preprocessor tokens until it finds 'b' returning it’.

It asks again and the preprocessor returns ';' changing it internal state
after consuming (but not returning) #endif.

The internal state (#include, #ifdefs, #endif) are kept in stacks.
Macro expansion is on demand as well.

This preprocessor state machine has an advantage.

It still happens before the compiler phase but it has the compiler
context and could access it.

That means that #if could ask anything for the compiler like sizeof
of the variable on scope, or if a variable/function exists etc.

Considering this.. I was wondering if the C language could change its
concepts to allow this in the future.

When we read code, we don't do the preprocessor "separated" in
your mind we do it "on demand" as well.

The invention of _Static_assert for instance would not be necessary

Instead of

_Static_assert( sizeof(x) == sizeof(int), "must be sizeof int");

we could just write:

#if sizeof(x) != sizeof(int)
#error x must be == sizeof int
#endif

On Sunday, October 3, 2021 at 10:47:58 AM UTC-3, Thiago Adams wrote:

> My parser asks 'give me the next token’ for a preprocessor state machine
> and it returns only tokens already preprocessed.

Something assumed to be true is that the compiler state does not change
while preprocessor is consuming pptokens. I guess this is true until
we find the next parser phase token.

The problem is when we need two tokens ahead because the parser
state could change after finding the first compiler phase token.

(the compiler depends on the preprocessor and vice versa)

{
​if (1) {
​​int a; /*here*/
​}
​ #if sizeof(a)
​#endif
}

If after ​​int a; /*here*/ the compiler asks two tokens
then the preprocessor will have the compiler state when it started
but it will have to consume more pptokens until it finds the
second '}'.

In this case the compiler state is outdated because the variable 'a'
is out of the scope but the preprocessor does not know because the
compiler did not consume the first token to change its state.

In this case the preprocessor will evaluated the condition as if a
where in the scope.

I am not sure how big is this problem. I think it cannot be well defined
because the way compiler will need/consume look ahead tokens cannot be
specified. Some compiler may use look ahead and others not and the evaluation
could give different results in each compiler.

On 03/10/2021 14:47, Thiago Adams wrote:
> I have implemented the preprocessor 'on demand' in my c compiler
> front end and I was kind of regretting it because the code become
> much more complex compared with preprocess in everything in
> advance and separately.
>
> I was planning to change it to simplify the code but I realized that
> this makes it possible the integration with the compiler.
>
> My parser asks 'give me the next token’ for a preprocessor state machine
> and it returns only tokens already preprocessed.
>
> For instance:
>
> void F() {
> char ch =
> #if 0
> 'a'
> #else
> 'b'
> #endif
> ;
> }
> after '=' the parser pulls the next token and the preprocessor move
> its internal state consuming preprocessor tokens until it finds 'b' returning it’.
>
> It asks again and the preprocessor returns ';' changing it internal state
> after consuming (but not returning) #endif.
>
> The internal state (#include, #ifdefs, #endif) are kept in stacks.
> Macro expansion is on demand as well.
>
> This preprocessor state machine has an advantage.
>
> It still happens before the compiler phase but it has the compiler
> context and could access it.
>
> That means that #if could ask anything for the compiler like sizeof
> of the variable on scope, or if a variable/function exists etc.
>
> Considering this.. I was wondering if the C language could change its
> concepts to allow this in the future.
>
> When we read code, we don't do the preprocessor "separated" in
> your mind we do it "on demand" as well.
>
> The invention of _Static_assert for instance would not be necessary
>
> Instead of
>
> _Static_assert( sizeof(x) == sizeof(int), "must be sizeof int");
>
> we could just write:
>
> #if sizeof(x) != sizeof(int)

This requires that the lexer knows about symbol tables, scopes and user
types, and that operations such as building those types and user types
and the name resolution required to figure out which x is intended, are
all done while parsing.

In other words, that part needs to be single pass, which puts a
constraint on the compiler.

Also, what happens when someone /only/ wants to do a preprocessing pass?
Then 'sizeof(x)' is meaningless.

Would you have to remove that ability from the language? I think more
than a few compilers depend on a third party preprocessors, and various
language processing tools may depend on it too.

So it's an unlikely retrofit for C.

For a new language though, or one which eventually transpiles to C, you
can define it as you like.

On Sunday, October 3, 2021 at 1:24:32 PM UTC-3, Bart wrote:
> On 03/10/2021 14:47, Thiago Adams wrote:
> > I have implemented the preprocessor 'on demand' in my c compiler
> > front end and I was kind of regretting it because the code become
> > much more complex compared with preprocess in everything in
> > advance and separately.
> >
> > I was planning to change it to simplify the code but I realized that
> > this makes it possible the integration with the compiler.
> >
> > My parser asks 'give me the next token’ for a preprocessor state machine
> > and it returns only tokens already preprocessed.
> >
> > For instance:
> >
> > void F() {
> > char ch =
> > #if 0
> > 'a'
> > #else
> > 'b'
> > #endif
> > ;
> > }
> > after '=' the parser pulls the next token and the preprocessor move
> > its internal state consuming preprocessor tokens until it finds 'b' returning it’.
> >
> > It asks again and the preprocessor returns ';' changing it internal state
> > after consuming (but not returning) #endif.
> >
> > The internal state (#include, #ifdefs, #endif) are kept in stacks.
> > Macro expansion is on demand as well.
> >
> > This preprocessor state machine has an advantage.
> >
> > It still happens before the compiler phase but it has the compiler
> > context and could access it.
> >
> > That means that #if could ask anything for the compiler like sizeof
> > of the variable on scope, or if a variable/function exists etc.
> >
> > Considering this.. I was wondering if the C language could change its
> > concepts to allow this in the future.
> >
> > When we read code, we don't do the preprocessor "separated" in
> > your mind we do it "on demand" as well.
> >
> > The invention of _Static_assert for instance would not be necessary
> >
> > Instead of
> >
> > _Static_assert( sizeof(x) == sizeof(int), "must be sizeof int");
> >
> > we could just write:
> >
> > #if sizeof(x) != sizeof(int)
> This requires that the lexer knows about symbol tables, scopes and user
> types, and that operations such as building those types and user types
> and the name resolution required to figure out which x is intended, are
> all done while parsing.

The lexer ask questions direct to the compiler and the compiler has
all the tables.

compiler says : "give me the next token, in case you need ask me".

> In other words, that part needs to be single pass, which puts a
> constraint on the compiler.

Yes, good point.
> Also, what happens when someone /only/ wants to do a preprocessing pass?
> Then 'sizeof(x)' is meaningless.

Would be an error.

> Would you have to remove that ability from the language? I think more
> than a few compilers depend on a third party preprocessors, and various
> language processing tools may depend on it too.
>
> So it's an unlikely retrofit for C.

I think all the preprocessor is already very mixed with the compiler
I have several samples.

-C++ added digit separators and they had to change pp-number (preprocessor
definition of number). You cannot use an external preprocessor that is
not synchronized.
-#pragma are used to control the compiler. what happen using an external preprocessor
will just remove the content?

__func__ is already funny.

On 03/10/2021 18:55, Thiago Adams wrote:
> On Sunday, October 3, 2021 at 1:24:32 PM UTC-3, Bart wrote:
>> On 03/10/2021 14:47, Thiago Adams wrote:
>>> I have implemented the preprocessor 'on demand' in my c compiler
>>> front end and I was kind of regretting it because the code become
>>> much more complex compared with preprocess in everything in
>>> advance and separately.
>>>
>>> I was planning to change it to simplify the code but I realized that
>>> this makes it possible the integration with the compiler.
>>>
>>> My parser asks 'give me the next token’ for a preprocessor state machine
>>> and it returns only tokens already preprocessed.
>>>
>>> For instance:
>>>
>>> void F() {
>>> char ch =
>>> #if 0
>>> 'a'
>>> #else
>>> 'b'
>>> #endif
>>> ;
>>> }
>>> after '=' the parser pulls the next token and the preprocessor move
>>> its internal state consuming preprocessor tokens until it finds 'b' returning it’.
>>>
>>> It asks again and the preprocessor returns ';' changing it internal state
>>> after consuming (but not returning) #endif.
>>>
>>> The internal state (#include, #ifdefs, #endif) are kept in stacks.
>>> Macro expansion is on demand as well.
>>>
>>> This preprocessor state machine has an advantage.
>>>
>>> It still happens before the compiler phase but it has the compiler
>>> context and could access it.
>>>
>>> That means that #if could ask anything for the compiler like sizeof
>>> of the variable on scope, or if a variable/function exists etc.
>>>
>>> Considering this.. I was wondering if the C language could change its
>>> concepts to allow this in the future.
>>>
>>> When we read code, we don't do the preprocessor "separated" in
>>> your mind we do it "on demand" as well.
>>>
>>> The invention of _Static_assert for instance would not be necessary
>>>
>>> Instead of
>>>
>>> _Static_assert( sizeof(x) == sizeof(int), "must be sizeof int");
>>>
>>> we could just write:
>>>
>>> #if sizeof(x) != sizeof(int)
>> This requires that the lexer knows about symbol tables, scopes and user
>> types, and that operations such as building those types and user types
>> and the name resolution required to figure out which x is intended, are
>> all done while parsing.
>
> The lexer ask questions direct to the compiler and the compiler has
> all the tables.
>
> compiler says : "give me the next token, in case you need ask me".
>
>
>> In other words, that part needs to be single pass, which puts a
>> constraint on the compiler.
>
> Yes, good point.
>
>> Also, what happens when someone /only/ wants to do a preprocessing pass?
>> Then 'sizeof(x)' is meaningless.
>
> Would be an error.
>
>> Would you have to remove that ability from the language? I think more
>> than a few compilers depend on a third party preprocessors, and various
>> language processing tools may depend on it too.
>>
>> So it's an unlikely retrofit for C.
>
> I think all the preprocessor is already very mixed with the compiler
> I have several samples.
>
> -C++ added digit separators and they had to change pp-number (preprocessor
> definition of number). You cannot use an external preprocessor that is
> not synchronized.

> -#pragma are used to control the compiler. what happen using an external preprocessor
> will just remove the content?

If an implementation doesn't understand what follows #pragma, then it
will just ignore it.

When it does understand it, it will just define some input, or set some
state, for the rest of the compiler to make use of. I don't think it
needs input from the compiler, or from later stages of it, unless you
have some examples?

When doing preprocessing only, then #pragma lines are passed unchanged.
(My compiler consumes them, so it's a bug, sort of. But there are worse
ones...)

> __func__ is already funny.

__func__ doesn't really belong in the processor. And actually it's a
predefined identifier, not a predefined macro.

If you try and use __func__ with -E, then you just get __func__. Which
means that when you try and compile that preprocessed output, /then/ it
will be expanded to the function name, if it's inside a function.

On Sunday, October 3, 2021 at 3:41:24 PM UTC-3, Bart wrote:
> On 03/10/2021 18:55, Thiago Adams wrote:
> > On Sunday, October 3, 2021 at 1:24:32 PM UTC-3, Bart wrote:
> >> On 03/10/2021 14:47, Thiago Adams wrote:
> >>> I have implemented the preprocessor 'on demand' in my c compiler
> >>> front end and I was kind of regretting it because the code become
> >>> much more complex compared with preprocess in everything in
> >>> advance and separately.
> >>>
> >>> I was planning to change it to simplify the code but I realized that
> >>> this makes it possible the integration with the compiler.
> >>>
> >>> My parser asks 'give me the next token’ for a preprocessor state machine
> >>> and it returns only tokens already preprocessed.
> >>>
> >>> For instance:
> >>>
> >>> void F() {
> >>> char ch =
> >>> #if 0
> >>> 'a'
> >>> #else
> >>> 'b'
> >>> #endif
> >>> ;
> >>> }
> >>> after '=' the parser pulls the next token and the preprocessor move
> >>> its internal state consuming preprocessor tokens until it finds 'b' returning it’.
> >>>
> >>> It asks again and the preprocessor returns ';' changing it internal state
> >>> after consuming (but not returning) #endif.
> >>>
> >>> The internal state (#include, #ifdefs, #endif) are kept in stacks.
> >>> Macro expansion is on demand as well.
> >>>
> >>> This preprocessor state machine has an advantage.
> >>>
> >>> It still happens before the compiler phase but it has the compiler
> >>> context and could access it.
> >>>
> >>> That means that #if could ask anything for the compiler like sizeof
> >>> of the variable on scope, or if a variable/function exists etc.
> >>>
> >>> Considering this.. I was wondering if the C language could change its
> >>> concepts to allow this in the future.
> >>>
> >>> When we read code, we don't do the preprocessor "separated" in
> >>> your mind we do it "on demand" as well.
> >>>
> >>> The invention of _Static_assert for instance would not be necessary
> >>>
> >>> Instead of
> >>>
> >>> _Static_assert( sizeof(x) == sizeof(int), "must be sizeof int");
> >>>
> >>> we could just write:
> >>>
> >>> #if sizeof(x) != sizeof(int)
> >> This requires that the lexer knows about symbol tables, scopes and user
> >> types, and that operations such as building those types and user types
> >> and the name resolution required to figure out which x is intended, are
> >> all done while parsing.
> >
> > The lexer ask questions direct to the compiler and the compiler has
> > all the tables.
> >
> > compiler says : "give me the next token, in case you need ask me".
> >
> >
> >> In other words, that part needs to be single pass, which puts a
> >> constraint on the compiler.
> >
> > Yes, good point.
> >
> >> Also, what happens when someone /only/ wants to do a preprocessing pass?
> >> Then 'sizeof(x)' is meaningless.
> >
> > Would be an error.
> >
> >> Would you have to remove that ability from the language? I think more
> >> than a few compilers depend on a third party preprocessors, and various
> >> language processing tools may depend on it too.
> >>
> >> So it's an unlikely retrofit for C.
> >
> > I think all the preprocessor is already very mixed with the compiler
> > I have several samples.
> >
> > -C++ added digit separators and they had to change pp-number (preprocessor
> > definition of number). You cannot use an external preprocessor that is
> > not synchronized.
>
> > -#pragma are used to control the compiler. what happen using an external preprocessor
> > will just remove the content?
> If an implementation doesn't understand what follows #pragma, then it
> will just ignore it.

Yes but if you think a traditional preprocessor that runs before and generate a file that
latter will be the input for the compiler then the #pragma is removed (empty line) when the file
is printed and the compiler will not have access to it.

Having the preprocessor integrated can keep #pragma in tokens (with internal representation)
and latter discard or use it.

> When it does understand it, it will just define some input, or set some
> state, for the rest of the compiler to make use of. I don't think it
> needs input from the compiler, or from later stages of it, unless you
> have some examples?

When the preprocessor is from the same compiler I think some tokens generated
by pragmas are used to control the compiler.. like disabling warnings.
(I am not sure If I understood the question)

> When doing preprocessing only, then #pragma lines are passed unchanged.
> (My compiler consumes them, so it's a bug, sort of. But there are worse
> ones...)
>
> > __func__ is already funny.
>
> __func__ doesn't really belong in the processor. And actually it's a
> predefined identifier, not a predefined macro.

yes... this is why it is funny. it is like a macro expansion by the compiler.

> If you try and use __func__ with -E, then you just get __func__. Which
> means that when you try and compile that preprocessed output, /then/ it
> will be expanded to the function name, if it's inside a function.

There is more ..

now C23 has attributes and we can ask if the compiler support it:

#if __has_c_attribute(nodiscard)
yes
#endif

-E returns yes.

not difficult to implement, but shows preprocessor and compiler are
interdependent.
If the information required by the preprocessor from the compiler is
static then everything is easy. sizeof(int) for instance could be
implemented easily if necessary, but sizeof(variable) not.

On 03/10/2021 20:16, Thiago Adams wrote:
> On Sunday, October 3, 2021 at 3:41:24 PM UTC-3, Bart wrote:

>>> -#pragma are used to control the compiler. what happen using an external preprocessor
>>> will just remove the content?
>> If an implementation doesn't understand what follows #pragma, then it
>> will just ignore it.
>
> Yes but if you think a traditional preprocessor that runs before and generate a file that
> latter will be the input for the compiler then the #pragma is removed (empty line) when the file
> is printed and the compiler will not have access to it.

No, I said the compiler keeps it in. (Well, I tried gcc, tcc, clang and
lccwin which did, and bcc which didn't.)

Probably precisely for that reason.

>

> yes... this is why it is funny. it is like a macro expansion by the compiler.

In my language this stuff is done by the parser anyway.

> If the information required by the preprocessor from the compiler is
> static then everything is easy. sizeof(int) for instance could be
> implemented easily if necessary, but sizeof(variable) not.

sizeof(int) maybe, but probably not sizeof(int32_t). First this will
require headers to be present, that's OK, but int32_t is likely to be a
typedef.

However this rings a bell ... yes, I've put sizeof() into my own
preprocessor, so that this is possible:

#if sizeof(int) == 4

but with the aforementioned problems using int32_t.

Bart <bc@freeuk.com> writes:
[...]
> However this rings a bell ... yes, I've put sizeof() into my own
> preprocessor, so that this is possible:
>
> #if sizeof(int) == 4
>
> but with the aforementioned problems using int32_t.

From a comp.std.c post in 1998:

> You are right. It was nice back in the days when things like
>
> #if (sizeof(int) == 8)
>
> actually worked (on some compilers).

Must have been before my time.

Dennis

Yes, the poster was Dennis Ritchie.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips
void Void(void) { Void(); } /* The recursive call of the void */

On 10/3/2021 6:47 AM, Thiago Adams wrote:
>
> It still happens before the compiler phase but it has the compiler
> context and could access it.
>

You are trying to invent C++'s `if constexpr` or some subset thereof.
The topic is sufficiently well researched. I just don't see any reason
to pin all this on "preprocessor".

--
Best regards,
Andrey Tarasevich

On Wednesday, October 6, 2021 at 12:41:21 PM UTC-3, Andrey Tarasevich wrote:
> On 10/3/2021 6:47 AM, Thiago Adams wrote:
> >
> > It still happens before the compiler phase but it has the compiler
> > context and could access it.
> >
> You are trying to invent C++'s `if constexpr` or some subset thereof.
> The topic is sufficiently well researched. I just don't see any reason
> to pin all this on "preprocessor".

To be used together with #if .

C++ 'if constexpr' can be used only where if can.

D has static if that can be used in more places.

https://dlang.org/spec/version.html#staticif