I was trying to find the history of pp-numbers.
I find them initially at C89 draft.

pp-numbers are "very open" and I was trying to find the reason
or justification for that.

Since the begging the objective seems be to map integer
and floating constants, but since C89 the definition allows much
more than integer and floating constants.

Can anyone see a reason for that?
Maybe it is just a matter of
"this is the way the first guy implemented it."?

Thiago Adams <thiago.adams@gmail.com> writes:
> I was trying to find the history of pp-numbers.
> I find them initially at C89 draft.
>
> pp-numbers are "very open" and I was trying to find the reason
> or justification for that.
>
> Since the begging the objective seems be to map integer
> and floating constants, but since C89 the definition allows much
> more than integer and floating constants.
>
> Can anyone see a reason for that?
> Maybe it is just a matter of
> "this is the way the first guy implemented it."?

This is addressed in the ANSI C Rationale:
The notion of preprocessing numbers has been introduced to
simplify the description of preprocessing. It provides a means
of talking about the tokenization of strings that look like
numbers, or initial substrings of numbers, prior to their
semantic interpretation. In the interests of keeping the
description simple, occasional spurious forms are scanned as
preprocessing numbers --- 0x123E+1 is a single token under the
rules. The Committee felt that it was better to tolerate such
anomalies than burden the preprocessor with a more exact, and
exacting, lexical specification. It felt that this anomaly was
no worse than the principle under which the characters a+++++b
are tokenized as a ++ ++ + b (an invalid expression), even though
the tokenization a ++ + ++ b would yield a syntactically correct
expression. In both cases, exercise of reasonable precaution
in coding style avoids surprises.
https://www.lysator.liu.se/c/rat/c1.html#3-1-8

I'll note that the proprocessor has to *evaluate* integer constants and
expressions, but it only has to *recognize* floating-point constants and
expressions, leaving evaluation to translation phase 7 (or to execution
time).

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

On Wednesday, August 30, 2023 at 5:57:46 PM UTC-3, Keith Thompson wrote:
> Thiago Adams <thiago...@gmail.com> writes:
> > I was trying to find the history of pp-numbers.
> > I find them initially at C89 draft.
> >
> > pp-numbers are "very open" and I was trying to find the reason
> > or justification for that.
> >
> > Since the begging the objective seems be to map integer
> > and floating constants, but since C89 the definition allows much
> > more than integer and floating constants.
> >
> > Can anyone see a reason for that?
> > Maybe it is just a matter of
> > "this is the way the first guy implemented it."?
> This is addressed in the ANSI C Rationale:
> The notion of preprocessing numbers has been introduced to
> simplify the description of preprocessing. It provides a means
> of talking about the tokenization of strings that look like
> numbers, or initial substrings of numbers, prior to their
> semantic interpretation. In the interests of keeping the
> description simple, occasional spurious forms are scanned as
> preprocessing numbers --- 0x123E+1 is a single token under the
> rules. The Committee felt that it was better to tolerate such
> anomalies than burden the preprocessor with a more exact, and
> exacting, lexical specification. It felt that this anomaly was
> no worse than the principle under which the characters a+++++b
> are tokenized as a ++ ++ + b (an invalid expression), even though
> the tokenization a ++ + ++ b would yield a syntactically correct
> expression. In both cases, exercise of reasonable precaution
> in coding style avoids surprises.
> https://www.lysator.liu.se/c/rat/c1.html#3-1-8
>
> I'll note that the proprocessor has to *evaluate* integer constants and
> expressions, but it only has to *recognize* floating-point constants and
> expressions, leaving evaluation to translation phase 7 (or to execution
> time).
>
Thanks!

I am thinking is just make pp-number as

(6.4.8) pp-number:
integer-constant
floating-constant

Thiago Adams <thiago.adams@gmail.com> writes:
> I am thinking is just make pp-number as
>
> (6.4.8) pp-number:
> integer-constant
> floating-constant

I agree that that would make sense, and I suspect that's how it would be
specified if the language were being defined from scratch today.

I think the idea is that parsing floating constants, and rejecting
invalid floating constants, is a bit complex. Given that the
preprocessor is a separate program from the compiler proper, and the
compiler has to reject something like 0x123E+1 anyway. It slightly
reduces the burden on the preprocessor -- something that may have been
more significant in 1989 than it would be today.

There's probably not much point in changing it now. It would mean that
0x123E+1 would be parsed as 0x123E + 1 rather than as a syntax error,
but that's not much of an advantage, since a programmer should just use
spaces in the first place.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

On Wednesday, August 30, 2023 at 7:25:01 PM UTC-3, Keith Thompson wrote:
> Thiago Adams <thiago...@gmail.com> writes:
> > I am thinking is just make pp-number as
> >
> > (6.4.8) pp-number:
> > integer-constant
> > floating-constant
> I agree that that would make sense, and I suspect that's how it would be
> specified if the language were being defined from scratch today.
>
> I think the idea is that parsing floating constants, and rejecting
> invalid floating constants, is a bit complex. Given that the
> preprocessor is a separate program from the compiler proper, and the
> compiler has to reject something like 0x123E+1 anyway. It slightly
> reduces the burden on the preprocessor -- something that may have been
> more significant in 1989 than it would be today.
>
> There's probably not much point in changing it now. It would mean that
> 0x123E+1 would be parsed as 0x123E + 1 rather than as a syntax error,
> but that's not much of an advantage, since a programmer should just use
> spaces in the first place.

My idea is a long term plan to move preprocessor phase to compiler phase.
Starting with removing phase 2( the line slicing) and removing pp-number.
I think it a simplification.
An optimisation that could be done is a different rule
inside inactive preprocessor regions, where #endif etc.. is what the preprocessor
is looking for to close.
the active regions will have to parse the numbers anyway, so having these p-numbers
only makes code more complicated instead of simplification.

one way to think in a long term C is emitting warnings in code that uses pp-number or
line slicing in a way not compatible with the future view.

On 2023-08-30, Thiago Adams <thiago.adams@gmail.com> wrote:
> I was trying to find the history of pp-numbers.
> I find them initially at C89 draft.
>
> pp-numbers are "very open" and I was trying to find the reason
> or justification for that.

[ ... ]

> Can anyone see a reason for that?
> Maybe it is just a matter of
> "this is the way the first guy implemented it."?

Initially, back in the late 1970's, the C preprocessor was
an add-on program run in front of the compiler.

It did only very light lexical analysis. For instance,
you could interpolate macros into character constants:

#define X A
'X'

would turn into

'A'

From the start, the preprocessor had its own idea of how to scan, and it
took shortcuts, not doing it exactly the same way as the underlying
language, but only well enough for its purposes.

In the language, you never have a meaningful combination of tokens like
123 abc; there is no phrase of the syntax in which that occurs.
Therefore there is no concern that input 123abc has to be unraveled into
two preprocessor tokens 123 abc. So the preprocessor can get away
with treating that as one item.

From that point of departure, preprocessing was a bit of a zoo, with
different people's preprocessors implementations things slightly
differently.

It could only be tightened so much without causing breakages
for users.

If you were doing a clean redesign of the preprocessing from scratch
today, unhampered by backward compatibility (and for some strange reason
still wanted a token-level macro preprocessor, instead of a structural
one), you would probably not have a token syntax that is different from
the language being preprocessed.

You would make sure that your preprocessor accurately handles the
precise token syntax of the language.

In fact, you would stick the preprocessor just behind the lexical
analyzer of the language implementation so that it works with the actual
tokens of that language, and not its own preprocessor tokens.

However, the looseness in the preprocessor has some benefits in the area
of language development. For instance, let's consider that in 1989's
ANSI C, the token 1ULL was already a pp-number. So when C99 gave that
suffix a meaning, it could do so without having to change the pp-number
syntax, thus requiring no change in the early translation phases in
order to support that token.

Let's look at a language that doesn't have a C-like preprocessor: Common
Lisp. Yet, ANSI Common lisp has a curious section:

2.3.1.1 Potential Numbers as Tokens

To allow implementors and future Common Lisp standards to extend the
syntax of numbers, a syntax for potential numbers is defined that is
more general than the syntax for numbers. A token is a potential
number if it satisfies all of the following requirements:

https://clhs.lisp.se/Body/02_caa.htm [note: self-signed SSL cert!]

Sound familiar? C's pp-numbers do something like this. They define a
syntax for numbers that is more general, and has been used for future
extension.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

On Thursday, August 31, 2023 at 4:42:02 AM UTC-3, Kaz Kylheku wrote:
> On 2023-08-30, Thiago Adams <thiago...@gmail.com> wrote:
> > I was trying to find the history of pp-numbers.
> > I find them initially at C89 draft.
> >
> > pp-numbers are "very open" and I was trying to find the reason
> > or justification for that.
> [ ... ]
> > Can anyone see a reason for that?
> > Maybe it is just a matter of
> > "this is the way the first guy implemented it."?
> Initially, back in the late 1970's, the C preprocessor was
> an add-on program run in front of the compiler.
>
> It did only very light lexical analysis. For instance,
> you could interpolate macros into character constants:
>
> #define X A
> 'X'
>
> would turn into
>
> 'A'
>
> From the start, the preprocessor had its own idea of how to scan, and it
> took shortcuts, not doing it exactly the same way as the underlying
> language, but only well enough for its purposes.
>
> In the language, you never have a meaningful combination of tokens like
> 123 abc; there is no phrase of the syntax in which that occurs.
> Therefore there is no concern that input 123abc has to be unraveled into
> two preprocessor tokens 123 abc. So the preprocessor can get away
> with treating that as one item.
>
> From that point of departure, preprocessing was a bit of a zoo, with
> different people's preprocessors implementations things slightly
> differently.
>
> It could only be tightened so much without causing breakages
> for users.
>
> If you were doing a clean redesign of the preprocessing from scratch
> today, unhampered by backward compatibility (and for some strange reason
> still wanted a token-level macro preprocessor, instead of a structural
> one), you would probably not have a token syntax that is different from
> the language being preprocessed.
>
> You would make sure that your preprocessor accurately handles the
> precise token syntax of the language.
>
> In fact, you would stick the preprocessor just behind the lexical
> analyzer of the language implementation so that it works with the actual
> tokens of that language, and not its own preprocessor tokens.
>
> However, the looseness in the preprocessor has some benefits in the area
> of language development. For instance, let's consider that in 1989's
> ANSI C, the token 1ULL was already a pp-number. So when C99 gave that
> suffix a meaning, it could do so without having to change the pp-number
> syntax, thus requiring no change in the early translation phases in
> order to support that token.

Now in C23, true and false were added into preprocessor, so adding things
Is not a problem in my view.

#if true
#else
#endif

Instead of big breaking changes I want to something this works as it is today, maybe
in a way that is already conceived in the mind of programmers and which
maybe implementation does not match. Like slicing identifiers tokens in the middle, that is
allowed but I am sure programmers don't think in to do that.
I am sure 99,99% of the programmers don't think in terms of pp-numbers.

Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:

[...]

> I'll note that the proprocessor has to *evaluate* integer
> constants and expressions, but it only has to *recognize*
> floating-point constants and expressions, leaving evaluation to
> translation phase 7 (or to execution time).

AFAICT there is no need for any of translation phases 1 to 6 to
recognize floating-point constants. All they need is to be able
to tell if a pp-number has the form of an integer constant, so
it can be evaluated, when in the context of some preprocessing
directives.

Tim Rentsch <tr.17687@z991.linuxsc.com> writes:
> Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:
>
> [...]
>
>> I'll note that the proprocessor has to *evaluate* integer
>> constants and expressions, but it only has to *recognize*
>> floating-point constants and expressions, leaving evaluation to
>> translation phase 7 (or to execution time).
>
> AFAICT there is no need for any of translation phases 1 to 6 to
> recognize floating-point constants. All they need is to be able
> to tell if a pp-number has the form of an integer constant, so
> it can be evaluated, when in the context of some preprocessing
> directives.

I agree.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:

> Thiago Adams <thiago.adams@gmail.com> writes:
>
>> I am thinking is just make pp-number as
>>
>> (6.4.8) pp-number:
>> integer-constant
>> floating-constant
>
> I agree that that would make sense, and I suspect that's how it
> would be specified if the language were being defined from scratch
> today.

There's an example in the Rationale document that indicates
otherwise. In C99RationaleV5.10.pdf, the example is at the
bottom of page 57, and starts "A new feature of C99:".