Just adding some better error reporting to my XML parser.

I started out by keeping track of the line number as I went. But of course that's very fiddly. There's a much better way of doing it, which is to rewind the file and count newlines.

It's slow of course. But you will call the function when in error procesing mode. So that shouldn't matter.

Here's the function I wrote.

int flineno(FILE *fp)
{ long pos;
int err;
int ch;
int answer = 1;

pos = ftell(fp);
if (pos < 0)
return 0;
err = fseek(fp, 0, SEEK_SET);
if (err)
return 0;
while (ftell(fp) < pos)
{
ch = fgetc(fp);
if (ch == '\n')
answer++;
if (ch == EOF)
return 0;
}
if (ch == '\n')
answer--;

return answer;
}

Note the slightly tricky detail. If the last character read was a newline, that probably meant that you read a line of text, parsed it, and found an error in that line. So the function needs to report the previous rather than the current line.

czwartek, 10 sierpnia 2023 o 13:10:52 UTC+2 Malcolm McLean napisał(a):
> Just adding some better error reporting to my XML parser.
>
> I started out by keeping track of the line number as I went. But of course that's very fiddly. There's a much better way of doing it, which is to rewind the file and count newlines.
>
> It's slow of course. But you will call the function when in error procesing mode. So that shouldn't matter.
>
> Here's the function I wrote.
>
> int flineno(FILE *fp)
> {
> long pos;
> int err;
> int ch;
> int answer = 1;
>
> pos = ftell(fp);
> if (pos < 0)
> return 0;
> err = fseek(fp, 0, SEEK_SET);
> if (err)
> return 0;
> while (ftell(fp) < pos)
> {
> ch = fgetc(fp);
> if (ch == '\n')
> answer++;
> if (ch == EOF)
> return 0;
> }
> if (ch == '\n')
> answer--;
>
> return answer;
> }
>
> Note the slightly tricky detail. If the last character read was a newline, that probably meant that you read a line of text, parsed it, and found an error in that line. So the function needs to report the previous rather than the current line.

i recently switched my monitor to more width and more resolution and then i disliked
to write such vertical codes - and yours is specially vertical here
(i used to keey things to 50-60 char width i guess now im rather go twice as that

it seem to depend on monitor someone use but recently i find the width style much better
as
1) you dont need so much optical readability of code you already written
2) and you need a code comacity becouse files you work are much more compact

so what you write strikes me as ty stylistycal point..its terrible waste
something like that would be better to make 2-4 vertical lines of it
c fault is afir one candt put the "int" ceclaration in some places and it is not visibel
below loop (its c fault as loop scope usually works on context of a function it is in not has its own only
besides bad names imo fp should be file and answer should be lines
besides it looks something weird logically imo - isnt it better just to coout "\n" and then say add 1

czwartek, 10 sierpnia 2023 o 13:58:15 UTC+2 fir napisał(a):
> czwartek, 10 sierpnia 2023 o 13:10:52 UTC+2 Malcolm McLean napisał(a):
> > Just adding some better error reporting to my XML parser.
> >
> > I started out by keeping track of the line number as I went. But of course that's very fiddly. There's a much better way of doing it, which is to rewind the file and count newlines.
> >
> > It's slow of course. But you will call the function when in error procesing mode. So that shouldn't matter.
> >
> > Here's the function I wrote.
> >
> > int flineno(FILE *fp)
> > {
> > long pos;
> > int err;
> > int ch;
> > int answer = 1;
> >
> > pos = ftell(fp);
> > if (pos < 0)
> > return 0;
> > err = fseek(fp, 0, SEEK_SET);
> > if (err)
> > return 0;
> > while (ftell(fp) < pos)
> > {
> > ch = fgetc(fp);
> > if (ch == '\n')
> > answer++;
> > if (ch == EOF)
> > return 0;
> > }
> > if (ch == '\n')
> > answer--;
> >
> > return answer;
> > }
> >
> > Note the slightly tricky detail. If the last character read was a newline, that probably meant that you read a line of text, parsed it, and found an error in that line. So the function needs to report the previous rather than the current line.
> i recently switched my monitor to more width and more resolution and then i disliked
> to write such vertical codes - and yours is specially vertical here
> (i used to keey things to 50-60 char width i guess now im rather go twice as that
>
> it seem to depend on monitor someone use but recently i find the width style much better
> as
> 1) you dont need so much optical readability of code you already written
> 2) and you need a code comacity becouse files you work are much more compact
>
> so what you write strikes me as ty stylistycal point..its terrible waste
> something like that would be better to make 2-4 vertical lines of it
> c fault is afir one candt put the "int" ceclaration in some places and it is not visibel
> below loop (its c fault as loop scope usually works on context of a function it is in not has its own only
> besides bad names imo fp should be file and answer should be lines
> besides it looks something weird logically imo - isnt it better just to coout "\n" and then say add 1
i mean if the logical core of this function is "for(int i=0; i<txt_len; i++) if(txt[i]=='\n') lines++"
then all that other things is unnecessary uglines

On Thursday, 10 August 2023 at 13:03:51 UTC+1, fir wrote:
> czwartek, 10 sierpnia 2023 o 13:58:15 UTC+2 fir napisał(a):
> > czwartek, 10 sierpnia 2023 o 13:10:52 UTC+2 Malcolm McLean napisał(a):
> > > Just adding some better error reporting to my XML parser.
> > >
> > > I started out by keeping track of the line number as I went. But of course that's very fiddly. There's a much better way of doing it, which is to rewind the file and count newlines.
> > >
> > > It's slow of course. But you will call the function when in error procesing mode. So that shouldn't matter.
> > >
> > > Here's the function I wrote.
> > >
> > > int flineno(FILE *fp)
> > > {
> > > long pos;
> > > int err;
> > > int ch;
> > > int answer = 1;
> > >
> > > pos = ftell(fp);
> > > if (pos < 0)
> > > return 0;
> > > err = fseek(fp, 0, SEEK_SET);
> > > if (err)
> > > return 0;
> > > while (ftell(fp) < pos)
> > > {
> > > ch = fgetc(fp);
> > > if (ch == '\n')
> > > answer++;
> > > if (ch == EOF)
> > > return 0;
> > > }
> > > if (ch == '\n')
> > > answer--;
> > >
> > > return answer;
> > > }
> > >
> > > Note the slightly tricky detail. If the last character read was a newline, that probably meant that you read a line of text, parsed it, and found an error in that line. So the function needs to report the previous rather than the current line.
> > i recently switched my monitor to more width and more resolution and then i disliked
> > to write such vertical codes - and yours is specially vertical here
> > (i used to keey things to 50-60 char width i guess now im rather go twice as that
> >
> > it seem to depend on monitor someone use but recently i find the width style much better
> > as
> > 1) you dont need so much optical readability of code you already written
> > 2) and you need a code comacity becouse files you work are much more compact
> >
> > so what you write strikes me as ty stylistycal point..its terrible waste
> > something like that would be better to make 2-4 vertical lines of it
> > c fault is afir one candt put the "int" ceclaration in some places and it is not visibel
> > below loop (its c fault as loop scope usually works on context of a function it is in not has its own only
> > besides bad names imo fp should be file and answer should be lines
> > besides it looks something weird logically imo - isnt it better just to coout "\n" and then say add 1
> i mean if the logical core of this function is "for(int i=0; i<txt_len; i++) if(txt[i]=='\n') lines++"
> then all that other things is unnecessary uglines
>
ftell() usually but not always returns the position of the file pointer in terms of calls to fgetc().
And whilst it is unlikely, you could get a IO error with the device. Then the stream might be
unseekable.
I agree that these are irritating little details. Fundamentally the logic is as you say.

On 10/08/2023 12:10, Malcolm McLean wrote:
> Just adding some better error reporting to my XML parser.
>
> I started out by keeping track of the line number as I went. But of course that's very fiddly. There's a much better way of doing it, which is to rewind the file and count newlines.
>
> It's slow of course. But you will call the function when in error procesing mode. So that shouldn't matter.
>
> Here's the function I wrote.
>
> int flineno(FILE *fp)
> {
> long pos;
> int err;
> int ch;
> int answer = 1;
>
> pos = ftell(fp);
> if (pos < 0)
> return 0;
> err = fseek(fp, 0, SEEK_SET);
> if (err)
> return 0;
> while (ftell(fp) < pos)
> {
> ch = fgetc(fp);
> if (ch == '\n')
> answer++;
> if (ch == EOF)
> return 0;
> }
> if (ch == '\n')
> answer--;
>
> return answer;
> }
>
> Note the slightly tricky detail. If the last character read was a newline, that probably meant that you read a line of text, parsed it, and found an error in that line. So the function needs to report the previous rather than the current line.

I use something similar (with parsing source code), but I load text
entirely into memory, and traverse it as a string, identifying and
returning tokens.

Each token has associated with it, its offset within the source string.

When an error occurs, it is linked to a specific token (maybe the last
one read, maybe the first of a particular syntactic construct; this is
not an exact science).

Searching backwards from the token's offset, determines the line-number,
and offset with the line.

On Thursday, 10 August 2023 at 13:21:12 UTC+1, Bart wrote:
> On 10/08/2023 12:10, Malcolm McLean wrote:
> > Just adding some better error reporting to my XML parser.
> >
> > I started out by keeping track of the line number as I went. But of course that's very fiddly. There's a much better way of doing it, which is to rewind the file and count newlines.
> >
> > It's slow of course. But you will call the function when in error procesing mode. So that shouldn't matter.
> >
> > Here's the function I wrote.
> >
> > int flineno(FILE *fp)
> > {
> > long pos;
> > int err;
> > int ch;
> > int answer = 1;
> >
> > pos = ftell(fp);
> > if (pos < 0)
> > return 0;
> > err = fseek(fp, 0, SEEK_SET);
> > if (err)
> > return 0;
> > while (ftell(fp) < pos)
> > {
> > ch = fgetc(fp);
> > if (ch == '\n')
> > answer++;
> > if (ch == EOF)
> > return 0;
> > }
> > if (ch == '\n')
> > answer--;
> >
> > return answer;
> > }
> >
> > Note the slightly tricky detail. If the last character read was a newline, that probably meant that you read a line of text, parsed it, and found an error in that line. So the function needs to report the previous rather than the current line.
> I use something similar (with parsing source code), but I load text
> entirely into memory, and traverse it as a string, identifying and
> returning tokens.
>
> Each token has associated with it, its offset within the source string.
>
> When an error occurs, it is linked to a specific token (maybe the last
> one read, maybe the first of a particular syntactic construct; this is
> not an exact science).
>
> Searching backwards from the token's offset, determines the line-number,
> and offset with the line.
>
There's a strong case for doing that. If a file is read in as a string, then any IO
erros can be dealt with in one place. Also, it's easy to integrate the parser into
main logic. Just use the Baby X resouce compiler to embed the file as a
string and you're ready to go.

However it's traditional to parse input from a stream, probably dating back to
the days when memory was expensive. That's the approach I use in the xml
parser.

czwartek, 10 sierpnia 2023 o 14:10:31 UTC+2 Malcolm McLean napisał(a):
> On Thursday, 10 August 2023 at 13:03:51 UTC+1, fir wrote:
> > czwartek, 10 sierpnia 2023 o 13:58:15 UTC+2 fir napisał(a):
> > > czwartek, 10 sierpnia 2023 o 13:10:52 UTC+2 Malcolm McLean napisał(a):
> > > > Just adding some better error reporting to my XML parser.
> > > >
> > > > I started out by keeping track of the line number as I went. But of course that's very fiddly. There's a much better way of doing it, which is to rewind the file and count newlines.
> > > >
> > > > It's slow of course. But you will call the function when in error procesing mode. So that shouldn't matter.
> > > >
> > > > Here's the function I wrote.
> > > >
> > > > int flineno(FILE *fp)
> > > > {
> > > > long pos;
> > > > int err;
> > > > int ch;
> > > > int answer = 1;
> > > >
> > > > pos = ftell(fp);
> > > > if (pos < 0)
> > > > return 0;
> > > > err = fseek(fp, 0, SEEK_SET);
> > > > if (err)
> > > > return 0;
> > > > while (ftell(fp) < pos)
> > > > {
> > > > ch = fgetc(fp);
> > > > if (ch == '\n')
> > > > answer++;
> > > > if (ch == EOF)
> > > > return 0;
> > > > }
> > > > if (ch == '\n')
> > > > answer--;
> > > >
> > > > return answer;
> > > > }
> > > >
> > > > Note the slightly tricky detail. If the last character read was a newline, that probably meant that you read a line of text, parsed it, and found an error in that line. So the function needs to report the previous rather than the current line.
> > > i recently switched my monitor to more width and more resolution and then i disliked
> > > to write such vertical codes - and yours is specially vertical here
> > > (i used to keey things to 50-60 char width i guess now im rather go twice as that
> > >
> > > it seem to depend on monitor someone use but recently i find the width style much better
> > > as
> > > 1) you dont need so much optical readability of code you already written
> > > 2) and you need a code comacity becouse files you work are much more compact
> > >
> > > so what you write strikes me as ty stylistycal point..its terrible waste
> > > something like that would be better to make 2-4 vertical lines of it
> > > c fault is afir one candt put the "int" ceclaration in some places and it is not visibel
> > > below loop (its c fault as loop scope usually works on context of a function it is in not has its own only
> > > besides bad names imo fp should be file and answer should be lines
> > > besides it looks something weird logically imo - isnt it better just to coout "\n" and then say add 1
> > i mean if the logical core of this function is "for(int i=0; i<txt_len; i++) if(txt[i]=='\n') lines++"
> > then all that other things is unnecessary uglines
> >
> ftell() usually but not always returns the position of the file pointer in terms of calls to fgetc().
> And whilst it is unlikely, you could get a IO error with the device. Then the stream might be
> unseekable.
> I agree that these are irritating little details. Fundamentally the logic is as you say.

if you do thet scanig whole file by this fgetc based on this hidden position in file
its imo rather bad design (its potentially square compelxity, though this depends on use case
as as you sey whan using it in in sole error comunicate one cand did it.. also those fgetc all
are probably cached in ram so one could meybe use that fgetc and fseek as an interface for ram-jumping and maybe sometimes its even good decision not to duplicate its ram mirror..
but i would rether need to know that... and hella know what it happens, windows is
probably able to delete cached area if is low on ram or something though i dont know
if it doeas it on opened files )..
it probably depends on how someone "loves" the api it uses.. i typically not love
this stdio.h api so much (though i dint mean it cant be loved if better known ;c
i got no idea)

overally with time i think i learned probably not be so paranoid and say it is bad (as it uses fgatc, fseeks and hidden pointer, yet potential killing square compelxity ) becouse it depends on use case
i would say assuming everything is okay everything is okay here, but it depends on how much someone want to chesk or study given topic and assumptions

im not sure if studting details of this stdio api is most interesting thing for me
(thats whan i answered mainly on code style which is more interesting to me by now)

(overally i wouldnt write thing like that as i use my own "sickle" (load into ram and then split)
way but you may write such way if you want, though i would suspect most people would say
it is bad and you will probably better keep track on current line number on the run or something like that)

czwartek, 10 sierpnia 2023 o 14:40:42 UTC+2 Malcolm McLean napisał(a):
> On Thursday, 10 August 2023 at 13:21:12 UTC+1, Bart wrote:
> > On 10/08/2023 12:10, Malcolm McLean wrote:
> > > Just adding some better error reporting to my XML parser.
> string and you're ready to go.
>
> However it's traditional to parse input from a stream, probably dating back to
> the days when memory was expensive. That's the approach I use in the xml
> parser.

those were 60-ties or 70-ties where you got 16KB ram now warious working structures i
guess are bigger... there are maybe two problems with that 1) stylistical dependency
of this stdio api (in windows for exampel you could use couple of altartantive ways to read
a file imo ) especially it is not much great 2) potential swuare complexity (i know ypu
sayd use in error routine which makes its not square but as a general "danger", becouse it some function is present there is temptation to use it in some other cases)

i seen recently this second problem (in thread on container for reducing space square complexity) - square komplexity is deadly killing 0 it kils modern ps as on low numbers of 1000
elements (as 1000x1000xC gives miliseconds when C is nanoseconds) its so damn
killing its just a disaster and its absolutely no way to go

czwartek, 10 sierpnia 2023 o 13:10:52 UTC+2 Malcolm McLean napisał(a):

> while (ftell(fp) < pos)
especially this - i wouldnt write that...hela knows what it doeas it probably does not much
then return and if but even if so it doesnt to do like that - use while and return on EOF ..

czwartek, 10 sierpnia 2023 o 16:05:43 UTC+2 fir napisał(a):
> czwartek, 10 sierpnia 2023 o 13:10:52 UTC+2 Malcolm McLean napisał(a):
>
> > while (ftell(fp) < pos)
> especially this - i wouldnt write that...hela knows what it doeas it probably does not much
> then return and if but even if so it doesnt to do like that - use while and return on EOF ..

basic advice do time measurment - write a function that returns nanoseconds from its previous call..name it delta_ns() or tic() or something like that

int main() {tic(); foo(); float f = tic(); printf("\n time of foo %1.f nanoseconds", t); return 'ok';}

this code will give times and you may see if such while(ftell(file)) has an impact over a while(1)
i probably could eventually test if here but im not sure if its worth

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>Just adding some better error reporting to my XML parser.
>
>I started out by keeping track of the line number as I went. But of course that's very fiddly. There's a much better way of doing it, which is to rewind the file and count newlines.
>

How so?

line = fgets(buffer, sizeof(buffer), stdin);
if (line != NULL) {
line_number++;
/* parse line. If the text exceeds sizeof(buffer) before
a new line is encountered, fread the remaining into an additional
buffer and concatenate buffers */
} else {
/* all done, go home */
}

>It's slow of course.

It's completely unnecessary and a waste of processing cycles.

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:

> Just adding some better error reporting to my XML parser.
>
> I started out by keeping track of the line number as I went. But of
> course that's very fiddly.

Why is it very fiddly?

> There's a much better way of doing it,
> which is to rewind the file and count newlines.
>
> It's slow of course. But you will call the function when in error
> procesing mode. So that shouldn't matter.
>
> Here's the function I wrote.
>
> int flineno(FILE *fp)
> {
> long pos;
> int err;
> int ch;
> int answer = 1;
>
> pos = ftell(fp);
> if (pos < 0)
> return 0;
> err = fseek(fp, 0, SEEK_SET);
> if (err)
> return 0;
> while (ftell(fp) < pos)
> {
> ch = fgetc(fp);
> if (ch == '\n')
> answer++;
> if (ch == EOF)
> return 0;
> }
> if (ch == '\n')

ch can be indeterminate at this point.

The fix gives you the option of a bit of a hack: if you initialise ch to
'\n' this test will report an error on line zero (for an empty file)
which you might want.

> answer--;
>
> return answer;
> }

But overall, I would not do this. It won't work when the input is not
seekable and that's just too useful a situation on Linux systems. I
don't think that matters so much to you, but I avoid tools that can't be
part of a pipeline.

The other consideration, for me, is that every lexer I've written ends
up needing state (the file name, the current token and so on) so
counting the lines is not fiddly at all. It's just another small bit of
state to maintain in the reader.

> Note the slightly tricky detail. If the last character read was a
> newline, that probably meant that you read a line of text, parsed it,
> and found an error in that line. So the function needs to report the
> previous rather than the current line.

If I were to do this, I'd consider also counting the characters since
the last newline to give a line offset as well. Mind you UTF-8 inputs
with lots of combining characters makes this hard to do correctly. In
practice, though, I don't think there are common.

--
Ben.

czwartek, 10 sierpnia 2023 o 16:13:59 UTC+2 fir napisał(a):
> czwartek, 10 sierpnia 2023 o 16:05:43 UTC+2 fir napisał(a):
> > czwartek, 10 sierpnia 2023 o 13:10:52 UTC+2 Malcolm McLean napisał(a):
> >
> > > while (ftell(fp) < pos)
> > especially this - i wouldnt write that...hela knows what it doeas it probably does not much
> > then return and if but even if so it doesnt to do like that - use while and return on EOF ..
> basic advice do time measurment - write a function that returns nanoseconds from its previous call..name it delta_ns() or tic() or something like that
>
> int main() {tic(); foo(); float f = tic(); printf("\n time of foo %1.f nanoseconds", t); return 'ok';}
>
> this code will give times and you may see if such while(ftell(file)) has an impact over a while(1)
> i probably could eventually test if here but im not sure if its worth

lol id showed to be worse than i thought (but at kleast worth of test)

tested
Les Trois Mousquetaires / The Three Musketeers
Bilingual Edition
Translated by William Robson

in my txt file 4 107 378 bytes

whisch your code and my says has 39 052 lines

your code with ftell in a loop takes 21 670 ms (when i set pos to end)
your code with while (pos--) takes 319 ms

my sickle code on splitter takes stable 13 though it does more work

int flineno(FILE *fp)
{ long pos;int err;int ch;int answer = 1;
fseek(fp, 0, SEEK_END); pos = ftell(fp); if (pos < 0) return 0;
err = fseek(fp, 0, SEEK_SET);if (err)return 0;
while (ftell(fp) < pos)
{ ch = fgetc(fp);
if (ch == '\n')answer++;
if (ch == EOF)return 0;
}

if (ch == '\n')answer--;return answer;
}

int flineno2(FILE *fp)
{ long pos;int err;int ch;int answer = 1;
fseek(fp, 0, SEEK_END); pos = ftell(fp); if (pos < 0) return 0;
err = fseek(fp, 0, SEEK_SET);if (err)return 0;

while (pos--)
{ ch = fgetc(fp);
if (ch == '\n')answer++;
if (ch == EOF) return answer;
}

if (ch == '\n')answer--;return answer;
}

int main()
{ // TestSpaceDirectory();

if(0)
{ TakeDeltaTimeNS(1);
chunk d = LoadChunk("les trois.txt");
chunks lines = SplitAfterCharacter(d, 0x0a);
int n = ChunksLength(lines);
float t = TakeDeltaTimeNS(1);

printf("\n newlines: %d %1.f ms", n, t/1000/1000);

}
else
{

FILE * fp = fopen("les trois.txt", "rt");
if(!fp) printf("error");

TakeDeltaTimeNS(1);
int n = flineno2(fp);
float t = TakeDeltaTimeNS(1);

printf("\n newlines: %d %1.f ms", n, t/1000/1000);
}

// TestDynamicArray();

return 'OK';
}

czwartek, 10 sierpnia 2023 o 17:11:46 UTC+2 fir napisał(a):
> czwartek, 10 sierpnia 2023 o 16:13:59 UTC+2 fir napisał(a):
> > czwartek, 10 sierpnia 2023 o 16:05:43 UTC+2 fir napisał(a):
> > > czwartek, 10 sierpnia 2023 o 13:10:52 UTC+2 Malcolm McLean napisał(a):
> > >
> > > > while (ftell(fp) < pos)
> > > especially this - i wouldnt write that...hela knows what it doeas it probably does not much
> > > then return and if but even if so it doesnt to do like that - use while and return on EOF ..
> > basic advice do time measurment - write a function that returns nanoseconds from its previous call..name it delta_ns() or tic() or something like that
> >
> > int main() {tic(); foo(); float f = tic(); printf("\n time of foo %1.f nanoseconds", t); return 'ok';}
> >
> > this code will give times and you may see if such while(ftell(file)) has an impact over a while(1)
> > i probably could eventually test if here but im not sure if its worth
> lol id showed to be worse than i thought (but at kleast worth of test)
>
> tested
> Les Trois Mousquetaires / The Three Musketeers
> Bilingual Edition
> Translated by William Robson
>
> in my txt file 4 107 378 bytes
>
> whisch your code and my says has 39 052 lines
>
> your code with ftell in a loop takes 21 670 ms (when i set pos to end)
> your code with while (pos--) takes 319 ms
>
> my sickle code on splitter takes stable 13 though it does more work
> int flineno(FILE *fp)
> {
> long pos;int err;int ch;int answer = 1;
> fseek(fp, 0, SEEK_END); pos = ftell(fp); if (pos < 0) return 0;
> err = fseek(fp, 0, SEEK_SET);if (err)return 0;
> while (ftell(fp) < pos)
> {
> ch = fgetc(fp);
> if (ch == '\n')answer++;
> if (ch == EOF)return 0;
> }
>
> if (ch == '\n')answer--;return answer;
> }
> int flineno2(FILE *fp)
> {
> long pos;int err;int ch;int answer = 1;
> fseek(fp, 0, SEEK_END); pos = ftell(fp); if (pos < 0) return 0;
> err = fseek(fp, 0, SEEK_SET);if (err)return 0;
> while (pos--)
> {
> ch = fgetc(fp);
> if (ch == '\n')answer++;
> if (ch == EOF) return answer;
> }
>
> if (ch == '\n')answer--;return answer;
> }
>
> int main()
> {
> // TestSpaceDirectory();
>
> if(0)
> {
> TakeDeltaTimeNS(1);
> chunk d = LoadChunk("les trois.txt");
> chunks lines = SplitAfterCharacter(d, 0x0a);
> int n = ChunksLength(lines);
> float t = TakeDeltaTimeNS(1);
>
> printf("\n newlines: %d %1.f ms", n, t/1000/1000);
>
> }
> else
> {
>
> FILE * fp = fopen("les trois.txt", "rt");
> if(!fp) printf("error");
>
> TakeDeltaTimeNS(1);
> int n = flineno2(fp);
> float t = TakeDeltaTimeNS(1);
>
> printf("\n newlines: %d %1.f ms", n, t/1000/1000);
> }
>
> // TestDynamicArray();
>
> return 'OK';
> }

if you want thsi test file https://fastupload.io/uMmGB2N7bsZx7mf/file
with test source and my dll - which i use coz it has this timer and sickle
routines inside (sickle (thsi is my routines for chunk based text processing)
may also be included as .c file but i use it sometimes as dll too)

On 2023-08-10, Malcolm McLean <malcolm.arthur.mclean@gmail.com> wrote:
> Just adding some better error reporting to my XML parser.
>
> I started out by keeping track of the line number as I went. But of
> course that's very fiddly.

It's only slightly fiddly if you have to recognize tokens that can
contain embedded newlines.

In a language in which a newline only occurs in a comment, or
whitespace, you can have a couple of dedicated tokenizing rules
which increment the line number.

You attach the current line number to all token objects and then
the parser can just refer to them.

If you allow tokens with newlines like

"multi
line
literal"

by a single token match that gobbles newlines, you then need
some slight fiddling, like walking the lexeme of that object after
it is extracted and bumping the line number for each newline
that occurs.

You can also have your lexer go through a stream abstraction which
tracks the line number (and possibly column position: popular nowadays).

> There's a much better way of doing it,
> which is to rewind the file and count newlines.

Doesn't work for non-seekable streams, like pipes and sockets.

I suspect that next to no language processors in the wild
get their error-reporting line numbers this way.

Also, it wouldn't be accurate because the current line number in the
underlying input file is not always the best line number for the error
diagnostic. Sometimes a diagnostic benefits from two line numbers.

Here is a trivial example: Lisp parser reporting unbalanced
parens, while reading multi-line input (in this case from a string):

1> (read "(foo\n(bar\n(baz" *stderr*)
string:3: syntax error
string:3: unterminated expression
string:1: while parsing expression starting on this line
** syntax error: read: string: errors encountered

We cannot use your approach to generate the aditional helpful diagnostic
about the starting line, unless at the start of every top-level
expression, we invoke the fseek-based logic to record the line number,
even when there is no error.

You're assuming it's okay for it to be "expensive" to get the
line number, because only the parsing ever needs it and only
when there is a syntax error.

Line numbers can also be usefully retained into later processing stages,
long after parsing is done.

There are errors that can be caught only after a second pass through the
parse tree or abstract syntax tree. E.g. simple example: expanding
macros. A structural macro expander works with the abstract syntax, by
walking the tree. The original stream (if any) from which the code came
is long gone.

1> (defmacro needs-arg (x))
needs-arg
2> (compile-toplevel '(needs-arg))
** expr-2:1: needs-arg: missing arguments for params (x)
2> (compile-toplevel '(list
1
2
(needs-arg)))
** expr-2:4: needs-arg: missing arguments for params (x)

Yet, we can report the error to a line number. Which means that
all expressions have line information attached to them, just in
case there may be some error found in them in a later code walk.

On 2023-08-10, Malcolm McLean <malcolm.arthur.mclean@gmail.com> wrote:
> On Thursday, 10 August 2023 at 13:21:12 UTC+1, Bart wrote:
>> Searching backwards from the token's offset, determines the line-number,
>> and offset with the line.
>>
> There's a strong case for doing that. If a file is read in as a string, then any IO
> erros can be dealt with in one place.

Only if you have lexical, syntax, type, semantic and all other error
checking that needs a line number all in one place, in one pass.

On Thursday, 10 August 2023 at 16:40:40 UTC+1, Kaz Kylheku wrote:
> On 2023-08-10, Malcolm McLean <malcolm.ar...@gmail.com> wrote:
> > On Thursday, 10 August 2023 at 13:21:12 UTC+1, Bart wrote:
> >> Searching backwards from the token's offset, determines the line-number,
> >> and offset with the line.
> >>
> > There's a strong case for doing that. If a file is read in as a string, then any IO
> > erros can be dealt with in one place.
> Only if you have lexical, syntax, type, semantic and all other error
> checking that needs a line number all in one place, in one pass.
>
The "lexer" is fgetc(). You can tell because the code makes large numbers of ungetc()
calls. I'm rewriting it any way. Maybe I should do a complete rewrite to have a
proper lexer with a "gettoken()" and "match()" function. Then, as you say, all calls
to fgetc() will be in the same place, and it becomes easy to count lines and columns.

The main problem with a lexer for XML is that the grammar specifies "anything except
a special characer" for the data between nodes. So either you have to have a special
mode, so the lexer isn't really a lexer any more, or the tokens have to be single characters
anyway.

IO errors are of a different nature to parse errors. By far the most likely IO error is
failure to open the file because the wrong name was provided. But any call to
fgetc() can return EOF. If you load all the data as a string in a first pass, then
you pick up any IO errors, and further ones are not possible.

On Thu, 10 Aug 2023 15:38:47 -0000 (UTC)
Kaz Kylheku <864-117-4973@kylheku.com> wrote:
> On 2023-08-10, Malcolm McLean <malcolm.arthur.mclean@gmail.com> wrote:
> > Just adding some better error reporting to my XML parser.
> >
> > I started out by keeping track of the line number as I went. But of
> > course that's very fiddly.
>
> It's only slightly fiddly if you have to recognize tokens that can
> contain embedded newlines.

I load the whole file into memory and , as I do so , I also recognise ends of
lines and create an array where line_numbers[i] = j means that the line
number i starts at byte j .So I can get from a byte number to a line number
by a binary search. All further parsing is done after I have the whole file
in memory.

> In a language in which a newline only occurs in a comment, or
> whitespace, you can have a couple of dedicated tokenizing rules
> which increment the line number.
>
> You attach the current line number to all token objects and then
> the parser can just refer to them.
>
> If you allow tokens with newlines like
>
> "multi
> line
> literal"
>
> by a single token match that gobbles newlines, you then need
> some slight fiddling, like walking the lexeme of that object after
> it is extracted and bumping the line number for each newline
> that occurs.

[...]

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:

> The main problem with a lexer for XML is that the grammar specifies
> "anything except a special characer" for the data between nodes. So
> either you have to have a special mode, so the lexer isn't really a
> lexer any more, or the tokens have to be single characters anyway.

No. It's almost universal to have such tokens. A C string is (to a
first approximation) anything except a '"'. A C comment is anything up
to a '*' followed by a '/'. In the most common kind of lexer, what you
think of as "modes" are states in a finite-state machine recogniser for
a regular language. For example, in C, when it sees a letter or '_' the
lexer enters the ID "mode" and accepts characters until the first
non-alphanumeric. It's just a state machine.

--
Ben.

piątek, 11 sierpnia 2023 o 13:21:38 UTC+2 Ben Bacarisse napisał(a):
> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>
> > The main problem with a lexer for XML is that the grammar specifies
> > "anything except a special characer" for the data between nodes. So
> > either you have to have a special mode, so the lexer isn't really a
> > lexer any more, or the tokens have to be single characters anyway.
> No. It's almost universal to have such tokens. A C string is (to a
> first approximation) anything except a '"'. A C comment is anything up
> to a '*' followed by a '/'. In the most common kind of lexer, what you
> think of as "modes" are states in a finite-state machine recogniser for
> a regular language. For example, in C, when it sees a letter or '_' the
> lexer enters the ID "mode" and accepts characters until the first
> non-alphanumeric. It's just a state machine.
>
interesting i didnt know c works that way (i man to be open on unknown characters and if so they are legal in c - this is kind of this opennes
i probably suffice when i say C is better language than i
most probably probbaly could invent ), hovever what is the proof
c is like that? (and standard does not convince me as imo
standard is totally not c, better tradition of early c which is imo much more
important)

fir <profesor.fir@gmail.com> writes:

> piątek, 11 sierpnia 2023 o 13:21:38 UTC+2 Ben Bacarisse napisał(a):
>> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>>
>> > The main problem with a lexer for XML is that the grammar specifies
>> > "anything except a special characer" for the data between nodes. So
>> > either you have to have a special mode, so the lexer isn't really a
>> > lexer any more, or the tokens have to be single characters anyway.
>>
>> No. It's almost universal to have such tokens. A C string is (to a
>> first approximation) anything except a '"'. A C comment is anything up
>> to a '*' followed by a '/'. In the most common kind of lexer, what you
>> think of as "modes" are states in a finite-state machine recogniser for
>> a regular language. For example, in C, when it sees a letter or '_' the
>> lexer enters the ID "mode" and accepts characters until the first
>> non-alphanumeric. It's just a state machine.
>>
> interesting i didnt know c works that way (i man to be open on unknown
> characters and if so they are legal in c - this is kind of this
> opennes i probably suffice when i say C is better language than i most
> probably probbaly could invent ), hovever what is the proof c is like
> that? (and standard does not convince me as imo standard is totally
> not c, better tradition of early c which is imo much more important)

There's a question about C in there but I don't really know what you are
asking. I think the answer is no, C does not work like that. When I
said "to a first approximation" I meant that I've omitted lots of
details and I think details are what you are asking about.

--
Ben.

On Friday, 11 August 2023 at 12:21:38 UTC+1, Ben Bacarisse wrote:
> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>
> > The main problem with a lexer for XML is that the grammar specifies
> > "anything except a special characer" for the data between nodes. So
> > either you have to have a special mode, so the lexer isn't really a
> > lexer any more, or the tokens have to be single characters anyway.
> No. It's almost universal to have such tokens. A C string is (to a
> first approximation) anything except a '"'. A C comment is anything up
> to a '*' followed by a '/'. In the most common kind of lexer, what you
> think of as "modes" are states in a finite-state machine recogniser for
> a regular language. For example, in C, when it sees a letter or '_' the
> lexer enters the ID "mode" and accepts characters until the first
> non-alphanumeric. It's just a state machine.
>
The difference is that in C, or for that matter, Minibasic, when you hit an
alpha (or underscore), you known that you just have to read off all
the subsequent alnums to get the token. Free form text can only appear
inside quotes or comments. So if you hit a quote you gobble the string,
and if you hit a comment you just pass through until you hit a "close
comment" sequence (or in Mininbasic, just skip the REM line)

Wiht XML it's not like that. Free form text is delineated by tags. But you've
got to know whether you are in a tag or not to read it. So if you are to have
a FREEFORMTEXT token, the lexer needs to know whether it has just parsed
an element opening tag or not.

On 2023-08-12, Malcolm McLean <malcolm.arthur.mclean@gmail.com> wrote:
> On Friday, 11 August 2023 at 12:21:38 UTC+1, Ben Bacarisse wrote:
>> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>>
>> > The main problem with a lexer for XML is that the grammar specifies
>> > "anything except a special characer" for the data between nodes. So
>> > either you have to have a special mode, so the lexer isn't really a
>> > lexer any more, or the tokens have to be single characters anyway.
>> No. It's almost universal to have such tokens. A C string is (to a
>> first approximation) anything except a '"'. A C comment is anything up
>> to a '*' followed by a '/'. In the most common kind of lexer, what you
>> think of as "modes" are states in a finite-state machine recogniser for
>> a regular language. For example, in C, when it sees a letter or '_' the
>> lexer enters the ID "mode" and accepts characters until the first
>> non-alphanumeric. It's just a state machine.
>>
> The difference is that in C, or for that matter, Minibasic, when you hit an
> alpha (or underscore), you known that you just have to read off all
> the subsequent alnums to get the token. Free form text can only appear
> inside quotes or comments. So if you hit a quote you gobble the string,
> and if you hit a comment you just pass through until you hit a "close
> comment" sequence (or in Mininbasic, just skip the REM line)
>
> Wiht XML it's not like that. Free form text is delineated by tags. But you've
> got to know whether you are in a tag or not to read it. So if you are to have
> a FREEFORMTEXT token, the lexer needs to know whether it has just parsed
> an element opening tag or not.

XML isn't HTML, but the things you're getting at are mostly common.

A sequence of characters that is not special can be treated as a token;
it's a repetition of an inverted regex character class.

A decade ago I made a tiny project called hc (HTML Cleaner). It parses
HTML and removes all tags that are not permitted, or certain disallowed
attributes of tags that are permitted.

The use case is this: allowing HTML e-mails into a Web-based mailing
list archive, with the HTML intact (but cleaned).

https://www.kylheku.com/cgit/hc/tree/hc.l

You can see that it's straight Lex rules.

The lexer has two exclusive states in addition to the initial one: ELM
and ATT.

The rule for matching a ream of text is one or more notspecial
characters:

{notspecial}+ { return tok_text; }

where that is defined as a negated character class; not any
of these characters.

notspecial [^"'<>/=& \t\n\r\v\t]

This stops at whitespace; which is returned as a different token.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

On Saturday, 12 August 2023 at 08:26:29 UTC+1, Kaz Kylheku wrote:
> On 2023-08-12, Malcolm McLean <malcolm.ar...@gmail.com> wrote:
> > On Friday, 11 August 2023 at 12:21:38 UTC+1, Ben Bacarisse wrote:
> >> Malcolm McLean <malcolm.ar...@gmail.com> writes:
> >>
> >> > The main problem with a lexer for XML is that the grammar specifies
> >> > "anything except a special characer" for the data between nodes. So
> >> > either you have to have a special mode, so the lexer isn't really a
> >> > lexer any more, or the tokens have to be single characters anyway.
> >> No. It's almost universal to have such tokens. A C string is (to a
> >> first approximation) anything except a '"'. A C comment is anything up
> >> to a '*' followed by a '/'. In the most common kind of lexer, what you
> >> think of as "modes" are states in a finite-state machine recogniser for
> >> a regular language. For example, in C, when it sees a letter or '_' the
> >> lexer enters the ID "mode" and accepts characters until the first
> >> non-alphanumeric. It's just a state machine.
> >>
> > The difference is that in C, or for that matter, Minibasic, when you hit an
> > alpha (or underscore), you known that you just have to read off all
> > the subsequent alnums to get the token. Free form text can only appear
> > inside quotes or comments. So if you hit a quote you gobble the string,
> > and if you hit a comment you just pass through until you hit a "close
> > comment" sequence (or in Mininbasic, just skip the REM line)
> >
> > Wiht XML it's not like that. Free form text is delineated by tags. But you've
> > got to know whether you are in a tag or not to read it. So if you are to have
> > a FREEFORMTEXT token, the lexer needs to know whether it has just parsed
> > an element opening tag or not.
> XML isn't HTML, but the things you're getting at are mostly common.
>
> A sequence of characters that is not special can be treated as a token;
> it's a repetition of an inverted regex character class.
>
> A decade ago I made a tiny project called hc (HTML Cleaner). It parses
> HTML and removes all tags that are not permitted, or certain disallowed
> attributes of tags that are permitted.
>
> The use case is this: allowing HTML e-mails into a Web-based mailing
> list archive, with the HTML intact (but cleaned).
>
> https://www.kylheku.com/cgit/hc/tree/hc.l
>
> You can see that it's straight Lex rules.
>
> The lexer has two exclusive states in addition to the initial one: ELM
> and ATT.
>
> The rule for matching a ream of text is one or more notspecial
> characters:
>
> {notspecial}+ { return tok_text; }
>
>
> where that is defined as a negated character class; not any
> of these characters.
>
> notspecial [^"'<>/=& \t\n\r\v\t]
>
> This stops at whitespace; which is returned as a different token.
>
Your lexer hardcodes the HTML element and attribute names. Which is of
course the core purpose of a lexer. To convert the keywords to single value
tokens to make the grammar easier to write. But XML doesn't have any keywords,
just a few special tokens.
So I wrote the "vanilla" XML parser without a lexer. But whilst it is OK for Baby X
resource script files, which are very simple XML, it won't stand up to XML
in the wild. Also, it only reports success or fail. The benefit of a lexer is that
you can easily store the number of the line where you encoutner a parse error,
which is more user-friendly.

I'm having a go at an XML parser mark 2, with a formal lexer. But the problem is
this this legal XML
<Text>Text</Text>
<Text attr="Text">More Text</Text>

Now you could say that "<{identifer pattern}>" matches the token "open tag", with
the value "Text". But that won't work for the second line. You could say that "<"
matches the token "start tag definfition", but then "Text" has to match the token
"element name", and you will match it in the freeform text. You can of course get the
attribute value "Text" out relatively easily because it is quoted string. The token is
"quoted string" with the value "Text".

If you've got. a single character = 1 token lexer you can do it all with grammar rules,
however.

On Friday, 11 August 2023 at 12:21:38 UTC+1, Ben Bacarisse wrote:
> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>
> > The main problem with a lexer for XML is that the grammar specifies
> > "anything except a special characer" for the data between nodes. So
> > either you have to have a special mode, so the lexer isn't really a
> > lexer any more, or the tokens have to be single characters anyway.
> No. It's almost universal to have such tokens. A C string is (to a
> first approximation) anything except a '"'. A C comment is anything up
> to a '*' followed by a '/'. In the most common kind of lexer, what you
> think of as "modes" are states in a finite-state machine recogniser for
> a regular language. For example, in C, when it sees a letter or '_' the
> lexer enters the ID "mode" and accepts characters until the first
> non-alphanumeric. It's just a state machine.
>
I've more or less rewritten the XML parser. The old ad hoc logic has been abandoned
and the parser written entirely from scratch, on top of a single character lexer.
You might say it's not a lexer at all, but the advantages are huge. Obviously keeping
track of the line number if no longer a problem. It's much easier to add informative
parse errors. And it can be trivially toggled between file or string input, or UTF-16 (you'd
have to have a little UTF-16 to UTF-8 converter in the get character function) The
structure of the program is far better. It's much more maintainable.

The only big disadvantge of. a single character lexer is that the tokens "<?" to introduce
the metadata tag, and "<!--" to introduce a comment can't be encoded as single tokens.
So when you hit a "<" you have to match it and parse from the character following it,
which isn't as nice as parsing from the "<".

Quoted strings and identifiers are all recognised using grammar rules, rather than in the
lexer, which would be more conventional.

Sadly the new XML parser is no going to make it into v1.2 because it's too big. a change
too close to release.