Some people here are interested in Haskell.
They might be interested in this:

https://chrisdone.com/posts/fast-haskell-c-parsing-xml/

Of course it's written from a pro-Haskell point of view, and writing an improved version when you've got the C in front of you isn't really a fair test. But he does match C for speed.

On 16/08/2023 08:31, Malcolm McLean wrote:
> Some people here are interested in Haskell.
> They might be interested in this:
>
> https://chrisdone.com/posts/fast-haskell-c-parsing-xml/
>
> Of course it's written from a pro-Haskell point of view, and writing an improved version when you've got the C in front of you isn't really a fair test. But he does match C for speed.
>

"Portability (i.e. Windows) is a pain in the arse with C."

I wonder what makes them say that?

Reading from a file must be the world's most portable kind of program.
While issues with filenames and paths will be the same whatever the
language.

So what is it?

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:

> Some people here are interested in Haskell.
> They might be interested in this:
>
> https://chrisdone.com/posts/fast-haskell-c-parsing-xml/

Interesting. Thanks.

> Of course it's written from a pro-Haskell point of view, and writing
> an improved version when you've got the C in front of you isn't really
> a fair test. But he does match C for speed.

A lot of the refinements are ones one might try anyway, so I don't think
it's as directed as you suggest. Sadly, a lot of the improvements come
from targeted hints to a clever implementation so it's not always
Haskell the language that is giving the speed, but the phenomenal piece
of work that is the Glasgow Haskell compiler.

--
Ben.

Bart <bc@freeuk.com> writes:

> On 16/08/2023 08:31, Malcolm McLean wrote:
>> Some people here are interested in Haskell.
>> They might be interested in this:
>> https://chrisdone.com/posts/fast-haskell-c-parsing-xml/
>> Of course it's written from a pro-Haskell point of view, and writing an
>> improved version when you've got the C in front of you isn't really a
>> fair test. But he does match C for speed.
>>
>
> "Portability (i.e. Windows) is a pain in the arse with C."
>
> I wonder what makes them say that?

Yes, I wondered that too, since the cut-down XML parsing they are doing
is one of the most potentially portable bits of C one could write (as
you say yourself):

> Reading from a file must be the world's most portable kind of
> program.

But reading more closely, the remark is a general one about dropping out
of a high-level language for some part of a program rather than being
specific to this task. None the less, I'd have liked a citation or
link.

> While issues with filenames and paths will be the same whatever
> the language.

Not always. Some languages have standard library functions to handle
such things (e.g. Python and Haskell). I imagine that's what the author
was thinking about.

--
Ben.

środa, 16 sierpnia 2023 o 09:31:17 UTC+2 Malcolm McLean napisał(a):
> Some people here are interested in Haskell.
> They might be interested in this:
>
> https://chrisdone.com/posts/fast-haskell-c-parsing-xml/
>
> Of course it's written from a pro-Haskell point of view, and writing an improved version when you've got the C in front of you isn't really a fair test. But he does match C for speed.

many articles on programing are written by noobs/ponts and judging from some sentences
here its probably a noob/pony (i wouldnt say definitely as some things may dependant on soem contextes, but being 90% a chance of being pony is enough to keep reserve for such kind of articles)

for example its terribly noobish to judge languages based on ibraries in those languages, its preposterously stupid ..many pones write pony code even in c - you itself wrote thsi pice with fgetc in loop which showed to be so slow - if some wnt to judge and conclude better compare real good code in c with real good code in other language .. besides all that conclusions are finally nonintersting as i generally know what things look like (in an aspect of speed - learning language for constructions may be valuable for sme other reasons)

środa, 16 sierpnia 2023 o 12:14:20 UTC+2 Bart napisał(a):
> On 16/08/2023 08:31, Malcolm McLean wrote:
> > Some people here are interested in Haskell.
> > They might be interested in this:
> >
> > https://chrisdone.com/posts/fast-haskell-c-parsing-xml/
> >
> > Of course it's written from a pro-Haskell point of view, and writing an improved version when you've got the C in front of you isn't really a fair test. But he does match C for speed.
> >
> "Portability (i.e. Windows) is a pain in the arse with C."
>
> I wonder what makes them say that?
>
> Reading from a file must be the world's most portable kind of program.
> While issues with filenames and paths will be the same whatever the
> language.
>
> So what is it?

ponys have special areas of ponys trash talking "portability" is one of that areas, other are for example "standard/undefined behaviour" or "dont diccover a wheel" or "oop is good" etc...its pony world full of belifs myths lies and general bulshit

On 2023-08-16, Bart <bc@freeuk.com> wrote:
> On 16/08/2023 08:31, Malcolm McLean wrote:
>> Some people here are interested in Haskell.
>> They might be interested in this:
>>
>> https://chrisdone.com/posts/fast-haskell-c-parsing-xml/
>>
>> Of course it's written from a pro-Haskell point of view, and writing an improved version when you've got the C in front of you isn't really a fair test. But he does match C for speed.
>>
>
> "Portability (i.e. Windows) is a pain in the arse with C."
>
> I wonder what makes them say that?
>
> Reading from a file must be the world's most portable kind of program.
> While issues with filenames and paths will be the same whatever the
> language.
>
> So what is it?

Portability of advanced programs that go beyond the standard C library.

For instance, say, a program that opens a serial device and sets the
baud rate, framing/parity bits and hardware handshaking, and then
reads from it --- with timeouts.

POSIX is quite different from Win32 in every way. Everything is done
differently: ranging from a bit differently, to radically.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

czwartek, 17 sierpnia 2023 o 02:37:57 UTC+2 Kaz Kylheku napisał(a):
> On 2023-08-16, Bart <b...@freeuk.com> wrote:
> > On 16/08/2023 08:31, Malcolm McLean wrote:
> >> Some people here are interested in Haskell.
> >> They might be interested in this:
> >>
> >> https://chrisdone.com/posts/fast-haskell-c-parsing-xml/
> >>
> >> Of course it's written from a pro-Haskell point of view, and writing an improved version when you've got the C in front of you isn't really a fair test. But he does match C for speed.
> >>
> >
> > "Portability (i.e. Windows) is a pain in the arse with C."
> >
> > I wonder what makes them say that?
> >
> > Reading from a file must be the world's most portable kind of program.
> > While issues with filenames and paths will be the same whatever the
> > language.
> >
> > So what is it?
> Portability of advanced programs that go beyond the standard C library.
>
> For instance, say, a program that opens a serial device and sets the
> baud rate, framing/parity bits and hardware handshaking, and then
> reads from it --- with timeouts.
>
> POSIX is quite different from Win32 in every way. Everything is done
> differently: ranging from a bit differently, to radically.
>
and in haskell you get it more portable?

czwartek, 17 sierpnia 2023 o 02:32:15 UTC+2 fir napisał(a):
> środa, 16 sierpnia 2023 o 12:14:20 UTC+2 Bart napisał(a):
> > On 16/08/2023 08:31, Malcolm McLean wrote:
> > > Some people here are interested in Haskell.
> > > They might be interested in this:
> > >
> > > https://chrisdone.com/posts/fast-haskell-c-parsing-xml/
> > >
> > > Of course it's written from a pro-Haskell point of view, and writing an improved version when you've got the C in front of you isn't really a fair test. But he does match C for speed.
> > >
> > "Portability (i.e. Windows) is a pain in the arse with C."
> >
> > I wonder what makes them say that?
> >
> > Reading from a file must be the world's most portable kind of program.
> > While issues with filenames and paths will be the same whatever the
> > language.
> >
> > So what is it?
> ponys have special areas of ponys trash talking "portability" is one of that areas, other are for example "standard/undefined behaviour" or "dont diccover a wheel" or "oop is good" etc...its pony world full of belifs myths lies and general bulshit

this pony areas list is ofc longer and it grows depending of how many things soem inspects itself the more myths pony lives in shows..this way reading pony articles that propagate myths is somewhat rather a negative value

On Thursday, 17 August 2023 at 00:23:26 UTC+1, Ben Bacarisse wrote:
> Bart <b...@freeuk.com> writes:
>
> > On 16/08/2023 08:31, Malcolm McLean wrote:
> >> Some people here are interested in Haskell.
> >> They might be interested in this:
> >> https://chrisdone.com/posts/fast-haskell-c-parsing-xml/
> >> Of course it's written from a pro-Haskell point of view, and writing an
> >> improved version when you've got the C in front of you isn't really a
> >> fair test. But he does match C for speed.
> >>
> >
> > "Portability (i.e. Windows) is a pain in the arse with C."
> >
> > I wonder what makes them say that?
> Yes, I wondered that too, since the cut-down XML parsing they are doing
> is one of the most potentially portable bits of C one could write (as
> you say yourself):
>
There are some gotchas with files, but not for the cut down parsing
they implement.
Windows used to accept "rt" for reading a text stream. And there's still
a mess with Unicode. And the XML people say that a parser must accept
UTF-16.
I implement this by having the lexer call a function pointer to read a
UTF-8 character from a stream. If the file is UTF-16, it converts to UTF-8
on the fly. If it's UTF-8, it's just an alias for fgetc().
But how do you know the file format? I have code that does this, but if
I called it, the XML parser would no longer be a single file module. So
I read the first few character of the file, the seek back to the start
position.
Bu this only works on seekable streams. So the high-level parse function
which accepts a stream rather than a file name either has to insist on
a seekable stream, or it has to insist that the stream be in known format,
or the stream access function has to maintain a little buffer. The last
solution is the real one, but it's such a fiddly thing that instead I decided on the
known format (if you call with a FILE * rather than a filename, the data must
be in UTF-8).
But it is a complete pain which would have been avoided with a high-level
language which just loads a text file.

On Thursday, August 17, 2023 at 3:37:57 AM UTC+3, Kaz Kylheku wrote:
> On 2023-08-16, Bart <b...@freeuk.com> wrote:
> > On 16/08/2023 08:31, Malcolm McLean wrote:
> >> Some people here are interested in Haskell.
> >> They might be interested in this:
> >>
> >> https://chrisdone.com/posts/fast-haskell-c-parsing-xml/
> >>
> >> Of course it's written from a pro-Haskell point of view, and writing an improved version when you've got the C in front of you isn't really a fair test. But he does match C for speed.
> >>
> >
> > "Portability (i.e. Windows) is a pain in the arse with C."
> >
> > I wonder what makes them say that?
> >
> > Reading from a file must be the world's most portable kind of program.
> > While issues with filenames and paths will be the same whatever the
> > language.
> >
> > So what is it?
> Portability of advanced programs that go beyond the standard C library.
>
> For instance, say, a program that opens a serial device and sets the
> baud rate, framing/parity bits and hardware handshaking, and then
> reads from it --- with timeouts.
>
> POSIX is quite different from Win32 in every way. Everything is done
> differently: ranging from a bit differently, to radically.
>

Semantic difference between POSIX and Win32 APIs is quite small
relatively to expected difference between randomly chosen pair of
unrelated operation systems.
More so, at OS API/concepts level Win32 is more similar to
[unrelated] POSIX than to [related] VMS.

Kernel-user; processes-threads-preemptive multitasking;
memory not shared between processes by default, but can be shared
on request; hierarchical file systems; files=collections of bytes;
block devices, character devices; everything is a file descriptor (handle)
except few things that are not; etc...
We are used to take all this similarities for granted due to little exposure
to more diverse world.

> --
> TXR Programming Language: http://nongnu.org/txr
> Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
> Mastodon: @Kazi...@mstdn.ca

środa, 16 sierpnia 2023 o 09:31:17 UTC+2 Malcolm McLean napisał(a):
> Some people here are interested in Haskell.
> They might be interested in this:
>
> https://chrisdone.com/posts/fast-haskell-c-parsing-xml/
>
> Of course it's written from a pro-Haskell point of view, and writing an improved version when you've got the C in front of you isn't really a fair test. But he does match C for speed.

he does match the c library.. and this library is probably sloppy written, so its a test library vs library (saying haskel environment is a library) ...and if so so what?
you got hevy thousands of libraries and saying this one non interesting is faster then thsi one not interesting is of not much of big value if someone want learbn something more "groundfull" ... more worse af it not inspect low lewel things
this overal outcome is propagating myths and this is in fact bad harmful activity

i name personally some peopple in programming that live in myths and propagate myths as ponys (though thsi is not definition, ponys are ponys, but they propagate myths and live in myths, write articles on myths etc)

czwartek, 17 sierpnia 2023 o 12:32:45 UTC+2 fir napisał(a):
> środa, 16 sierpnia 2023 o 09:31:17 UTC+2 Malcolm McLean napisał(a):
> > Some people here are interested in Haskell.
> > They might be interested in this:
> >
> > https://chrisdone.com/posts/fast-haskell-c-parsing-xml/
> >
> > Of course it's written from a pro-Haskell point of view, and writing an improved version when you've got the C in front of you isn't really a fair test. But he does match C for speed.
> he does match the c library.. and this library is probably sloppy written, so its a test library vs library (saying haskel environment is a library) ..and if so so what?
> you got hevy thousands of libraries and saying this one non interesting is faster then thsi one not interesting is of not much of big value if someone want learbn something more "groundfull" ... more worse af it not inspect low lewel things
> this overal outcome is propagating myths and this is in fact bad harmful activity
>
> i name personally some peopple in programming that live in myths and propagate myths as ponys (though thsi is not definition, ponys are ponys, but they propagate myths and live in myths, write articles on myths etc)

btw those numbers 0.2 ms for 0.2 MB are besides not bad /quite high so
i wouldnt maybe say its neccesarely badly written as speed is good,
but if some of haskel beats c lib then those part of c lib is sloppy written (or test
may also be uiunfair) but this is banal and obvious conclusion so this article
outcome is at most banal but it is worse in fact as it uses this myth-talking

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:

> On Thursday, 17 August 2023 at 00:23:26 UTC+1, Ben Bacarisse wrote:
>> Bart <b...@freeuk.com> writes:
>>
>> > On 16/08/2023 08:31, Malcolm McLean wrote:
>> >> Some people here are interested in Haskell.
>> >> They might be interested in this:
>> >> https://chrisdone.com/posts/fast-haskell-c-parsing-xml/
>> >> Of course it's written from a pro-Haskell point of view, and writing an
>> >> improved version when you've got the C in front of you isn't really a
>> >> fair test. But he does match C for speed.
>> >>
>> >
>> > "Portability (i.e. Windows) is a pain in the arse with C."
>> >
>> > I wonder what makes them say that?
>> Yes, I wondered that too, since the cut-down XML parsing they are doing
>> is one of the most potentially portable bits of C one could write (as
>> you say yourself):
>>
> There are some gotchas with files, but not for the cut down parsing
> they implement.
> Windows used to accept "rt" for reading a text stream. And there's
> still a mess with Unicode.

None of that matters for the case in point. The C code treats the input
like a stream of 8-bit bytes. You can do that without regard to line
convention.

> And the XML people say that a parser must
> accept UTF-16.

Again, that's not relevant to the case in the article. But it's also a
completely different issue. An XML parser that must handle either UTF-8
or UTF-16 needs a layer below the parser (conceptually) to detect the
encoding and return "characters" (as I think you have done). There is
no reason to suppose that that can't be written in portable C.

> I implement this by having the lexer call a function pointer to read a
> UTF-8 character from a stream. If the file is UTF-16, it converts to UTF-8
> on the fly.

Exactly -- though I think I would not have converted to UTF-8 in a plain
parser. Maybe your application make that a good choice.

> If it's UTF-8, it's just an alias for fgetc().
> But how do you know the file format?

The first character much be '<' (and, technically, it must be the '<' that
opens an XML declaration). The encoding should be clear from the first
two bytes.

> I have code that does this, but if
> I called it, the XML parser would no longer be a single file module. So
> I read the first few character of the file, the seek back to the start
> position.
> But this only works on seekable streams.

Actually, you don't need to read more that one character to determine if
the file is UTF-8 or UTF-16, all you need to do is an ungetc call and
that works on non-seekable streams.

> So the high-level parse function which accepts a stream rather than a
> file name either has to insist on a seekable stream, or it has to
> insist that the stream be in known format, or the stream access
> function has to maintain a little buffer. The last solution is the
> real one, but it's such a fiddly thing

Given that you convert UTF-16 to UTF-8, I'd have thought it was the
natural choice, even though you can get away with just an ungetc
call. But then I don't know how your code is organised. What's fiddly
about it?

> that instead I decided on the
> known format (if you call with a FILE * rather than a filename, the
> data must be in UTF-8). But it is a complete pain which would have
> been avoided with a high-level language which just loads a text file.

Agreed. Though I don't think the world is that good at agreeing things.
It's possible that this is not what the author had in mind with the
high/low-level portable code remark, but it's not a clear-cut case. When
the world has decided that a text file can just be opened and read, such
a facility could be provided by standard C, and even if not standard, it
could probably be written in portable C.

--
Ben.

On 2023-08-17, Michael S <already5chosen@yahoo.com> wrote:
> On Thursday, August 17, 2023 at 3:37:57 AM UTC+3, Kaz Kylheku wrote:
>> On 2023-08-16, Bart <b...@freeuk.com> wrote:
>> > On 16/08/2023 08:31, Malcolm McLean wrote:
>> >> Some people here are interested in Haskell.
>> >> They might be interested in this:
>> >>
>> >> https://chrisdone.com/posts/fast-haskell-c-parsing-xml/
>> >>
>> >> Of course it's written from a pro-Haskell point of view, and writing an improved version when you've got the C in front of you isn't really a fair test. But he does match C for speed.
>> >>
>> >
>> > "Portability (i.e. Windows) is a pain in the arse with C."
>> >
>> > I wonder what makes them say that?
>> >
>> > Reading from a file must be the world's most portable kind of program.
>> > While issues with filenames and paths will be the same whatever the
>> > language.
>> >
>> > So what is it?
>> Portability of advanced programs that go beyond the standard C library.
>>
>> For instance, say, a program that opens a serial device and sets the
>> baud rate, framing/parity bits and hardware handshaking, and then
>> reads from it --- with timeouts.
>>
>> POSIX is quite different from Win32 in every way. Everything is done
>> differently: ranging from a bit differently, to radically.
>>
>
> Semantic difference between POSIX and Win32 APIs is quite small
> relatively to expected difference between randomly chosen pair of
> unrelated operation systems.

Yes; if we take a broad, and especially historic perspective, sure.

TOPS-10 versus OS/360, or whatever random pair.

It's the syntactic differences that cause the difficulty. If
the concept behind a functional area is very similar, but the
actual functional area looks different at the API level, then
you're basically writing the code twice.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

On Thursday, 17 August 2023 at 12:19:55 UTC+1, Ben Bacarisse wrote:
> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>
> > On Thursday, 17 August 2023 at 00:23:26 UTC+1, Ben Bacarisse wrote:
> >> Bart <b...@freeuk.com> writes:
> >>
> >> > On 16/08/2023 08:31, Malcolm McLean wrote:
> >> >> Some people here are interested in Haskell.
> >> >> They might be interested in this:
> >> >> https://chrisdone.com/posts/fast-haskell-c-parsing-xml/
> >> >> Of course it's written from a pro-Haskell point of view, and writing an
> >> >> improved version when you've got the C in front of you isn't really a
> >> >> fair test. But he does match C for speed.
> >> >>
> >> >
> >> > "Portability (i.e. Windows) is a pain in the arse with C."
> >> >
> >> > I wonder what makes them say that?
> >> Yes, I wondered that too, since the cut-down XML parsing they are doing
> >> is one of the most potentially portable bits of C one could write (as
> >> you say yourself):
> >>
> > There are some gotchas with files, but not for the cut down parsing
> > they implement.
> > Windows used to accept "rt" for reading a text stream. And there's
> > still a mess with Unicode.
> None of that matters for the case in point. The C code treats the input
> like a stream of 8-bit bytes. You can do that without regard to line
> convention.
> > And the XML people say that a parser must
> > accept UTF-16.
> Again, that's not relevant to the case in the article. But it's also a
> completely different issue. An XML parser that must handle either UTF-8
> or UTF-16 needs a layer below the parser (conceptually) to detect the
> encoding and return "characters" (as I think you have done). There is
> no reason to suppose that that can't be written in portable C.
> > I implement this by having the lexer call a function pointer to read a
> > UTF-8 character from a stream. If the file is UTF-16, it converts to UTF-8
> > on the fly.
> Exactly -- though I think I would not have converted to UTF-8 in a plain
> parser. Maybe your application make that a good choice.
> > If it's UTF-8, it's just an alias for fgetc().
> > But how do you know the file format?
> The first character much be '<' (and, technically, it must be the '<' that
> opens an XML declaration). The encoding should be clear from the first
> two bytes.
> > I have code that does this, but if
> > I called it, the XML parser would no longer be a single file module. So
> > I read the first few character of the file, the seek back to the start
> > position.
> > But this only works on seekable streams.
>
> Actually, you don't need to read more that one character to determine if
> the file is UTF-8 or UTF-16, all you need to do is an ungetc call and
> that works on non-seekable streams.
>
You need two characters, because you might have a UTF-16 little-endian
stream without a BOM. So the first character in 8 bit bytes would be '<'.
But there's a simple hack, which is to read the first character from the
stream, then set up the lexer with a "<' sitting in its token. So of course
you also have to read the first character when passed a string, which
is a bit of a nuisance (and that's the sort of thing that gives programming
such a bad reputation). But it should work now when piped a non-seekable
UTF-16 stream.

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:

> On Thursday, 17 August 2023 at 12:19:55 UTC+1, Ben Bacarisse wrote:
>> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>>
>> > On Thursday, 17 August 2023 at 00:23:26 UTC+1, Ben Bacarisse wrote:
>> >> Bart <b...@freeuk.com> writes:
>> >>
>> >> > On 16/08/2023 08:31, Malcolm McLean wrote:
>> >> >> Some people here are interested in Haskell.
>> >> >> They might be interested in this:
>> >> >> https://chrisdone.com/posts/fast-haskell-c-parsing-xml/
>> >> >> Of course it's written from a pro-Haskell point of view, and writing an
>> >> >> improved version when you've got the C in front of you isn't really a
>> >> >> fair test. But he does match C for speed.
>> >> >>
>> >> >
>> >> > "Portability (i.e. Windows) is a pain in the arse with C."
>> >> >
>> >> > I wonder what makes them say that?
>> >> Yes, I wondered that too, since the cut-down XML parsing they are doing
>> >> is one of the most potentially portable bits of C one could write (as
>> >> you say yourself):
>> >>
>> > There are some gotchas with files, but not for the cut down parsing
>> > they implement.
>> > Windows used to accept "rt" for reading a text stream. And there's
>> > still a mess with Unicode.
>> None of that matters for the case in point. The C code treats the input
>> like a stream of 8-bit bytes. You can do that without regard to line
>> convention.
>> > And the XML people say that a parser must
>> > accept UTF-16.
>> Again, that's not relevant to the case in the article. But it's also a
>> completely different issue. An XML parser that must handle either UTF-8
>> or UTF-16 needs a layer below the parser (conceptually) to detect the
>> encoding and return "characters" (as I think you have done). There is
>> no reason to suppose that that can't be written in portable C.
>> > I implement this by having the lexer call a function pointer to read a
>> > UTF-8 character from a stream. If the file is UTF-16, it converts to UTF-8
>> > on the fly.
>> Exactly -- though I think I would not have converted to UTF-8 in a plain
>> parser. Maybe your application make that a good choice.
>> > If it's UTF-8, it's just an alias for fgetc().
>> > But how do you know the file format?
>> The first character much be '<' (and, technically, it must be the '<' that
>> opens an XML declaration). The encoding should be clear from the first
>> two bytes.

I ended up looking at the spec (that's an hour I'll never get back!) and
it's more complicated...

>> > I have code that does this, but if
>> > I called it, the XML parser would no longer be a single file module. So
>> > I read the first few character of the file, the seek back to the start
>> > position.
>> > But this only works on seekable streams.
>>
>> Actually, you don't need to read more that one character to determine if
>> the file is UTF-8 or UTF-16, all you need to do is an ungetc call and
>> that works on non-seekable streams.
>>
> You need two characters, because you might have a UTF-16 little-endian
> stream without a BOM. So the first character in 8 bit bytes would be
> '<'.

Yes, I wasn't thinking. Thanks. You can't always tell until the second
byte, but you don't have to "unget" anything in that case because you
now know the character.

But as it happens I spoke way too soon... The full picture is a mess.

> But there's a simple hack, which is to read the first character from the
> stream, then set up the lexer with a "<' sitting in its token. So of course
> you also have to read the first character when passed a string, which
> is a bit of a nuisance (and that's the sort of thing that gives programming
> such a bad reputation).

What do you mean "when passed a string"? Do you mean when the parser is
acting on in-memory data?

> But it should work now when piped a non-seekable
> UTF-16 stream.

It turns out that if you want to be 100% conforming you need to be able
to detect both UCS-4 and (eye roll) EBCDIC. What's more, you need to
set up just enough of the reading mechanism to be able to read the XML
declaration and then adjust the reading mechanism to handle the named
encoding. For your application, ISO-8859-1 might be effectively the
same as ISO-8859-15, but UCS-4 is a complication and you might want to
flag certain errors if the encoding is named as ISO-10646-UCS-2 rather
than UTF-16.

While this can obviously be done in C, I would much rather do it in
Haskell. Haskell's lazy evaluation gives you stream IO for free (so to
speak), and handling the tail of a lazy stream with functions computed
by looking at the start of it comes naturally in Haskell.

--
Ben.

On Saturday, 19 August 2023 at 00:15:25 UTC+1, Ben Bacarisse wrote:
> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>
> > On Thursday, 17 August 2023 at 12:19:55 UTC+1, Ben Bacarisse wrote:
> >> Malcolm McLean <malcolm.ar...@gmail.com> writes:
> >>
> >> > On Thursday, 17 August 2023 at 00:23:26 UTC+1, Ben Bacarisse wrote:
> >> >> Bart <b...@freeuk.com> writes:
> >> >>
> >> >> > On 16/08/2023 08:31, Malcolm McLean wrote:
> >> >> >> Some people here are interested in Haskell.
> >> >> >> They might be interested in this:
> >> >> >> https://chrisdone.com/posts/fast-haskell-c-parsing-xml/
> >> >> >> Of course it's written from a pro-Haskell point of view, and writing an
> >> >> >> improved version when you've got the C in front of you isn't really a
> >> >> >> fair test. But he does match C for speed.
> >> >> >>
> >> >> >
> >> >> > "Portability (i.e. Windows) is a pain in the arse with C."
> >> >> >
> >> >> > I wonder what makes them say that?
> >> >> Yes, I wondered that too, since the cut-down XML parsing they are doing
> >> >> is one of the most potentially portable bits of C one could write (as
> >> >> you say yourself):
> >> >>
> >> > There are some gotchas with files, but not for the cut down parsing
> >> > they implement.
> >> > Windows used to accept "rt" for reading a text stream. And there's
> >> > still a mess with Unicode.
> >> None of that matters for the case in point. The C code treats the input
> >> like a stream of 8-bit bytes. You can do that without regard to line
> >> convention.
> >> > And the XML people say that a parser must
> >> > accept UTF-16.
> >> Again, that's not relevant to the case in the article. But it's also a
> >> completely different issue. An XML parser that must handle either UTF-8
> >> or UTF-16 needs a layer below the parser (conceptually) to detect the
> >> encoding and return "characters" (as I think you have done). There is
> >> no reason to suppose that that can't be written in portable C.
> >> > I implement this by having the lexer call a function pointer to read a
> >> > UTF-8 character from a stream. If the file is UTF-16, it converts to UTF-8
> >> > on the fly.
> >> Exactly -- though I think I would not have converted to UTF-8 in a plain
> >> parser. Maybe your application make that a good choice.
> >> > If it's UTF-8, it's just an alias for fgetc().
> >> > But how do you know the file format?
> >> The first character much be '<' (and, technically, it must be the '<' that
> >> opens an XML declaration). The encoding should be clear from the first
> >> two bytes.
> I ended up looking at the spec (that's an hour I'll never get back!) and
> it's more complicated...
> >> > I have code that does this, but if
> >> > I called it, the XML parser would no longer be a single file module. So
> >> > I read the first few character of the file, the seek back to the start
> >> > position.
> >> > But this only works on seekable streams.
> >>
> >> Actually, you don't need to read more that one character to determine if
> >> the file is UTF-8 or UTF-16, all you need to do is an ungetc call and
> >> that works on non-seekable streams.
> >>
> > You need two characters, because you might have a UTF-16 little-endian
> > stream without a BOM. So the first character in 8 bit bytes would be
> > '<'.
> Yes, I wasn't thinking. Thanks. You can't always tell until the second
> byte, but you don't have to "unget" anything in that case because you
> now know the character.
>
> But as it happens I spoke way too soon... The full picture is a mess.
>
Yes, it's awful. You have an "encoding" field in XML 1.0. But you can't
depend on it because not all XML is version 1.0, some of it is bare. Now
I don't have much experience with text, but I reckon that it's entirely
possible that someone would run XML through a program like iconv,
and it won't be clever enough to change the "encoding" field:
>
> > But there's a simple hack, which is to read the first character from the
> > stream, then set up the lexer with a "<' sitting in its token. So of course
> > you also have to read the first character when passed a string, which
> > is a bit of a nuisance (and that's the sort of thing that gives programming
> > such a bad reputation).
> What do you mean "when passed a string"? Do you mean when the parser is
> acting on in-memory data?
>
Sorry, I was so close to the program that I forgot that everybody else knows
nothing of the code (It's on GitHub but not in the resource compiler, it's in a
separate project). You can pass it either a file name, an open stream, or a
string. The string has to be UTF-8 because it is a char *. Of course I have to
read the first character of the string to make the string work the same way
as the rest of the code, all to support UTF-16 without a BOM on non-seekable
streams.
> > But it should work now when piped a non-seekable
> > UTF-16 stream.
> It turns out that if you want to be 100% conforming you need to be able
> to detect both UCS-4 and (eye roll) EBCDIC. What's more, you need to
> set up just enough of the reading mechanism to be able to read the XML
> declaration and then adjust the reading mechanism to handle the named
> encoding. For your application, ISO-8859-1 might be effectively the
> same as ISO-8859-15, but UCS-4 is a complication and you might want to
> flag certain errors if the encoding is named as ISO-10646-UCS-2 rather
> than UTF-16.
>
The XML people say that a parser must accept UTF-8 and UTF-16. I have
heard of files which switch encodings, but I think they are largely mythical.
The basic idea of XML was very good, but I'm not impressed with the standard.
>
> While this can obviously be done in C, I would much rather do it in
> Haskell. Haskell's lazy evaluation gives you stream IO for free (so to
> speak), and handling the tail of a lazy stream with functions computed
> by looking at the start of it comes naturally in Haskell.
>
The structure of the C function is massively improved by going to a lexer and
having a proper hierarchical, recursive grammar rather than the old ad-hoc
system. (Which was tempting because basic XML is so simple).
However it might be possible to do a much better job in Haskell. Unfortunately
I can't do that better job.
I'm confident that it is shaping up as a very good single file C XML parser,
however.

On Saturday, 19 August 2023 at 00:15:25 UTC+1, Ben Bacarisse wrote:
>
> It turns out that if you want to be 100% conforming you need to be able
> to detect both UCS-4 and (eye roll) EBCDIC.
>
I had a go at ECBDIC.

If anyone has an EBCDIC XML file they'd like to test, please post a link.

Of course the next challenge is to support ECBDIC as the execution character
set. This means all the if (ch == '<') statements have to come out and be replaced
by if (ch == ASCII_LESSTHEN). And the strings have to be replaced with hex codes.

Here's where the Baby X resource compiler shows its power. Simply set up the input
<BabyXRC>
<utf8 name="cdata"><CDATA</utf8>
</BabyXRC>

And so on, and you get all the strings in hex-encoded UTF-8, ready to cut and paste.

On Sat, 19 Aug 2023 03:04:27 -0700, Malcolm McLean wrote:

> On Saturday, 19 August 2023 at 00:15:25 UTC+1, Ben Bacarisse wrote:
>>
>> It turns out that if you want to be 100% conforming you need to be able
>> to detect both UCS-4 and (eye roll) EBCDIC.
>>
> I had a go at ECBDIC.
>
> If anyone has an EBCDIC XML file they'd like to test, please post a link.

Be careful of what you ask for, Malcolm

You /do/ realize that "EBCDIC" refers to a whole family of charactersets,
(at least 46 individual charactersets, most with /some/ common elements)
and /not/ to a single characterset like Unicode or US-ASCII (although, you
could argue that ASCII embodied multiple charactersets, just with fewer
variants).

FWIW, there are a number EBCDIC charactersets that you could not reliably use
in XML, as they lack a few of the required characters. You might take a
look at the DKUUG's characterset standards website[1] - they contributed to
the ISO/IEC JTC 1/SC 2 [2] effort to catalogue and standardize charactersets.

[1] http://std.dkuug.dk/i18n/charmaps/
[2] https://en.wikipedia.org/wiki/ISO/IEC_JTC_1/SC_2

[snip]

--
Lew Pitcher
"In Skills We Trust"

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>On Saturday, 19 August 2023 at 00:15:25 UTC+1, Ben Bacarisse wrote:
>>
>> It turns out that if you want to be 100% conforming you need to be able
>> to detect both UCS-4 and (eye roll) EBCDIC.
>>
>I had a go at ECBDIC.
>
>If anyone has an EBCDIC XML file they'd like to test, please post a link.

Here's one:

Lo���@�������~�K�@��������~���`�on%Lo���`����������@����~���m��������K���@����~����a���@on%LO�������@���������@������@���������K���n%LO``@Ö�������@M�]@����`����@���@Ӊ�����K@���@������@��������K@``n%LO``@㈉�@��������@��@Ö����������K@㈉�@��������@���@����@��@����@���@�����������@��@����������@����@���@�����@��@���@���������@�������@����@��@���@���@���@�����@����@���@���������@����@��������@��K@``n%LO``@㈅@����@���������@��@����@��������@��@�����������@���@�������@��@������@��@����������@���������@�������@������K@``n%L��������m����n%@@L���������n%@@@@L��������@��m��������~㙤�@��m��������~Ɓ���@��m������~Ɓ���@��m��������~Ɓ���@��m����m�����~Ɓ���n%@@@@@@L���m�����m����n����m���La���m�����m����n%@@@@@@L���m����m����nԁ��@��@م������La���m����m����n%@@@@@@%@@@@@@%@@@@@@%@@@@@@L���m�������@��������m������~㙤�@���m���m������~㙤�@�����m������~ą���n%@@@@@@@@L���m���������ną���La���m���������n%@@@@@@@@%@@@@@@@@L���m������nL���������n����La���������nLa���m������n%@@@@@@La���m�������n%@@@@@@L���m�����m�����n���La���m�����m�����n%@@@@@@L���m������n%@@@@@@@@L���m������m�����n%@@@@@@@@@@L���m������m�����ną�����La���m������m�����n%@@@@
@@@@@@L���m������m����n��La���m������m����n%@@@@@@@@La���m������m�����n@%@@@@@@La���m������n%@@@@@@L���m��������n%@@@@@@@@L���m�������n%@@@@@@@@@@L������m����@��������~�������`����m���K���n����m���La������m����n%@@@@@@@@@@L������m����n�������������La������m����n%@@@@@@@@@@L������m���������m�����n�������La������m���������m�����n%@@@@@@@@@@%@@@@@@@@@@%@@@@@@@@@@%@@@@@@@@@@%@@@@@@@@@@%@@@@@@@@@@%@@@@@@@@@@%@@@@@@@@@@%@@@@@@@@La���m�������nL���m�������n%@@@@@@@@@@L������m����@��������~�������`����K���n����La������m����n%@@@@@@@@@@L������m����n�������������La������m����n%@@@@@@@@@@L������m���������m�����n�������La������m���������m�����n%@@@@@@@@@@%@@@@@@@@@@%@@@@@@@@@@%@@@@@@@@@@%@@@@@@@@@@%@@@@@@@@@@%@@@@@@@@@@%@@@@@@@@@@%@@@@@@@@La���m�������n%@@@@@@La���m��������n%@@@@@@L���m�������n%@@@@@@@@L�������m����nL����nי������@��������������@�����������@���@���@��k@���������@��@�����������@����@���@���@������@���@�@������@��@������KLa����nLa�������m����n%@@@@@@@@%@@@@@@La���m�������n%@@@@@@L���m������n%@@@@@@@@L���m�����nɄ������������@���������La���m�����n%@@@@@@La���m������n%@@@@@@L���m�����m�����������n%@@@@@@@@%@@@@@@La���m�����m�����������n%
@@@@@@L���m�������������n%@@@@@@@@%@@@@@@La���m�������������n%@@@@@@%@@@@@@L���m����������n%@@@@@@@@L����������m����nL����n����m���@��@�@��`���@��������KLa����nLa����������m����n%@@@@@@@@%@@@@@@La���m����������n%@@@@@@L���m���������n%@@@@@@%@@@@@@L������@������~��n%@@@@@@%@@@@@@%@@@@@@@@%L�����@��m��������m������~Ɓ���@���m�������m��������~Ɓ���@��m������m��m�������m��������~Ɓ���n%@@L�����m����nɔ���������La�����m����n%@@%@@L�����m���n��La�����m���n%@@L�����m���n��La�����m���n%@@%@@L�����m�����������@�����~������nL����n㈅@ɔ���������@����K@㈉�@�����@����@����@��@�����������@����@����@���@����@��������@��@���K@��������@�����@�������@���@���������zLa����nLa�����m�����������n%@@L�����m�����������@�����~������nL�����nL������@����~�nL�����nL���nL�����nȅ�@��������������La�����nL�����n�����@��������������La�����nL�����nɔ���������La�����nLa���nLa�����nL�����nL���nL�����nL���������n���La���������nLa�����nL�����n�La�����nL�����n���@Ӊ�����La�����nLa���nL���nL�����nL���������n���La���������nLa�����nL�����n�La�����nL�����n™������@Ö���������La�����nLa���nL���nL�����nL���������n���La���������nLa�����nL�����n�La�����nL�����nÁ����@ɕ�KLa�����nLa���
nL���nL�����nL���������n���La���������nLa�����nL�����n�La�����nL�����nĉ�����@Ř�������@Ö���������La�����nLa���nL���nL�����nL���������n��La���������nLa�����nL�����n�La�����nL�����nɕ������@ㅃ���������@��La�����nLa���nL���nL�����nL���������n���La���������nLa�����nL�����n�La�����nL�����nԖ������@��@ƙ�������@⅔����������@ɕ�KLa�����nLa���nL���nL�����nL���������n���La���������nLa�����nL�����n�La�����nL�����n�����@Ö���������La�����nLa���nL���nL�����nL���������n��La���������nLa�����nL�����n�La�����nL�����n�������@ԉ���@É������@Ö���������La�����nLa���nL���nL�����nL���������n��La���������nLa�����nL�����n�La�����nL�����nؤ������@ɕ�KLa�����nLa���nL���nL�����nL���������n�La���������nLa�����nL�����n�La�����nL�����nԁ�����@ɕ�����������@ӣ�KLa�����nLa���nL���nL�����nL���������n�La���������nLa�����nL�����n�La�����nL�����nɕ���@Ö���������La�����nLa���nLa�����nLa������nLa�����nLa�����m�����������n%@@L�����m�����������@�����~������nL����n���@���@������@�����@����@���@���@���������@��@����@������K@���@������@���@��������@��@���@���@��������@���@����@���@��@����KLa����nLa�����m�����������n%@@%@@L�����m������n%@@@@%@@@@%@@@@%@@La�����m������n%@@%@@%@@L�����m������
n%@@@@%@@La�����m������n%@@%@@%@@%La�����n%%@@@@@@%@@@@@@@@%L�����@��m��������m������~Ɓ���@���m�������m��������~Ɓ���@��m������m��m�������m��������~Ɓ���n%@@L�����m����n偙����La�����m����n%@@%@@L�����m���n��La�����m���n%@@L�����m���n��La�����m���n%@@%@@L�����m�����������@�����~������nL����n��@L���`�������`����n�������������@�������La���`�������`����n@�������@������K@㨗������k@����@�����@��@����@��@�����������@�������@���������@�������@��������k@��@�����@���������@��@�@�������KLa����nLa�����m�����������n%@@%@@L�����m������n%@@@@%@@@@%@@@@%@@La�����m������n%@@%@@%@@L�����m������n%@@@@%@@La�����m������n%@@%@@%@@%La�����n%%@@@@@@%@@@@@@@@%L�����@��m��������m������~Ɓ���@���m�������m��������~Ɓ���@��m������m��m�������m��������~Ɓ���n%@@L�����m����n������������La�����m����n%@@%@@L�����m���n�La�����m���n%@@L�����m���n�La�����m���n%@@%@@L�����m�����������@�����~������nL����n㈅@���������@������@��@����@�����@���zLa����nLa�����m�����������n%@@%@@L�����m������n%@@@@%@@@@%@@@@L�����m�����m��������n%@@@@@@L�����m�����n����La�����m�����n%@@@@@@%@@@@@@L�����m�����m�����������nL����n��ԥ�La����nLa�����m�����m�����������n
%@@@@@@%@@@@@@%@@@@La�����m�����m��������nL�����m�����m��������n%@@@@@@L�����m�����n����La�����m�����n%@@@@@@%@@@@@@L�����m�����m�����������nL����n��ԥ��La����nLa�����m�����m�����������n%@@@@@@%@@@@@@%@@@@La�����m�����m��������nL�����m�����m��������n%@@@@@@L�����m�����n����La�����m�����n%@@@@@@%@@@@@@L�����m�����m�����������nL����n��ԥ�@M��������]La����nLa�����m�����m�����������n%@@@@@@%@@@@@@%@@@@La�����m�����m��������nL�����m�����m��������n%@@@@@@L�����m�����n����La�����m�����n%@@@@@@%@@@@@@L�����m�����m�����������nL����n��ԥ��La����nLa�����m�����m�����������n%@@@@@@%@@@@@@%@@@@La�����m�����m��������nL�����m�����m��������n%@@@@@@L�����m�����n����La�����m�����n%@@@@@@%@@@@@@L�����m�����m�����������nL����n��ԥ���La����nLa�����m�����m�����������n%@@@@@@%@@@@@@%@@@@La�����m�����m��������nL�����m�����m��������n%@@@@@@L�����m�����n����La�����m�����n%@@@@@@%@@@@@@L�����m�����m�����������nL����n��ԥ����La����nLa�����m�����m�����������n%@@@@@@%@@@@@@%@@@@La�����m�����m��������nL�����m�����m��������n%@@@@@@L�����m�����n����La�����m�����n%@@@@@@%@@@@@@L�����m�����m�����������nL����n��ԥ�La����nLa�����m�����m�����������n%@@@@@@%@@@@@@%@@@@La�����m�����m
��������nL�����m�����m��������n%@@@@@@L�����m�����n����La�����m�����n%@@@@@@%@@@@@@L�����m�����m�����������nL����ną�����@��@�����@������La����nLa�����m�����m�����������n%@@@@@@%@@@@@@%@@@@La�����m�����m��������n%@@La�����m������n%@@L�����m�����������@�����~�����nL����n���@�����@������@���@��������KLa����nLa�����m�����������n%@@%@@%@@L�����m������n%@@@@%@@La�����m������n%@@%@@%@@%La�����n%%@@@@@@%@@@@@@@@%L�����@��m��������m������~Ɓ���@���m�������m��������~Ɓ���@��m������m��m�������m��������~Ɓ���n%@@L�����m����nׁ��դ�La�����m����n%@@%@@L�����m���n�La�����m���n%@@L�����m���n�La�����m���n%@@%@@L�����m�����������@�����~������nL����n��@L���`�������`����n�������������@�������La���`�������`����n@�������@����@������@���@���@������KLa����nLa�����m�����������n%@@L�����m�����������@�����~������nL����n֕@����������@�����������@��@���k@��@���@���@����@����@��@���@�������@����@������@���@L���������n��La���������n@��@L���������n�La���������nk@���@�������@���@������������@���@�������@�����������KLa����nLa�����m�����������n%@@%@@L�����m������n%@@@@%@@@@%@@@@%@@La�����m������n%@@%@@%@@L�����m������n%@@@@%@@La�����m������n%@@%@@%@@%La
�����n%%@@@@@@%@@@@@@@@%L�����@��m��������m������~Ɓ���@���m�������m��������~Ɓ���@��m������m��m�������m��������~Ɓ���n%@@L�����m����nم������La�����m����n%@@%@@L�����m���n�La�����m���n%@@L�����m���n�La�����m���n%@@%@@L�����m�����������@�����~������nL����n��@L���`�������`����n�������������@�������La���`�������`����n@��������@������@���@���@������KLa����nLa�����m�����������n%@@%@@L�����m������n%@@@@%@@@@%@@@@%@@La�����m������n%@@%@@%@@L�����m������n%@@@@%@@La�����m������n%@@%@@%@@%La�����n%%@@@@@@%@@@@@@@@La������n%@@@@@@La���m���������n%@@@@@@%@@@@La��������n%@@La���������n%@@L���������n��a��a����@�z�La���������n%La��������m����n

On Sat, 19 Aug 2023 14:48:08 +0000, Scott Lurndal wrote:

> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>On Saturday, 19 August 2023 at 00:15:25 UTC+1, Ben Bacarisse wrote:
>>>
>>> It turns out that if you want to be 100% conforming you need to be able
>>> to detect both UCS-4 and (eye roll) EBCDIC.
>>>
>>I had a go at ECBDIC.
>>
>>If anyone has an EBCDIC XML file they'd like to test, please post a link.
>
> Here's one:
[snip]

And that's an excellent illustration of my point about some EBCDIC
charactersets lacking the necessary characters to properly express XML.

Here are the first four lines of the ASCII equivalent of that message,
as generated by
dd if=ebcdic.msg of=ascii.msg conv=ascii
where
conv=ascii
will convert "from EBCDIC to ASCII" (dd(1) manpage)

Note the (translated) format of the DOCTYPE entities
<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet href="one_register.xsl" type="text/xsl" ?>
<|DOCTYPE registers SYSTEM "registers.dtd">
<|-- Copyright (c) 2010-2014 ARM Limited. All rights reserved. -->

Apparently, you used a variant of EBCDIC that includes an exclamation mark
at codepoint 0x4f; dd uses EBCDIC-US which, at codepoint 0x4f encodes
a "VERTICAL LINE"

--
Lew Pitcher
"In Skills We Trust"

On Sat, 19 Aug 2023 15:09:32 +0000, Lew Pitcher wrote:

> On Sat, 19 Aug 2023 14:48:08 +0000, Scott Lurndal wrote:
>
>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>On Saturday, 19 August 2023 at 00:15:25 UTC+1, Ben Bacarisse wrote:
>>>>
>>>> It turns out that if you want to be 100% conforming you need to be able
>>>> to detect both UCS-4 and (eye roll) EBCDIC.
>>>>
>>>I had a go at ECBDIC.
>>>
>>>If anyone has an EBCDIC XML file they'd like to test, please post a link.
>>
>> Here's one:
> [snip]
>
> And that's an excellent illustration of my point about some EBCDIC
> charactersets lacking the necessary characters to properly express XML.
>
> Here are the first four lines of the ASCII equivalent of that message,
> as generated by
> dd if=ebcdic.msg of=ascii.msg conv=ascii
> where
> conv=ascii
> will convert "from EBCDIC to ASCII" (dd(1) manpage)
>
> Note the (translated) format of the DOCTYPE entities
> <?xml version="1.0" encoding="utf-8"?>

Oh, and bye the way, that "encoding" value is incorrect for
the XML document you posted. It should have named the
EBCDIC variant you used, not "utf-8".

I suspect that you just machine or hand encoded an existing
utf-8 XML document, rather than compose a completely new
document in EBCDIC

FWIW, I spent many years working in an EBCDIC environment,
manipulating XML documents (in EBCDIC) with a tool developed
in-house. I had to write a number of "white papers" on the
subjects of characterset translation (to/from EBCDIC, and
between EBCDIC variants), and on XML handling in an EBCDIC
environment. :-)

--
Lew Pitcher
"In Skills We Trust"

scott@slp53.sl.home (Scott Lurndal) writes:

> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>On Saturday, 19 August 2023 at 00:15:25 UTC+1, Ben Bacarisse wrote:
>>>
>>> It turns out that if you want to be 100% conforming you need to be able
>>> to detect both UCS-4 and (eye roll) EBCDIC.
>>>
>>I had a go at ECBDIC.
>>
>>If anyone has an EBCDIC XML file they'd like to test, please post a link.
>
> Here's one:
>
> Lo...

<EBCDIC-encoded XML deleted>

Is that legal? I thought an EBCDIC XML file must give the correct
encoding in the XML declaration. xmllint rejects it unless I edit the
declaration.

--
Ben.

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:

> On Saturday, 19 August 2023 at 00:15:25 UTC+1, Ben Bacarisse wrote:
>> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>>
>> > On Thursday, 17 August 2023 at 12:19:55 UTC+1, Ben Bacarisse wrote:
>> >> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>> >>
>> >> > On Thursday, 17 August 2023 at 00:23:26 UTC+1, Ben Bacarisse wrote:
>> >> >> Bart <b...@freeuk.com> writes:
>> >> >>
>> >> >> > On 16/08/2023 08:31, Malcolm McLean wrote:
>> >> >> >> Some people here are interested in Haskell.
>> >> >> >> They might be interested in this:
>> >> >> >> https://chrisdone.com/posts/fast-haskell-c-parsing-xml/
>> >> >> >> Of course it's written from a pro-Haskell point of view, and writing an
>> >> >> >> improved version when you've got the C in front of you isn't really a
>> >> >> >> fair test. But he does match C for speed.
>> >> >> >>
>> >> >> >
>> >> >> > "Portability (i.e. Windows) is a pain in the arse with C."
>> >> >> >
>> >> >> > I wonder what makes them say that?
>> >> >> Yes, I wondered that too, since the cut-down XML parsing they are doing
>> >> >> is one of the most potentially portable bits of C one could write (as
>> >> >> you say yourself):
>> >> >>
>> >> > There are some gotchas with files, but not for the cut down parsing
>> >> > they implement.
>> >> > Windows used to accept "rt" for reading a text stream. And there's
>> >> > still a mess with Unicode.
>> >> None of that matters for the case in point. The C code treats the input
>> >> like a stream of 8-bit bytes. You can do that without regard to line
>> >> convention.
>> >> > And the XML people say that a parser must
>> >> > accept UTF-16.
>> >> Again, that's not relevant to the case in the article. But it's also a
>> >> completely different issue. An XML parser that must handle either UTF-8
>> >> or UTF-16 needs a layer below the parser (conceptually) to detect the
>> >> encoding and return "characters" (as I think you have done). There is
>> >> no reason to suppose that that can't be written in portable C.
>> >> > I implement this by having the lexer call a function pointer to read a
>> >> > UTF-8 character from a stream. If the file is UTF-16, it converts to UTF-8
>> >> > on the fly.
>> >> Exactly -- though I think I would not have converted to UTF-8 in a plain
>> >> parser. Maybe your application make that a good choice.
>> >> > If it's UTF-8, it's just an alias for fgetc().
>> >> > But how do you know the file format?
>> >> The first character much be '<' (and, technically, it must be the '<' that
>> >> opens an XML declaration). The encoding should be clear from the first
>> >> two bytes.
>> I ended up looking at the spec (that's an hour I'll never get back!) and
>> it's more complicated...
>> >> > I have code that does this, but if
>> >> > I called it, the XML parser would no longer be a single file module. So
>> >> > I read the first few character of the file, the seek back to the start
>> >> > position.
>> >> > But this only works on seekable streams.
>> >>
>> >> Actually, you don't need to read more that one character to determine if
>> >> the file is UTF-8 or UTF-16, all you need to do is an ungetc call and
>> >> that works on non-seekable streams.
>> >>
>> > You need two characters, because you might have a UTF-16 little-endian
>> > stream without a BOM. So the first character in 8 bit bytes would be
>> > '<'.
>> Yes, I wasn't thinking. Thanks. You can't always tell until the second
>> byte, but you don't have to "unget" anything in that case because you
>> now know the character.
>>
>> But as it happens I spoke way too soon... The full picture is a mess.
>>
> Yes, it's awful. You have an "encoding" field in XML 1.0. But you can't
> depend on it because not all XML is version 1.0, some of it is bare. Now
> I don't have much experience with text, but I reckon that it's entirely
> possible that someone would run XML through a program like iconv,
> and it won't be clever enough to change the "encoding" field:

Maybe. But you are providing a tool and you don't have to accept
everything. A converted document with the wrong XML declaration is an
error and you could just reject it. You don't have to bend over
backwards for bad input.

Not being a Windows user, I've not seen a UTF-16 encoded file in the
wild, so I would not even accept that. In the Linux world, I'd probably
accept only UTF-8 and point my users to xmllint.

xmllint can read any valid XML file and re-write it using UTF-8 (or,
indeed, many other encodings), changing the XML declaration on the fly.
Hence

xmllint -encode UTF-8 myresources | babyxrc

would be close to a universal XML processor with little extra work on
your part. But maybe you probably can't assume your users will want to
do that.

>> It turns out that if you want to be 100% conforming you need to be able
>> to detect both UCS-4 and (eye roll) EBCDIC. What's more, you need to
>> set up just enough of the reading mechanism to be able to read the XML
>> declaration and then adjust the reading mechanism to handle the named
>> encoding. For your application, ISO-8859-1 might be effectively the
>> same as ISO-8859-15, but UCS-4 is a complication and you might want to
>> flag certain errors if the encoding is named as ISO-10646-UCS-2 rather
>> than UTF-16.
>>
> The XML people say that a parser must accept UTF-8 and UTF-16.

Don't they go further? I thought they did. Maybe the others are
optional and those two are the only must-haves.

> I have
> heard of files which switch encodings, but I think they are largely mythical.
> The basic idea of XML was very good, but I'm not impressed with the
> standard.

There are two kinds of standards: those that incorporate lots of options
because of all the interested parties, and those that make decisive
choices between competing candidates.

--
Ben.