Lynn's comment inspired me to add Unicode support to the Baby X resource compiler. But despite searching for quite a long time, I can't find a test suite of Unicode files in various formats. In fact it's hard to find any Unicode files at all which are not UTF-8.

On Thu, 27 Jul 2023 10:22:12 -0700 (PDT)
Malcolm McLean <malcolm.arthur.mclean@gmail.com> wrote:
> Lynn's comment inspired me to add Unicode support to the Baby X resource
> compiler. But despite searching for quite a long time, I can't find a test
> suite of Unicode files in various formats. In fact it's hard to find any
> Unicode files at all which are not UTF-8.

What do you want to test and what do you mean by "formats" ? You can use
iconv to transform UTF-8 to any other encoding.

What is the C connection ?

On Thursday, 27 July 2023 at 18:47:28 UTC+1, Spiros Bousbouras wrote:
> On Thu, 27 Jul 2023 10:22:12 -0700 (PDT)
> Malcolm McLean <malcolm.ar...@gmail.com> wrote:
> > Lynn's comment inspired me to add Unicode support to the Baby X resource
> > compiler. But despite searching for quite a long time, I can't find a test
> > suite of Unicode files in various formats. In fact it's hard to find any
> > Unicode files at all which are not UTF-8.
> What do you want to test and what do you mean by "formats" ? You can use
> iconv to transform UTF-8 to any other encoding.
>
> What is the C connection ?
>
The Baby X resource compiler is a tool for C programmers.
It takes data in various formats and transforms it to compileable C code.
Currently it doesn't have any support for Unicode data. That's because my
background is in games programming and it's not something I'm very
familiar with.

I'm trying to write a function char *loadasutf8(const char *filename, int *error)
which will accept any text file, smart detect the format, and return the
contents in UTF-8. So I'm looking for Unicode files in various formats to
test it on.

On 27/07/2023 18:55, Malcolm McLean wrote:
> On Thursday, 27 July 2023 at 18:47:28 UTC+1, Spiros Bousbouras wrote:
>> On Thu, 27 Jul 2023 10:22:12 -0700 (PDT)
>> Malcolm McLean <malcolm.ar...@gmail.com> wrote:
>>> Lynn's comment inspired me to add Unicode support to the Baby X resource
>>> compiler. But despite searching for quite a long time, I can't find a test
>>> suite of Unicode files in various formats. In fact it's hard to find any
>>> Unicode files at all which are not UTF-8.
>> What do you want to test and what do you mean by "formats" ? You can use
>> iconv to transform UTF-8 to any other encoding.
>>
>> What is the C connection ?
>>
> The Baby X resource compiler is a tool for C programmers.
> It takes data in various formats and transforms it to compileable C code.
> Currently it doesn't have any support for Unicode data. That's because my
> background is in games programming and it's not something I'm very
> familiar with.
>
> I'm trying to write a function char *loadasutf8(const char *filename, int *error)
> which will accept any text file, smart detect the format, and return the
> contents in UTF-8. So I'm looking for Unicode files in various formats to
> test it on.

Most text files will already be either UTF8, or ASCII, which is a subset
of UTF8. So what format is there to detect? What formats might be expected?

Bart <bc@freeuk.com> writes:
>On 27/07/2023 18:55, Malcolm McLean wrote:
>> On Thursday, 27 July 2023 at 18:47:28 UTC+1, Spiros Bousbouras wrote:
>>> On Thu, 27 Jul 2023 10:22:12 -0700 (PDT)
>>> Malcolm McLean <malcolm.ar...@gmail.com> wrote:
>>>> Lynn's comment inspired me to add Unicode support to the Baby X resource
>>>> compiler. But despite searching for quite a long time, I can't find a test
>>>> suite of Unicode files in various formats. In fact it's hard to find any
>>>> Unicode files at all which are not UTF-8.
>>> What do you want to test and what do you mean by "formats" ? You can use
>>> iconv to transform UTF-8 to any other encoding.
>>>
>>> What is the C connection ?
>>>
>> The Baby X resource compiler is a tool for C programmers.
>> It takes data in various formats and transforms it to compileable C code.
>> Currently it doesn't have any support for Unicode data. That's because my
>> background is in games programming and it's not something I'm very
>> familiar with.
>>
>> I'm trying to write a function char *loadasutf8(const char *filename, int *error)
>> which will accept any text file, smart detect the format, and return the
>> contents in UTF-8. So I'm looking for Unicode files in various formats to
>> test it on.
>
>Most text files will already be either UTF8, or ASCII, which is a subset
>of UTF8. So what format is there to detect? What formats might be expected?

Most likely would be UCS-2 or UTF-16, both standard windows character sets.

On Thursday, 27 July 2023 at 19:22:16 UTC+1, Bart wrote:
> On 27/07/2023 18:55, Malcolm McLean wrote:
> > On Thursday, 27 July 2023 at 18:47:28 UTC+1, Spiros Bousbouras wrote:
> >> On Thu, 27 Jul 2023 10:22:12 -0700 (PDT)
> >> Malcolm McLean <malcolm.ar...@gmail.com> wrote:
> >>> Lynn's comment inspired me to add Unicode support to the Baby X resource
> >>> compiler. But despite searching for quite a long time, I can't find a test
> >>> suite of Unicode files in various formats. In fact it's hard to find any
> >>> Unicode files at all which are not UTF-8.
> >> What do you want to test and what do you mean by "formats" ? You can use
> >> iconv to transform UTF-8 to any other encoding.
> >>
> >> What is the C connection ?
> >>
> > The Baby X resource compiler is a tool for C programmers.
> > It takes data in various formats and transforms it to compileable C code.
> > Currently it doesn't have any support for Unicode data. That's because my
> > background is in games programming and it's not something I'm very
> > familiar with.
> >
> > I'm trying to write a function char *loadasutf8(const char *filename, int *error)
> > which will accept any text file, smart detect the format, and return the
> > contents in UTF-8. So I'm looking for Unicode files in various formats to
> > test it on.
> Most text files will already be either UTF8, or ASCII, which is a subset
> of UTF8. So what format is there to detect? What formats might be expected?
>
Of course. But the spirit of the resource compiler is that it's quite liberal in
what it will accept. The vast majority of Unicode files are UTF-8, so it's
trival to write a function to read them and return the contents as UTF-8.
But you can also have UTF-16 and UTF-32, with or without a byte order
marker, and in little endian or big endian format.

On 2023-07-27, Malcolm McLean <malcolm.arthur.mclean@gmail.com> wrote:
> Lynn's comment inspired me to add Unicode support to the Baby X
> resource compiler. But despite searching for quite a long time, I
> can't find a test suite of Unicode files in various formats. In fact
> it's hard to find any Unicode files at all which are not UTF-8.

It's a fool's errand to support Unicode formats other than UTF-8.

Also, I recommend ignoring the existence of the BOM (byte order
marker) that can be present at the start of a UTF-8 stream.

It makes no sense: UTF-8 has a single defined byte order: the high
order part of a code point is encoded first, when the code point
requires multiple bytes.

I've only seen BOM in UTF-8 on Windows. I suspect it's some Microsoft
bug whereby, in conversion from UTF-16, the BOM is accidentally
retained, and they may have lobbied to get that idiocy into the
Unicode spec instead of fixing their shit.

If enough programs ignore the existence of BOM and just decode and
retain it as a character, the practice of generating it will have to go
away.

The BOM could appear in a file that contains nothing but characters in
the USASCII range, thereby wantonly breaking the ASCII compatibility
that Pike and Thompson designed into UTF-8.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

On 2023-07-27, Malcolm McLean <malcolm.arthur.mclean@gmail.com> wrote:
> I'm trying to write a function char *loadasutf8(const char *filename, int *error)
> which will accept any text file, smart detect the format, and return the
> contents in UTF-8.

That function is indistinguishable from:

char *loadfile(const char *filename, int *err);

if all you do is return the UTF-8 that is already in the file.

Now, UTF-8 can contain bad bytes that don't conform to UTF-8, and it
makes sense to validate that. But the "int *err" argument can be
extended to reporting UTF-8 decoding errors.

If you just make it so that Baby X resource files are UTF-8, and nothing
but UTF-8, that handles ASCII also.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

On Thursday, 27 July 2023 at 19:54:29 UTC+1, Kaz Kylheku wrote:
> On 2023-07-27, Malcolm McLean <malcolm.ar...@gmail.com> wrote:
> > I'm trying to write a function char *loadasutf8(const char *filename, int *error)
> > which will accept any text file, smart detect the format, and return the
> > contents in UTF-8.
> That function is indistinguishable from:
>
> char *loadfile(const char *filename, int *err);
>
> if all you do is return the UTF-8 that is already in the file.
>
> Now, UTF-8 can contain bad bytes that don't conform to UTF-8, and it
> makes sense to validate that. But the "int *err" argument can be
> extended to reporting UTF-8 decoding errors.
>
> If you just make it so that Baby X resource files are UTF-8, and nothing
> but UTF-8, that handles ASCII also.
>
Whilst I work with images every day, I can't remember when I last had to
work with textual data for a serious purpose. It might be that all textual
data is either UTF-8 or ASCII, or so close that it makes no difference.

The idea of the Baby X resource compiler is that you don't need to modify
your raw source files. You give it the data and it puts it into a standard
format, and emits it as compileable C source.

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
> On Thursday, 27 July 2023 at 19:22:16 UTC+1, Bart wrote:
>> On 27/07/2023 18:55, Malcolm McLean wrote:
[...]
>> > I'm trying to write a function char *loadasutf8(const char *filename, int *error)
>> > which will accept any text file, smart detect the format, and return the
>> > contents in UTF-8. So I'm looking for Unicode files in various formats to
>> > test it on.
>> Most text files will already be either UTF8, or ASCII, which is a subset
>> of UTF8. So what format is there to detect? What formats might be expected?
>>
> Of course. But the spirit of the resource compiler is that it's quite liberal in
> what it will accept. The vast majority of Unicode files are UTF-8, so it's
> trival to write a function to read them and return the contents as UTF-8.
> But you can also have UTF-16 and UTF-32, with or without a byte order
> marker, and in little endian or big endian format.

UTF-8 files are easy to find. (Some of them are also ASCII.)

UTF-16 files are common on Windows, usually little-endian, usually with
a BOM. (UCS-2 is UTF-16 with no characters outside the Basic
Multilingual Plane, characters 0-65535).

UTF-32 files are rare.

You can generate all these formats from UTF-8 using the "iconv" command.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

On Thursday, 27 July 2023 at 21:51:47 UTC+1, Keith Thompson wrote:
> Malcolm McLean <malcolm.ar...@gmail.com> writes:
> > On Thursday, 27 July 2023 at 19:22:16 UTC+1, Bart wrote:
> >> On 27/07/2023 18:55, Malcolm McLean wrote:
> [...]
> >> > I'm trying to write a function char *loadasutf8(const char *filename, int *error)
> >> > which will accept any text file, smart detect the format, and return the
> >> > contents in UTF-8. So I'm looking for Unicode files in various formats to
> >> > test it on.
> >> Most text files will already be either UTF8, or ASCII, which is a subset
> >> of UTF8. So what format is there to detect? What formats might be expected?
> >>
> > Of course. But the spirit of the resource compiler is that it's quite liberal in
> > what it will accept. The vast majority of Unicode files are UTF-8, so it's
> > trival to write a function to read them and return the contents as UTF-8.
> > But you can also have UTF-16 and UTF-32, with or without a byte order
> > marker, and in little endian or big endian format.
> UTF-8 files are easy to find. (Some of them are also ASCII.)
>
> UTF-16 files are common on Windows, usually little-endian, usually with
> a BOM. (UCS-2 is UTF-16 with no characters outside the Basic
> Multilingual Plane, characters 0-65535).
>
> UTF-32 files are rare.
>
> You can generate all these formats from UTF-8 using the "iconv" command.
>
Wow, iconv --list gives far more encodings than I could possibly support.
But that solves the problem.

On 2023-07-27, Malcolm McLean <malcolm.arthur.mclean@gmail.com> wrote:
> On Thursday, 27 July 2023 at 21:51:47 UTC+1, Keith Thompson wrote:
>> You can generate all these formats from UTF-8 using the "iconv" command.
>>
> Wow, iconv --list gives far more encodings than I could possibly support.
> But that solves the problem.

It punts the problem to the user because you can say that Baby X
resource files are UTF-8, and that's it. If the user writes
the resource file in any other way, or is converting some other kind
of resource file that is in another format, they can use the iconv
tool, or something else; you never have to touch iconv.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

Kaz Kylheku <864-117-4973@kylheku.com> writes:
> On 2023-07-27, Malcolm McLean <malcolm.arthur.mclean@gmail.com> wrote:
>> On Thursday, 27 July 2023 at 21:51:47 UTC+1, Keith Thompson wrote:
>>> You can generate all these formats from UTF-8 using the "iconv" command.
>>>
>> Wow, iconv --list gives far more encodings than I could possibly support.
>> But that solves the problem.
>
> It punts the problem to the user because you can say that Baby X
> resource files are UTF-8, and that's it. If the user writes
> the resource file in any other way, or is converting some other kind
> of resource file that is in another format, they can use the iconv
> tool, or something else; you never have to touch iconv.

Sure -- assuming that the iconv application is installed, and that
the user doesn't mind an extra manual step.

UTF-16 text files are (unfortunately) common on Windows. Having a
tool quietly handle such files when they're recognized is a perfectly
valid choice.

In a similar context, I've had to work with UTF-8 and UTF-16 files
in a large software project. The fact that vim quietly handles both
formats was tremendously more convenient than if I had convert them
manually (and convert them back before updating them in git).

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

Malcolm McLean ha scritto:
> On Thursday, 27 July 2023 at 21:51:47 UTC+1, Keith Thompson wrote:
>> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>>> On Thursday, 27 July 2023 at 19:22:16 UTC+1, Bart wrote:
>>>> On 27/07/2023 18:55, Malcolm McLean wrote:
>> [...]
>>>>> I'm trying to write a function char *loadasutf8(const char *filename, int *error)
>>>>> which will accept any text file, smart detect the format, and return the
>>>>> contents in UTF-8. So I'm looking for Unicode files in various formats to
>>>>> test it on.
>>>> Most text files will already be either UTF8, or ASCII, which is a subset
>>>> of UTF8. So what format is there to detect? What formats might be expected?
>>>>
>>> Of course. But the spirit of the resource compiler is that it's quite liberal in
>>> what it will accept. The vast majority of Unicode files are UTF-8, so it's
>>> trival to write a function to read them and return the contents as UTF-8.
>>> But you can also have UTF-16 and UTF-32, with or without a byte order
>>> marker, and in little endian or big endian format.
>> UTF-8 files are easy to find. (Some of them are also ASCII.)
>>
>> UTF-16 files are common on Windows, usually little-endian, usually with
>> a BOM. (UCS-2 is UTF-16 with no characters outside the Basic
>> Multilingual Plane, characters 0-65535).
>>
>> UTF-32 files are rare.
>>
>> You can generate all these formats from UTF-8 using the "iconv" command.
>>
> Wow, iconv --list gives far more encodings than I could possibly support.
> But that solves the problem.
>

You are starting a battle that you will lose like all those who have
started it. There is no way to identify the format of a text with
safety (especially if it is imported from the web). Allow me to
recommend you to ask your users to convert their files into UTF-8
format.

jak <nospam@please.ty> writes:
> Malcolm McLean ha scritto:
>> On Thursday, 27 July 2023 at 21:51:47 UTC+1, Keith Thompson wrote:
>>> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>>>> On Thursday, 27 July 2023 at 19:22:16 UTC+1, Bart wrote:
>>>>> On 27/07/2023 18:55, Malcolm McLean wrote:
>>> [...]
>>>>>> I'm trying to write a function char *loadasutf8(const char *filename, int *error)
>>>>>> which will accept any text file, smart detect the format, and return the
>>>>>> contents in UTF-8. So I'm looking for Unicode files in various formats to
>>>>>> test it on.
>>>>> Most text files will already be either UTF8, or ASCII, which is a subset
>>>>> of UTF8. So what format is there to detect? What formats might be expected?
>>>>>
>>>> Of course. But the spirit of the resource compiler is that it's quite liberal in
>>>> what it will accept. The vast majority of Unicode files are UTF-8, so it's
>>>> trival to write a function to read them and return the contents as UTF-8.
>>>> But you can also have UTF-16 and UTF-32, with or without a byte order
>>>> marker, and in little endian or big endian format.
>>> UTF-8 files are easy to find. (Some of them are also ASCII.)
>>>
>>> UTF-16 files are common on Windows, usually little-endian, usually with
>>> a BOM. (UCS-2 is UTF-16 with no characters outside the Basic
>>> Multilingual Plane, characters 0-65535).
>>>
>>> UTF-32 files are rare.
>>>
>>> You can generate all these formats from UTF-8 using the "iconv" command.
>>>
>> Wow, iconv --list gives far more encodings than I could possibly support.
>> But that solves the problem.
>
> You are starting a battle that you will lose like all those who have
> started it. There is no way to identify the format of a text with
> safety (especially if it is imported from the web). Allow me to
> recommend you to ask your users to convert their files into UTF-8
> format.

There's no 100% reliable way to do so, but there are some very good
heuristics. The file(1) command does a reasonably good job.

Anything with a BOM is identifiable. (I agree that UTF-8 files
generally should not have a BOM, but if it's there you might as well
take advantage of it.) If there's no BOM and the first N bytes are in
the range 0..127, ASCII is a reasonable assumption; otherwise, if the
first N bytes are valid UTF-8, UTF-8 is a reasonable assumption.

(Background: a BOM, Byte Order Mark, is a character U+FEFF, ZERO WIDTH
NO-BREAK SPACE, at the beginning of a text file. It's represented as FE
FF in big-endian UTF-16, FF FE in little-endian UTF-16, and EF BB BF in
UTF-8.)

If a tool is going to be used on Windows, rejecting Windows-native text
files seems unnecessarily hostile.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

On Friday, 28 July 2023 at 09:52:55 UTC+1, Keith Thompson wrote:
> jak <nos...@please.ty> writes:
> > Malcolm McLean ha scritto:
> >> On Thursday, 27 July 2023 at 21:51:47 UTC+1, Keith Thompson wrote:
> >>> Malcolm McLean <malcolm.ar...@gmail.com> writes:
> >>>> On Thursday, 27 July 2023 at 19:22:16 UTC+1, Bart wrote:
> >>>>> On 27/07/2023 18:55, Malcolm McLean wrote:
> >>> [...]
> >>>>>> I'm trying to write a function char *loadasutf8(const char *filename, int *error)
> >>>>>> which will accept any text file, smart detect the format, and return the
> >>>>>> contents in UTF-8. So I'm looking for Unicode files in various formats to
> >>>>>> test it on.
> >>>>> Most text files will already be either UTF8, or ASCII, which is a subset
> >>>>> of UTF8. So what format is there to detect? What formats might be expected?
> >>>>>
> >>>> Of course. But the spirit of the resource compiler is that it's quite liberal in
> >>>> what it will accept. The vast majority of Unicode files are UTF-8, so it's
> >>>> trival to write a function to read them and return the contents as UTF-8.
> >>>> But you can also have UTF-16 and UTF-32, with or without a byte order
> >>>> marker, and in little endian or big endian format.
> >>> UTF-8 files are easy to find. (Some of them are also ASCII.)
> >>>
> >>> UTF-16 files are common on Windows, usually little-endian, usually with
> >>> a BOM. (UCS-2 is UTF-16 with no characters outside the Basic
> >>> Multilingual Plane, characters 0-65535).
> >>>
> >>> UTF-32 files are rare.
> >>>
> >>> You can generate all these formats from UTF-8 using the "iconv" command.
> >>>
> >> Wow, iconv --list gives far more encodings than I could possibly support.
> >> But that solves the problem.
> >
> > You are starting a battle that you will lose like all those who have
> > started it. There is no way to identify the format of a text with
> > safety (especially if it is imported from the web). Allow me to
> > recommend you to ask your users to convert their files into UTF-8
> > format.
> There's no 100% reliable way to do so, but there are some very good
> heuristics. The file(1) command does a reasonably good job.
>
> Anything with a BOM is identifiable. (I agree that UTF-8 files
> generally should not have a BOM, but if it's there you might as well
> take advantage of it.) If there's no BOM and the first N bytes are in
> the range 0..127, ASCII is a reasonable assumption; otherwise, if the
> first N bytes are valid UTF-8, UTF-8 is a reasonable assumption.
>
> (Background: a BOM, Byte Order Mark, is a character U+FEFF, ZERO WIDTH
> NO-BREAK SPACE, at the beginning of a text file. It's represented as FE
> FF in big-endian UTF-16, FF FE in little-endian UTF-16, and EF BB BF in
> UTF-8.)
>
> If a tool is going to be used on Windows, rejecting Windows-native text
> files seems unnecessarily hostile.
>
The route I took was to borrow someone else's heuristic text format
detection code. It seemed to perfom pretty well on tests. I might add
a "format" attribute to the utf8 tag to allow people to force it to load
in a certain format if the automatic detection fails. However it needs a
lot of documentation and isn't very discoverable.

Since the main point of Unicode is to support non-English languages,
there's now an "international" tag which supports internationalisation.
Strings and UTF-8 tage now have a "language" attribute. The resource
compiler then writes a simple little function which takes a string representing
the language wanted and returns the resource. This is a big change in
scope since the resource compiler is now writing executable code.

As I said, I don't work with text data professionally. So I'm very open
to comments from people with more experience.

The Baby X resource compiler is here.

https://github.com/MalcolmMcLean/babyxrc

On 28/07/2023 14:50, Malcolm McLean wrote:
> On Friday, 28 July 2023 at 09:52:55 UTC+1, Keith Thompson wrote:
>> jak <nos...@please.ty> writes:
>>> Malcolm McLean ha scritto:
>>>> On Thursday, 27 July 2023 at 21:51:47 UTC+1, Keith Thompson wrote:
>>>>> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>>>>>> On Thursday, 27 July 2023 at 19:22:16 UTC+1, Bart wrote:
>>>>>>> On 27/07/2023 18:55, Malcolm McLean wrote:
>>>>> [...]
>>>>>>>> I'm trying to write a function char *loadasutf8(const char
*filename, int *error)
>>>>>>>> which will accept any text file, smart detect the format, and
return the
>>>>>>>> contents in UTF-8. So I'm looking for Unicode files in various
formats to
>>>>>>>> test it on.
>>>>>>> Most text files will already be either UTF8, or ASCII, which is
a subset
>>>>>>> of UTF8. So what format is there to detect? What formats might
be expected?
>>>>>>>
>>>>>> Of course. But the spirit of the resource compiler is that it's
quite liberal in
>>>>>> what it will accept. The vast majority of Unicode files are
UTF-8, so it's
>>>>>> trival to write a function to read them and return the contents
as UTF-8.
>>>>>> But you can also have UTF-16 and UTF-32, with or without a byte
order
>>>>>> marker, and in little endian or big endian format.
>>>>> UTF-8 files are easy to find. (Some of them are also ASCII.)
>>>>>
>>>>> UTF-16 files are common on Windows, usually little-endian,
usually with
>>>>> a BOM. (UCS-2 is UTF-16 with no characters outside the Basic
>>>>> Multilingual Plane, characters 0-65535).
>>>>>
>>>>> UTF-32 files are rare.
>>>>>
>>>>> You can generate all these formats from UTF-8 using the "iconv"
command.
>>>>>
>>>> Wow, iconv --list gives far more encodings than I could possibly
support.
>>>> But that solves the problem.
>>>
>>> You are starting a battle that you will lose like all those who have
>>> started it. There is no way to identify the format of a text with
>>> safety (especially if it is imported from the web). Allow me to
>>> recommend you to ask your users to convert their files into UTF-8
>>> format.
>> There's no 100% reliable way to do so, but there are some very good
>> heuristics. The file(1) command does a reasonably good job.
>>
>> Anything with a BOM is identifiable. (I agree that UTF-8 files
>> generally should not have a BOM, but if it's there you might as well
>> take advantage of it.) If there's no BOM and the first N bytes are in
>> the range 0..127, ASCII is a reasonable assumption; otherwise, if the
>> first N bytes are valid UTF-8, UTF-8 is a reasonable assumption.
>>
>> (Background: a BOM, Byte Order Mark, is a character U+FEFF, ZERO WIDTH
>> NO-BREAK SPACE, at the beginning of a text file. It's represented as FE
>> FF in big-endian UTF-16, FF FE in little-endian UTF-16, and EF BB BF in
>> UTF-8.)
>>
>> If a tool is going to be used on Windows, rejecting Windows-native text
>> files seems unnecessarily hostile.
>>
> The route I took was to borrow someone else's heuristic text format
> detection code. It seemed to perfom pretty well on tests. I might add
> a "format" attribute to the utf8 tag to allow people to force it to load
> in a certain format if the automatic detection fails. However it needs a
> lot of documentation and isn't very discoverable.
>
> Since the main point of Unicode is to support non-English languages,
> there's now an "international" tag which supports internationalisation.
> Strings and UTF-8 tage now have a "language" attribute. The resource
> compiler then writes a simple little function which takes a string
representing
> the language wanted and returns the resource. This is a big change in
> scope since the resource compiler is now writing executable code.
>
> As I said, I don't work with text data professionally. So I'm very open
> to comments from people with more experience.
>
> The Baby X resource compiler is here.
>
> https://github.com/MalcolmMcLean/babyxrc
>

I took hello.c and saved it variously as UTF8, UTF8 with BOM, UTF16LE
and UTF16BE (the choices from Notepad were limited).

gcc only worked properly with the UTF8 files, with or without BOM. With
the others, it saw 'nul' every other character, but the special BOM on
those files gave errors.

Since presumably your product will be used by programmers in conjunction
with C source programs, is there any point in supporting a format that
is not also supported by the C compiler that will be needed?

Or will these diverse formats be encountered at the application level by
end-users?

On Friday, 28 July 2023 at 16:28:48 UTC+1, Bart wrote:
> On 28/07/2023 14:50, Malcolm McLean wrote:
> > On Friday, 28 July 2023 at 09:52:55 UTC+1, Keith Thompson wrote:
> >> jak <nos...@please.ty> writes:
> >>> Malcolm McLean ha scritto:
> >>>> On Thursday, 27 July 2023 at 21:51:47 UTC+1, Keith Thompson wrote:
> >>>>> Malcolm McLean <malcolm.ar...@gmail.com> writes:
> >>>>>> On Thursday, 27 July 2023 at 19:22:16 UTC+1, Bart wrote:
> >>>>>>> On 27/07/2023 18:55, Malcolm McLean wrote:
> >>>>> [...]
> >>>>>>>> I'm trying to write a function char *loadasutf8(const char
> *filename, int *error)
> >>>>>>>> which will accept any text file, smart detect the format, and
> return the
> >>>>>>>> contents in UTF-8. So I'm looking for Unicode files in various
> formats to
> >>>>>>>> test it on.
> >>>>>>> Most text files will already be either UTF8, or ASCII, which is
> a subset
> >>>>>>> of UTF8. So what format is there to detect? What formats might
> be expected?
> >>>>>>>
> >>>>>> Of course. But the spirit of the resource compiler is that it's
> quite liberal in
> >>>>>> what it will accept. The vast majority of Unicode files are
> UTF-8, so it's
> >>>>>> trival to write a function to read them and return the contents
> as UTF-8.
> >>>>>> But you can also have UTF-16 and UTF-32, with or without a byte
> order
> >>>>>> marker, and in little endian or big endian format.
> >>>>> UTF-8 files are easy to find. (Some of them are also ASCII.)
> >>>>>
> >>>>> UTF-16 files are common on Windows, usually little-endian,
> usually with
> >>>>> a BOM. (UCS-2 is UTF-16 with no characters outside the Basic
> >>>>> Multilingual Plane, characters 0-65535).
> >>>>>
> >>>>> UTF-32 files are rare.
> >>>>>
> >>>>> You can generate all these formats from UTF-8 using the "iconv"
> command.
> >>>>>
> >>>> Wow, iconv --list gives far more encodings than I could possibly
> support.
> >>>> But that solves the problem.
> >>>
> >>> You are starting a battle that you will lose like all those who have
> >>> started it. There is no way to identify the format of a text with
> >>> safety (especially if it is imported from the web). Allow me to
> >>> recommend you to ask your users to convert their files into UTF-8
> >>> format.
> >> There's no 100% reliable way to do so, but there are some very good
> >> heuristics. The file(1) command does a reasonably good job.
> >>
> >> Anything with a BOM is identifiable. (I agree that UTF-8 files
> >> generally should not have a BOM, but if it's there you might as well
> >> take advantage of it.) If there's no BOM and the first N bytes are in
> >> the range 0..127, ASCII is a reasonable assumption; otherwise, if the
> >> first N bytes are valid UTF-8, UTF-8 is a reasonable assumption.
> >>
> >> (Background: a BOM, Byte Order Mark, is a character U+FEFF, ZERO WIDTH
> >> NO-BREAK SPACE, at the beginning of a text file. It's represented as FE
> >> FF in big-endian UTF-16, FF FE in little-endian UTF-16, and EF BB BF in
> >> UTF-8.)
> >>
> >> If a tool is going to be used on Windows, rejecting Windows-native text
> >> files seems unnecessarily hostile.
> >>
> > The route I took was to borrow someone else's heuristic text format
> > detection code. It seemed to perfom pretty well on tests. I might add
> > a "format" attribute to the utf8 tag to allow people to force it to load
> > in a certain format if the automatic detection fails. However it needs a
> > lot of documentation and isn't very discoverable.
> >
> > Since the main point of Unicode is to support non-English languages,
> > there's now an "international" tag which supports internationalisation.
> > Strings and UTF-8 tage now have a "language" attribute. The resource
> > compiler then writes a simple little function which takes a string
> representing
> > the language wanted and returns the resource. This is a big change in
> > scope since the resource compiler is now writing executable code.
> >
> > As I said, I don't work with text data professionally. So I'm very open
> > to comments from people with more experience.
> >
> > The Baby X resource compiler is here.
> >
> > https://github.com/MalcolmMcLean/babyxrc
> >
> I took hello.c and saved it variously as UTF8, UTF8 with BOM, UTF16LE
> and UTF16BE (the choices from Notepad were limited).
>
> gcc only worked properly with the UTF8 files, with or without BOM. With
> the others, it saw 'nul' every other character, but the special BOM on
> those files gave errors.
>
> Since presumably your product will be used by programmers in conjunction
> with C source programs, is there any point in supporting a format that
> is not also supported by the C compiler that will be needed?
>
> Or will these diverse formats be encountered at the application level by
> end-users?
>
You've got the wrong end of the stick.

The Baby X resource compiler loads resource files (images, audio, or what we
are concerend with here, text) and saves them as portable ANSI C. So the .c
output files are ascii.
However the text data might be Unicode. So instead of using a string tag, which
creartes a C string literal, the user uses an "uft8" tag which writes the data in UTF-8,
as an array of chars.
However a good question is whether most C compilers will accept UTF-8.
UTF-8 string literals are better than hex dumped UTF-8 data, because they are
human-readable. However the standard frowns on UTF-8 in strings.

On 2023-07-28, Malcolm McLean <malcolm.arthur.mclean@gmail.com> wrote:
> On Friday, 28 July 2023 at 16:28:48 UTC+1, Bart wrote:
>> On 28/07/2023 14:50, Malcolm McLean wrote:
>> > On Friday, 28 July 2023 at 09:52:55 UTC+1, Keith Thompson wrote:
>> >> jak <nos...@please.ty> writes:
>> >>> Malcolm McLean ha scritto:
>> >>>> On Thursday, 27 July 2023 at 21:51:47 UTC+1, Keith Thompson wrote:
>> >>>>> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>> >>>>>> On Thursday, 27 July 2023 at 19:22:16 UTC+1, Bart wrote:
>> >>>>>>> On 27/07/2023 18:55, Malcolm McLean wrote:
>> >>>>> [...]
>> >>>>>>>> I'm trying to write a function char *loadasutf8(const char
>> *filename, int *error)
>> >>>>>>>> which will accept any text file, smart detect the format, and
>> return the
>> >>>>>>>> contents in UTF-8. So I'm looking for Unicode files in various
>> formats to
>> >>>>>>>> test it on.
>> >>>>>>> Most text files will already be either UTF8, or ASCII, which is
>> a subset
>> >>>>>>> of UTF8. So what format is there to detect? What formats might
>> be expected?
>> >>>>>>>
>> >>>>>> Of course. But the spirit of the resource compiler is that it's
>> quite liberal in
>> >>>>>> what it will accept. The vast majority of Unicode files are
>> UTF-8, so it's
>> >>>>>> trival to write a function to read them and return the contents
>> as UTF-8.
>> >>>>>> But you can also have UTF-16 and UTF-32, with or without a byte
>> order
>> >>>>>> marker, and in little endian or big endian format.
>> >>>>> UTF-8 files are easy to find. (Some of them are also ASCII.)
>> >>>>>
>> >>>>> UTF-16 files are common on Windows, usually little-endian,
>> usually with
>> >>>>> a BOM. (UCS-2 is UTF-16 with no characters outside the Basic
>> >>>>> Multilingual Plane, characters 0-65535).
>> >>>>>
>> >>>>> UTF-32 files are rare.
>> >>>>>
>> >>>>> You can generate all these formats from UTF-8 using the "iconv"
>> command.
>> >>>>>
>> >>>> Wow, iconv --list gives far more encodings than I could possibly
>> support.
>> >>>> But that solves the problem.
>> >>>
>> >>> You are starting a battle that you will lose like all those who have
>> >>> started it. There is no way to identify the format of a text with
>> >>> safety (especially if it is imported from the web). Allow me to
>> >>> recommend you to ask your users to convert their files into UTF-8
>> >>> format.
>> >> There's no 100% reliable way to do so, but there are some very good
>> >> heuristics. The file(1) command does a reasonably good job.
>> >>
>> >> Anything with a BOM is identifiable. (I agree that UTF-8 files
>> >> generally should not have a BOM, but if it's there you might as well
>> >> take advantage of it.) If there's no BOM and the first N bytes are in
>> >> the range 0..127, ASCII is a reasonable assumption; otherwise, if the
>> >> first N bytes are valid UTF-8, UTF-8 is a reasonable assumption.
>> >>
>> >> (Background: a BOM, Byte Order Mark, is a character U+FEFF, ZERO WIDTH
>> >> NO-BREAK SPACE, at the beginning of a text file. It's represented as FE
>> >> FF in big-endian UTF-16, FF FE in little-endian UTF-16, and EF BB BF in
>> >> UTF-8.)
>> >>
>> >> If a tool is going to be used on Windows, rejecting Windows-native text
>> >> files seems unnecessarily hostile.
>> >>
>> > The route I took was to borrow someone else's heuristic text format
>> > detection code. It seemed to perfom pretty well on tests. I might add
>> > a "format" attribute to the utf8 tag to allow people to force it to load
>> > in a certain format if the automatic detection fails. However it needs a
>> > lot of documentation and isn't very discoverable.
>> >
>> > Since the main point of Unicode is to support non-English languages,
>> > there's now an "international" tag which supports internationalisation.
>> > Strings and UTF-8 tage now have a "language" attribute. The resource
>> > compiler then writes a simple little function which takes a string
>> representing
>> > the language wanted and returns the resource. This is a big change in
>> > scope since the resource compiler is now writing executable code.
>> >
>> > As I said, I don't work with text data professionally. So I'm very open
>> > to comments from people with more experience.
>> >
>> > The Baby X resource compiler is here.
>> >
>> > https://github.com/MalcolmMcLean/babyxrc
>> >
>> I took hello.c and saved it variously as UTF8, UTF8 with BOM, UTF16LE
>> and UTF16BE (the choices from Notepad were limited).
>>
>> gcc only worked properly with the UTF8 files, with or without BOM. With
>> the others, it saw 'nul' every other character, but the special BOM on
>> those files gave errors.
>>
>> Since presumably your product will be used by programmers in conjunction
>> with C source programs, is there any point in supporting a format that
>> is not also supported by the C compiler that will be needed?
>>
>> Or will these diverse formats be encountered at the application level by
>> end-users?
>>
> You've got the wrong end of the stick.
>
> The Baby X resource compiler loads resource files (images, audio, or what we
> are concerend with here, text) and saves them as portable ANSI C. So the .c
> output files are ascii.
> However the text data might be Unicode. So instead of using a string tag, which
> creartes a C string literal, the user uses an "uft8" tag which writes the data in UTF-8,
> as an array of chars.

UTF-8 data can be a string literal. Firstly, it's commonly supported
directly nowadays, as in plonk UTF-8 between quotes and you're done.

Secondly, you can emit it as escapes, or make it optional whether
that is done or raw:

const char *utf8 = "\xE6\x97\xA5\xE6\x9C\xAC\xE8\xAA\x9E"; // 日本語

The raw UTF-8 can be (optionally) in a comment.

Unless you plan on supporting multiple encodings in the same file,
there is no point in requiring users to annotate the encoding of
individual strings.

Unicode is supposed to eliminate that "balkanization", whereby the same
set of data wrestles with multiple encodings.

The only thing you have to indicate in Unicode resource files is the
language. You have certain strings in a UI, such as "Enter your
password", which can then have multiple translations into different
languages. You have to indicate that that one is German, that one is
Korean and so on. But everything is Unicode otherwise.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
[...]
> You've got the wrong end of the stick.
>
> The Baby X resource compiler loads resource files (images, audio, or
> what we are concerend with here, text) and saves them as portable ANSI
> C. So the .c output files are ascii.
> However the text data might be Unicode. So instead of using a string
> tag, which creartes a C string literal, the user uses an "uft8" tag
> which writes the data in UTF-8, as an array of chars.
> However a good question is whether most C compilers will accept UTF-8.
> UTF-8 string literals are better than hex dumped UTF-8 data, because
> they are human-readable. However the standard frowns on UTF-8 in
> strings.

C11 introduced UTF-8 string literals. N1570 6.4.5.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

On Friday, 28 July 2023 at 19:41:14 UTC+1, Keith Thompson wrote:
> Malcolm McLean <malcolm.ar...@gmail.com> writes:
> [...]
> > You've got the wrong end of the stick.
> >
> > The Baby X resource compiler loads resource files (images, audio, or
> > what we are concerend with here, text) and saves them as portable ANSI
> > C. So the .c output files are ascii.
> > However the text data might be Unicode. So instead of using a string
> > tag, which creartes a C string literal, the user uses an "uft8" tag
> > which writes the data in UTF-8, as an array of chars.
> > However a good question is whether most C compilers will accept UTF-8.
> > UTF-8 string literals are better than hex dumped UTF-8 data, because
> > they are human-readable. However the standard frowns on UTF-8 in
> > strings.
> C11 introduced UTF-8 string literals. N1570 6.4.5.
>
I just researched this. I'd never heard of "u8" before (I don't do text processing).
Then I read this on Wikipedia:

"Since C++20 and C23, a char8_t type was added that is meant to store UTF-8
characters and the types of u8 prefixed character and string literals were
changed to char8_t and char8_t[] respectively."

If I wanted to deal with sort of thing then I'd program in C++. I learned about a
new feature and found that it was obsolete within the space of five minutes.

I don't want the Baby X resource compiler to break.
However it would be nice to get human-readable Unicode into the C source
files. But not at cost of things not working.

Incidentally "u8" will be of interest to Bart. Apparently it is "obviously" an
unsigned 8 bit type. In C11, it means "utf-8 data". Not quite the same thing.

On 29/07/2023 13:56, Malcolm McLean wrote:
> On Friday, 28 July 2023 at 19:41:14 UTC+1, Keith Thompson wrote:
>> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>> [...]
>>> You've got the wrong end of the stick.
>>>
>>> The Baby X resource compiler loads resource files (images, audio, or
>>> what we are concerend with here, text) and saves them as portable ANSI
>>> C. So the .c output files are ascii.
>>> However the text data might be Unicode. So instead of using a string
>>> tag, which creartes a C string literal, the user uses an "uft8" tag
>>> which writes the data in UTF-8, as an array of chars.
>>> However a good question is whether most C compilers will accept UTF-8.
>>> UTF-8 string literals are better than hex dumped UTF-8 data, because
>>> they are human-readable. However the standard frowns on UTF-8 in
>>> strings.
>> C11 introduced UTF-8 string literals. N1570 6.4.5.
>>
> I just researched this. I'd never heard of "u8" before (I don't do
text processing).
> Then I read this on Wikipedia:
>
> "Since C++20 and C23, a char8_t type was added that is meant to store
UTF-8
> characters and the types of u8 prefixed character and string literals
were
> changed to char8_t and char8_t[] respectively."
>
> If I wanted to deal with sort of thing then I'd program in C++. I
learned about a
> new feature and found that it was obsolete within the space of five
minutes.
>
> I don't want the Baby X resource compiler to break.
> However it would be nice to get human-readable Unicode into the C source
> files. But not at cost of things not working.
>
> Incidentally "u8" will be of interest to Bart. Apparently it is
"obviously" an
> unsigned 8 bit type. In C11, it means "utf-8 data". Not quite the
same thing.

In C11, "u8" is a valid user identifier, so that you can do this:

typedef unsigned byte u8;

But it can also be a prefix to a string literal. I believe it's called a
'contextual keyword'.

I have something similar:

f"c:\abc\def.g" # 'f' denotes a raw string literal

int f # f is also a variable (a 64-bit one!)

Actually, I can't see the point of such a prefix in C. All you have to
do is to decree that string literals are UTF8 anyway.

In any case, a C string literal defines a sequence of *any* byte values,
which or may not represent text. UTF8 extension bytes are also likely to
be negative values. It's a mess you don't really want to stir up too much.

On 2023-07-29, Malcolm McLean <malcolm.arthur.mclean@gmail.com> wrote:
> "Since C++20 and C23, a char8_t type was added that is meant to store UTF-8
> characters and the types of u8 prefixed character and string literals were
> changed to char8_t and char8_t[] respectively."

The author of that useless cruft is is still mentally living in a world
of balkanized internationalization, where one deals with multiple
encodings and character sets in the same situation.

You can handle UTF-8 in a purely C90 program. That's deliberate.
Ken Thompson and Rob Pike would not have designed an encoding scheme
that needed special literals and character types!

On Saturday, 29 July 2023 at 15:12:05 UTC+1, Kaz Kylheku wrote:
> On 2023-07-29, Malcolm McLean <malcolm.ar...@gmail.com> wrote:
> > "Since C++20 and C23, a char8_t type was added that is meant to store UTF-8
> > characters and the types of u8 prefixed character and string literals were
> > changed to char8_t and char8_t[] respectively."
> The author of that useless cruft is is still mentally living in a world
> of balkanized internationalization, where one deals with multiple
> encodings and character sets in the same situation.
>
> You can handle UTF-8 in a purely C90 program. That's deliberate.
> Ken Thompson and Rob Pike would not have designed an encoding scheme
> that needed special literals and character types!
>
Unfortunately UTF-8 naive routines only usually work with UTF-8 input,
not always. By using a special type, you can indicate that the function is
expecting UTF-8. However very many functions which take strings put
some restrictions on which strings are acceptable as arguments, and
UTF-8 isn't really in a special category in this respect.

Then ASCII files can't display human-readable Unicode. Which is a
disadvantage. But any C compiler will compile ASCII source and any
editor will display it. I'm not so sure what the situation is with UTF-8
source. clang is fine with it, of course.

czwartek, 27 lipca 2023 o 20:50:09 UTC+2 Kaz Kylheku napisał(a):
> On 2023-07-27, Malcolm McLean <malcolm.ar...@gmail.com> wrote:
> > Lynn's comment inspired me to add Unicode support to the Baby X
> > resource compiler. But despite searching for quite a long time, I
> > can't find a test suite of Unicode files in various formats. In fact
> > it's hard to find any Unicode files at all which are not UTF-8.
> It's a fool's errand to support Unicode formats other than UTF-8.
>
dont know what fools errand mean but probbaly seems so, i dont remember waht
i was sayin back then on this but probably if utf8 become the standard in use it is
probably firtunate to stay with it, wchich is kinda fortunate (most comapatible in
ascii

btw if utf8 become de facto standard there is also defacto standard of csolo haracter?

is this just 32 bit integer which is ascii below 127 and ofical number from unicode tables
in above? i mean does things liek arrow left has one oficiall integer value?