On 1/9/22 8:19 PM, Manfred wrote:
> On 1/9/2022 11:35 PM, Öö Tiib wrote:
>> On Sunday, 9 January 2022 at 18:34:25 UTC+2, Manfred wrote:
>>> On 1/9/2022 1:44 PM, Öö Tiib wrote:
>>>> On Sunday, 9 January 2022 at 03:17:10 UTC+2, Manfred wrote:
>>>>> On 1/8/2022 11:50 PM, Öö Tiib wrote:
>>>>>> On Saturday, 8 January 2022 at 20:49:01 UTC+2,
>>>>>> james...@alumni.caltech.edu wrote:
>>>>>>> On Saturday, January 8, 2022 at 11:17:53 AM UTC-5, Öö Tiib wrote:
>>>>>>>> On Saturday, 8 January 2022 at 06:52:33 UTC+2,
>>>>>>>> james...@alumni.caltech.edu wrote:
>> ...
>>>>
>>>>>>>> FILE *f = fopen( "Foo😀Bar.txt", "w");
>>>>>>>> That should work unless underlying file system does not support
>>>>>>>> files
>>>>>>>> named "Foo😀Bar.txt" If it supports but the code does not work
>>>>>>>> then it indicates
>>>>>>>> bad standard that allows implementations to weasel away. No
>>>>>>>> garbage like
>>>>>>>> u8fopen( u8"Foo😀Bar.txt", "w") coming somewhere maybe in C35 or
>>>>>>>> so is
>>>>>>>> needed as it already works like in my example on vast majority
>>>>>>>> of things.
>>>>>>>
>>>>>>> Nothing in the standard prevents an implementation from doing
>>>>>>> that. If one doesn't
>>>>>>> already do so, that's a choice made by the implementors, and you
>>>>>>> should ask them
>>>>>>> about it. Your real beef is with the implementors, not the standard.
>>>>>>
>>>>>> My beef is with standards. Adding garbage that does not work to
>>>>>> standard is wrong
>>>>>> and not adding what everybody at least half sane does use to
>>>>>> standard is also wrong.
>>>>>>
>>>>> Also agreed, but since utf-8 is transparent to ascii functions, what
>>>>> should have been added?
>>>>
>>>> Something that makes it clear that it is defect when
>>>> "Foo😀Bar.txt" is silently opened
>>>> on file-system that fully supports files named "Foo😀Bar.txt" I
>>>> suppose.
>>>>
>>> Assuming that "Foo≡ƒÿÇBar.txt" "Foo😀Bar.txt" have the same binary
>>> representation, what's the difference? One form or the other shows up
>>> only when it is displayed in some UI - the filesystem isn't one, which
>>> leads to the implementation's runtime behavior.
>>
>> How you mean same binary representation? Both "Foo😀Bar.txt" and
>> "Foo😀Bar.txt" files can be in same directory. Both have Unicode
>> names in underlying file system precisely as posted.
>
> I mean the same byte sequence in their name, but different UI
> representation, e.g. when decoded as utf-8 or w-1252 or whatever.

But that seems to imply that the file system keeps track of file name
encoding at the entry level, which I don't know any that do that.

>
> What you are saying assumes a Unicode-aware filesystem, that's not free
> from the point of view of the standard.
> But, in order to support utf-8, it would be enough to have a char based
> filesystem that treats names as plain 0-terminated char[]. That's
> easier, probably free on most platforms, but it's different from
> Unicode-aware (which could be UTF-16 like Windows, and there you have
> your problems).
>
>>
>>> If they are actually different in their binary sequence, and this is the
>>> result of the utf-8 string being wrongly converted multiple times, this
>>> looks like a bad implementation, rather than a problem with the
>>> standard.
>>> IIUC you are advocating for some statement in the standard that prevents
>>> implementations from messing up with "character sets" in null terminated
>>> char strings?
>>
>> I mean that standard should require that all char* texts are treated as
>> UTF-8 by standard library unless said otherwise. If implementation needs
>> some other encoding of such byte sequence then it provides
>> platform-specific functions or compiler switches and/or extends language
>> with implementation-defined char_iso8859_1_t character types and
>> prefixes. If it is noteworthy handy type then add it to standards too, I
>> don't care.
>
> I see this hard to win, and probably not ideal - suppose in 10 years
> some better encoding than utf-8 shows up, then you are screwed again.
>
> I'd rather stick to the fact that utf-8 is compatible with 0-terminated
> char[], and so a plausible wish would be that such strings are not
> screwed by the implementation; for example when you store a file name in
> a filesystem with fopen() and the name is given as char[], then the
> standard could mandate that reading back that same name as char[] gives
> back the same byte sequence.
>
> Currently I guess one could use a utf-8 string as a name to fopen() on
> Windows, then the OS assumes it is W-1252 and converts it into UTF-16,
> at which point it is screwed, and when you read it back into char[] it
> is garbage.
>
>>
>> If standard can define that overflow in signed atomics is  well defined
>> and two's complement is mandated there then it also can define that all
>> char* texts are UTF-8. The only question is if what I suggest is
>> reasonable
>> or not. From viewpoint of implementer of standard library or users it
>> is likely blessing ... so I think it is question of
>> business/politics/religions.
>
> I agree with Richard here. Two's complement is not like utf-8.
> I still think it's technical rather than business/politics/religions in
> this case - as I said above I'm not sure it would even be ideal.

On Monday, 10 January 2022 at 03:20:06 UTC+2, Manfred wrote:
> On 1/9/2022 11:35 PM, Öö Tiib wrote:
> > On Sunday, 9 January 2022 at 18:34:25 UTC+2, Manfred wrote:
> >> On 1/9/2022 1:44 PM, Öö Tiib wrote:
> >>> On Sunday, 9 January 2022 at 03:17:10 UTC+2, Manfred wrote:
> >>>> On 1/8/2022 11:50 PM, Öö Tiib wrote:
> >>>>> On Saturday, 8 January 2022 at 20:49:01 UTC+2, james...@alumni.caltech.edu wrote:
> >>>>>> On Saturday, January 8, 2022 at 11:17:53 AM UTC-5, Öö Tiib wrote:
> >>>>>>> On Saturday, 8 January 2022 at 06:52:33 UTC+2, james...@alumni.caltech.edu wrote:
> > ...
> >>>
> >>>>>>> FILE *f = fopen( "Foo😀Bar.txt", "w");
> >>>>>>> That should work unless underlying file system does not support files
> >>>>>>> named "Foo😀Bar.txt" If it supports but the code does not work then it indicates
> >>>>>>> bad standard that allows implementations to weasel away. No garbage like
> >>>>>>> u8fopen( u8"Foo😀Bar.txt", "w") coming somewhere maybe in C35 or so is
> >>>>>>> needed as it already works like in my example on vast majority of things.
> >>>>>>
> >>>>>> Nothing in the standard prevents an implementation from doing that.. If one doesn't
> >>>>>> already do so, that's a choice made by the implementors, and you should ask them
> >>>>>> about it. Your real beef is with the implementors, not the standard.
> >>>>>
> >>>>> My beef is with standards. Adding garbage that does not work to standard is wrong
> >>>>> and not adding what everybody at least half sane does use to standard is also wrong.
> >>>>>
> >>>> Also agreed, but since utf-8 is transparent to ascii functions, what
> >>>> should have been added?
> >>>
> >>> Something that makes it clear that it is defect when "Foo😀Bar.txt" is silently opened
> >>> on file-system that fully supports files named "Foo😀Bar.txt" I suppose.
> >>>
> >> Assuming that "Foo≡ƒÿÇBar.txt" "Foo😀Bar.txt" have the same binary
> >> representation, what's the difference? One form or the other shows up
> >> only when it is displayed in some UI - the filesystem isn't one, which
> >> leads to the implementation's runtime behavior.
> >
> > How you mean same binary representation? Both "Foo😀Bar.txt" and
> > "Foo😀Bar.txt" files can be in same directory. Both have Unicode
> > names in underlying file system precisely as posted.
> I mean the same byte sequence in their name, but different UI
> representation, e.g. when decoded as utf-8 or w-1252 or whatever.

Nope. NTFS for example has file names as UTF-16 plus Windows uses hard links
to also give legacy Radix-50 style short (8.3) filenames to all files. Trivia
question: Why it is named "Radix-50" when there are only 40 characters in it?

> What you are saying assumes a Unicode-aware filesystem, that's not free
> from the point of view of the standard.
> But, in order to support utf-8, it would be enough to have a char based
> filesystem that treats names as plain 0-terminated char[]. That's
> easier, probably free on most platforms, but it's different from
> Unicode-aware (which could be UTF-16 like Windows, and there you have
> your problems).

Problems were with Japanese Shift JIS or EUC encodings in file
systems... it was expensive to guess what it is and so they switched
to Unicode. With UTF-16 one needs to know or to detect endianess
of it ... otherwise turning to UTF-8 and back is absurdly trivial.
Certainly less code than between Windows-1252 and UTF-16.

> >
> >> If they are actually different in their binary sequence, and this is the
> >> result of the utf-8 string being wrongly converted multiple times, this
> >> looks like a bad implementation, rather than a problem with the standard.
> >> IIUC you are advocating for some statement in the standard that prevents
> >> implementations from messing up with "character sets" in null terminated
> >> char strings?
> >
> > I mean that standard should require that all char* texts are treated as
> > UTF-8 by standard library unless said otherwise. If implementation needs
> > some other encoding of such byte sequence then it provides
> > platform-specific functions or compiler switches and/or extends language
> > with implementation-defined char_iso8859_1_t character types and
> > prefixes. If it is noteworthy handy type then add it to standards too, I
> > don't care.
>
> I see this hard to win, and probably not ideal - suppose in 10 years
> some better encoding than utf-8 shows up, then you are screwed again.
>
> I'd rather stick to the fact that utf-8 is compatible with 0-terminated
> char[], and so a plausible wish would be that such strings are not
> screwed by the implementation; for example when you store a file name in
> a filesystem with fopen() and the name is given as char[], then the
> standard could mandate that reading back that same name as char[] gives
> back the same byte sequence.
>
> Currently I guess one could use a utf-8 string as a name to fopen() on
> Windows, then the OS assumes it is W-1252 and converts it into UTF-16,
> at which point it is screwed, and when you read it back into char[] it
> is garbage.

Yes, something like that happens. Microsoft was amazingly innovative
and wanted to push all kinds of good things up to 1995. But then some
kind of browser and compiler and other incompatibility wars within its
own operating system started ... and its positions started to shrink and
get damage. But it is their own business so they have full right to burn
it however they please.

> >
> > If standard can define that overflow in signed atomics is well defined
> > and two's complement is mandated there then it also can define that all
> > char* texts are UTF-8. The only question is if what I suggest is reasonable
> > or not. From viewpoint of implementer of standard library or users it
> > is likely blessing ... so I think it is question of business/politics/religions.
>
> I agree with Richard here. Two's complement is not like utf-8.
> I still think it's technical rather than business/politics/religions in
> this case - as I said above I'm not sure it would even be ideal.

Mateusz Viste <mateusz@xyz.invalid> writes:

> While UTF-8 is neat, it is also complex to decode. Even a simple
> strlen() can be challenging.

Hmmm... Since UTF-8 is a multi-byte encoding of Unicode code points, I
would interpret a "UTF-8 strlen" as being a function the counts the
number of encoded code points, and that's simple enough. Every byte,
before the null, that does not have 10xxxx it it's top two bits is the
start of a code point:

size_t ustrlen(char *s)
{
size_t len = 0;
while (*s) len += (*s++ & 0xc0) != 0x80;
return len;
}

Obviously, for some uses, this is too simple as it does not detect
incorrect encodings.

--
Ben.

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:

> On Sunday, 9 January 2022 at 17:52:50 UTC, Richard Damon wrote:
>>
>> On the other hand, some languages add things like 'vowel points' to
>> characters, and those are seperate graphemes even though they are added
>> by a similar manner. This comes down to what the original language
>> though of as a 'character', which just makes things even more complicated.
>>
> In Hebrew the "vowel points" are optional. They are used in beginners' and
> religious texts, but not in general use. So if we take a text from scripture,
> and represent it with and without vowels, is that the same text or a different
> text? Almost all Hebrew speakers would say "It's the same text". So
> strcmp() doesn't necessarily work in a Hebrew context.

strcmp fails must closer to home (at least closer to my geographic home)
because, in Spanish, ch and ll are, transitionally, considered separate
letters. All c* words collate before any ch* words, and all l* words
before and ll* ones.

This has proved so inconvenient that I believe that the Real Academia
Española has ruled that, now, only ñ must be considered separately.

--
Ben.

On Monday, 10 January 2022 at 03:59:00 UTC+2, Richard Damon wrote:
> On 1/9/22 7:51 PM, Öö Tiib wrote:
> > On Monday, 10 January 2022 at 01:49:07 UTC+2, Richard Damon wrote:
> >> On 1/9/22 5:35 PM, Öö Tiib wrote:
> >>> On Sunday, 9 January 2022 at 18:34:25 UTC+2, Manfred wrote:
> >>>> On 1/9/2022 1:44 PM, Öö Tiib wrote:
> >>>>> On Sunday, 9 January 2022 at 03:17:10 UTC+2, Manfred wrote:
> >>>>>> On 1/8/2022 11:50 PM, Öö Tiib wrote:
> >>>>>>> On Saturday, 8 January 2022 at 20:49:01 UTC+2, james...@alumni.caltech.edu wrote:
> >>>>>>>> On Saturday, January 8, 2022 at 11:17:53 AM UTC-5, Öö Tiib wrote:
> >>>>>>>>> On Saturday, 8 January 2022 at 06:52:33 UTC+2, james...@alumni.caltech.edu wrote:
> >>> ...
> >>>>>
> >>>>>>>>> FILE *f = fopen( "Foo😀Bar.txt", "w");
> >>>>>>>>> That should work unless underlying file system does not support files
> >>>>>>>>> named "Foo😀Bar.txt" If it supports but the code does not work then it indicates
> >>>>>>>>> bad standard that allows implementations to weasel away. No garbage like
> >>>>>>>>> u8fopen( u8"Foo😀Bar.txt", "w") coming somewhere maybe in C35 or so is
> >>>>>>>>> needed as it already works like in my example on vast majority of things.
> >>>>>>>>
> >>>>>>>> Nothing in the standard prevents an implementation from doing that. If one doesn't
> >>>>>>>> already do so, that's a choice made by the implementors, and you should ask them
> >>>>>>>> about it. Your real beef is with the implementors, not the standard.
> >>>>>>>
> >>>>>>> My beef is with standards. Adding garbage that does not work to standard is wrong
> >>>>>>> and not adding what everybody at least half sane does use to standard is also wrong.
> >>>>>>>
> >>>>>> Also agreed, but since utf-8 is transparent to ascii functions, what
> >>>>>> should have been added?
> >>>>>
> >>>>> Something that makes it clear that it is defect when "Foo😀Bar.txt" is silently opened
> >>>>> on file-system that fully supports files named "Foo😀Bar.txt" I suppose.
> >>>>>
> >>>> Assuming that "Foo≡ƒÿÇBar.txt" "Foo😀Bar.txt" have the same binary
> >>>> representation, what's the difference? One form or the other shows up
> >>>> only when it is displayed in some UI - the filesystem isn't one, which
> >>>> leads to the implementation's runtime behavior.
> >>>
> >>> How you mean same binary representation? Both "Foo😀Bar.txt" and
> >>> "Foo😀Bar.txt" files can be in same directory. Both have Unicode
> >>> names in underlying file system precisely as posted.
> >>>
> >>>> If they are actually different in their binary sequence, and this is the
> >>>> result of the utf-8 string being wrongly converted multiple times, this
> >>>> looks like a bad implementation, rather than a problem with the standard.
> >>>> IIUC you are advocating for some statement in the standard that prevents
> >>>> implementations from messing up with "character sets" in null terminated
> >>>> char strings?
> >>>
> >>> I mean that standard should require that all char* texts are treated as
> >>> UTF-8 by standard library unless said otherwise. If implementation needs
> >>> some other encoding of such byte sequence then it provides
> >>> platform-specific functions or compiler switches and/or extends language
> >>> with implementation-defined char_iso8859_1_t character types and
> >>> prefixes. If it is noteworthy handy type then add it to standards too, I
> >>> don't care.
> >>>
> >>> If standard can define that overflow in signed atomics is well defined
> >>> and two's complement is mandated there then it also can define that all
> >>> char* texts are UTF-8. The only question is if what I suggest is reasonable
> >>> or not. From viewpoint of implementer of standard library or users it
> >>> is likely blessing ... so I think it is question of business/politics/religions.
> >>
> >> The difference is that in these days, the existence of computers that
> >> aren't going to be able to support two's complement that will still want
> >> to support modern 'C' is effectively non-existent.
> >>
> >> The existance of machines that might still want to be able to support
> >> non-UTF-8 strings is not.
> >
> > But do there exist machines that do want to support texts as char* but
> > do not want to support UTF-8? Describe those machines, give examples.
> Small embedded micros with no need for large character sets.

You diligently avoid giving examples?
You mean if it displays only Arabic numbers then it needs only 10
characters, if minuses and dots too then 12. ASCII and UTF-8 are identical
in that processing. So UTF-8 adds no extra bytes to such system.

> >
> >> Perhaps the biggest is the embedded market where needing to support
> >
> > You mean tiny things like SD cards or flash sticks? I can store there
> > "Foo😀Bar.txt" just fine. Either embedded system does not need to
> > communicate in char* text at all, can fully ignore its encoding or has
> > to deal with UTF-8 anyway. I know of no other examples, despite I've
> > participated in programming whole pile of embedded systems over
> > the decades.
>
> Many such system communicate in command strings, maybe even with a
> minimal TCP/IP but have no need for processing data beyond pure ASCII.

Same as with numbers, if no need to show the degree in 74.3°F so no
need for to process anything beyond pure ASCII. Otherwise the software
needs to detect that there are bytes C2 B0 for to show ° also no biggie.

> >> beyond plain ASCII isn't needed, and DEFINING that strings will follow
> >> UTF-8 rules adds a LOT of complications for some operations that just
> >> aren't needed on many of the systems.
> >
> > WHAT complications? Give examples? Both ASCII and UTF-8 are row of
> > bytes that end with zero. ASCII is proper subset of UTF-8. Tell about
> > use-case where UTF-8 hurts? Human languages and typography
> > are horribly complicated but UTF-8 is genially trivial. Either embedded
> > system does not do linguistic analyses of poems or if it does then
> > it needs to use Unicode anyway. But commonly if it can't display
> > something then it shows � and done.
>
> Once you have your char as being defined as a Multi-Byte Character Set,
> then wchar_t must be big enough to hold any of them. If you just support
> ASCII, then wchar_t can be just 8 Bits if you want, and (almost?) all
> the wchar_t stuff can just be alias for the char stuff.
>
> Thus forcing char to be UTF-8 adds a lot of complexity to the system.

Most embedded systems that I programmed used wchar_t for nothing.
So the compiler generated precisely 0 bytes of wchar_t processing
into image that was flashed into those.

> >
> >> The Standard does ALLOW a system to define char to be UTF-8 (at least
> >> until you get into issues of what it requires for wide characters).
> >
> > Allowing is apparently not enough as the support rots in standards.
> > Wide characters are wchar_t, char16_t and char32_t. These are
> > in horrible state too but I ignore it for now. Not related to issues
> > with char* and far less important in industry.
> >
> But that is part of the problem with supporting UTF-8, as that by
> definiition brings in all the wide character issues into play.
>
> If you define that your character set is ASCII, then wchar_t becomes
> trivial.
>
> A big part of the issue with char16_t is that it is fundamentally broken
> with Unicode, but lives on due to trying to maintain the backwards
> bandaids that basically can't be removed without admitting that a large
> segment of code just will live as being openly non-complient.
>
> Too much legacy code assumes that 16 bit characters are 'big enough' for
> most people, and pretty much do work if you aren't being a stickler for
> full conformance to the rules, which no one is because you can't be.

On 1/9/22 9:18 PM, Ben Bacarisse wrote:
> Mateusz Viste <mateusz@xyz.invalid> writes:
>
>> While UTF-8 is neat, it is also complex to decode. Even a simple
>> strlen() can be challenging.
>
> Hmmm... Since UTF-8 is a multi-byte encoding of Unicode code points, I
> would interpret a "UTF-8 strlen" as being a function the counts the
> number of encoded code points, and that's simple enough. Every byte,
> before the null, that does not have 10xxxx it it's top two bits is the
> start of a code point:
>
> size_t ustrlen(char *s)
> {
> size_t len = 0;
> while (*s) len += (*s++ & 0xc0) != 0x80;
> return len;
> }
>
> Obviously, for some uses, this is too simple as it does not detect
> incorrect encodings.
>

My understanding is that for MBCS the function strlen returns the number
of BYTES in the string, not the number of Multi-Byte Characters in the
string.

This means that strlen can be used to determine how much space is needs
to store the string.

On 1/9/22 5:41 AM, Po Lu wrote:
> "james...@alumni.caltech.edu" <jameskuyper@alumni.caltech.edu> writes:
>
>> No, I am quite accurately and honestly expressing my confusion. You
>> object to something being prohibited by the standards that is, to the
>> best of my understanding, allowed. It would make more sense if you
>> were objecting the fact that it isn't mandatory, and if you were
>> making such claims, I would disagree with you about whether it would
>> be a good idea to make it mandatory - but as far as I can tell, you're
>> claiming it isn't allowed.
>
> AFAICT, he's complaining about Microsoft's specific implementations of
> some standards.

He's repeatedly asserted that it's the standards themselves that he's
complaining about. His actual complaints, however, seem to be about
Microsoft-specific behavior. That's not a contradiction - he's
complaining about the fact that the standards allow that behavior.

On 1/9/22 9:40 PM, Öö Tiib wrote:
> On Monday, 10 January 2022 at 03:59:00 UTC+2, Richard Damon wrote:
>> On 1/9/22 7:51 PM, Öö Tiib wrote:
>>> On Monday, 10 January 2022 at 01:49:07 UTC+2, Richard Damon wrote:
>>>> On 1/9/22 5:35 PM, Öö Tiib wrote:
>>>>> On Sunday, 9 January 2022 at 18:34:25 UTC+2, Manfred wrote:
>>>>>> On 1/9/2022 1:44 PM, Öö Tiib wrote:
>>>>>>> On Sunday, 9 January 2022 at 03:17:10 UTC+2, Manfred wrote:
>>>>>>>> On 1/8/2022 11:50 PM, Öö Tiib wrote:
>>>>>>>>> On Saturday, 8 January 2022 at 20:49:01 UTC+2, james...@alumni.caltech.edu wrote:
>>>>>>>>>> On Saturday, January 8, 2022 at 11:17:53 AM UTC-5, Öö Tiib wrote:
>>>>>>>>>>> On Saturday, 8 January 2022 at 06:52:33 UTC+2, james...@alumni.caltech.edu wrote:
>>>>> ...
>>>>>>>
>>>>>>>>>>> FILE *f = fopen( "Foo😀Bar.txt", "w");
>>>>>>>>>>> That should work unless underlying file system does not support files
>>>>>>>>>>> named "Foo😀Bar.txt" If it supports but the code does not work then it indicates
>>>>>>>>>>> bad standard that allows implementations to weasel away. No garbage like
>>>>>>>>>>> u8fopen( u8"Foo😀Bar.txt", "w") coming somewhere maybe in C35 or so is
>>>>>>>>>>> needed as it already works like in my example on vast majority of things.
>>>>>>>>>>
>>>>>>>>>> Nothing in the standard prevents an implementation from doing that. If one doesn't
>>>>>>>>>> already do so, that's a choice made by the implementors, and you should ask them
>>>>>>>>>> about it. Your real beef is with the implementors, not the standard.
>>>>>>>>>
>>>>>>>>> My beef is with standards. Adding garbage that does not work to standard is wrong
>>>>>>>>> and not adding what everybody at least half sane does use to standard is also wrong.
>>>>>>>>>
>>>>>>>> Also agreed, but since utf-8 is transparent to ascii functions, what
>>>>>>>> should have been added?
>>>>>>>
>>>>>>> Something that makes it clear that it is defect when "Foo😀Bar.txt" is silently opened
>>>>>>> on file-system that fully supports files named "Foo😀Bar.txt" I suppose.
>>>>>>>
>>>>>> Assuming that "Foo≡ƒÿÇBar.txt" "Foo😀Bar.txt" have the same binary
>>>>>> representation, what's the difference? One form or the other shows up
>>>>>> only when it is displayed in some UI - the filesystem isn't one, which
>>>>>> leads to the implementation's runtime behavior.
>>>>>
>>>>> How you mean same binary representation? Both "Foo😀Bar.txt" and
>>>>> "Foo😀Bar.txt" files can be in same directory. Both have Unicode
>>>>> names in underlying file system precisely as posted.
>>>>>
>>>>>> If they are actually different in their binary sequence, and this is the
>>>>>> result of the utf-8 string being wrongly converted multiple times, this
>>>>>> looks like a bad implementation, rather than a problem with the standard.
>>>>>> IIUC you are advocating for some statement in the standard that prevents
>>>>>> implementations from messing up with "character sets" in null terminated
>>>>>> char strings?
>>>>>
>>>>> I mean that standard should require that all char* texts are treated as
>>>>> UTF-8 by standard library unless said otherwise. If implementation needs
>>>>> some other encoding of such byte sequence then it provides
>>>>> platform-specific functions or compiler switches and/or extends language
>>>>> with implementation-defined char_iso8859_1_t character types and
>>>>> prefixes. If it is noteworthy handy type then add it to standards too, I
>>>>> don't care.
>>>>>
>>>>> If standard can define that overflow in signed atomics is well defined
>>>>> and two's complement is mandated there then it also can define that all
>>>>> char* texts are UTF-8. The only question is if what I suggest is reasonable
>>>>> or not. From viewpoint of implementer of standard library or users it
>>>>> is likely blessing ... so I think it is question of business/politics/religions.
>>>>
>>>> The difference is that in these days, the existence of computers that
>>>> aren't going to be able to support two's complement that will still want
>>>> to support modern 'C' is effectively non-existent.
>>>>
>>>> The existance of machines that might still want to be able to support
>>>> non-UTF-8 strings is not.
>>>
>>> But do there exist machines that do want to support texts as char* but
>>> do not want to support UTF-8? Describe those machines, give examples.
>> Small embedded micros with no need for large character sets.
>
> You diligently avoid giving examples?
> You mean if it displays only Arabic numbers then it needs only 10
> characters, if minuses and dots too then 12. ASCII and UTF-8 are identical
> in that processing. So UTF-8 adds no extra bytes to such system.

The problem is that once you define that your 'Character Set' is UTF-8,
and thus wide characters are wider than 8 bits, then a number of
mechanizism need to be provided by the library, and it can be hard to
keep that down to zero cost if not used.

A big issue is locales. An ASCII only system can easily just define very
crude locale support that is very cheap. Once you introduce UTF-8, it
becomes a very slippery slope that can make it hard to keep the size of
the library code brought in under control.

Remember, even simple things like printf pulls in some locale code, even
if you don't actaully ever set a locale.

>
>>>
>>>> Perhaps the biggest is the embedded market where needing to support
>>>
>>> You mean tiny things like SD cards or flash sticks? I can store there
>>> "Foo😀Bar.txt" just fine. Either embedded system does not need to
>>> communicate in char* text at all, can fully ignore its encoding or has
>>> to deal with UTF-8 anyway. I know of no other examples, despite I've
>>> participated in programming whole pile of embedded systems over
>>> the decades.
>>
>> Many such system communicate in command strings, maybe even with a
>> minimal TCP/IP but have no need for processing data beyond pure ASCII.
>
> Same as with numbers, if no need to show the degree in 74.3°F so no
> need for to process anything beyond pure ASCII. Otherwise the software
> needs to detect that there are bytes C2 B0 for to show ° also no biggie.

Again, the problem is that once you have defined that Multi-byte
characters exist, things like printf will use locale support that might
pull in classifaction routines that might needs to classify what
characters are 'digits' or 'letters' in the full Unicode range.

For a PC, with a large OS, that support is fairly cheap, and might even
be just built in, but in a small embedded system that can be costly.

I HAVE had systems that defined that characters were UTF-8 and the
result was I couldn't use a lot of the library because it pulled in too
much locale code to fit into my machine.

>
>>>> beyond plain ASCII isn't needed, and DEFINING that strings will follow
>>>> UTF-8 rules adds a LOT of complications for some operations that just
>>>> aren't needed on many of the systems.
>>>
>>> WHAT complications? Give examples? Both ASCII and UTF-8 are row of
>>> bytes that end with zero. ASCII is proper subset of UTF-8. Tell about
>>> use-case where UTF-8 hurts? Human languages and typography
>>> are horribly complicated but UTF-8 is genially trivial. Either embedded
>>> system does not do linguistic analyses of poems or if it does then
>>> it needs to use Unicode anyway. But commonly if it can't display
>>> something then it shows � and done.
>>
>> Once you have your char as being defined as a Multi-Byte Character Set,
>> then wchar_t must be big enough to hold any of them. If you just support
>> ASCII, then wchar_t can be just 8 Bits if you want, and (almost?) all
>> the wchar_t stuff can just be alias for the char stuff.
>>
>> Thus forcing char to be UTF-8 adds a lot of complexity to the system.
>
> Most embedded systems that I programmed used wchar_t for nothing.
> So the compiler generated precisely 0 bytes of wchar_t processing
> into image that was flashed into those.
>

On Monday, 10 January 2022 at 05:35:08 UTC+2, Richard Damon wrote:
> On 1/9/22 9:40 PM, Öö Tiib wrote:
> > On Monday, 10 January 2022 at 03:59:00 UTC+2, Richard Damon wrote:
> >> On 1/9/22 7:51 PM, Öö Tiib wrote:
> >>> On Monday, 10 January 2022 at 01:49:07 UTC+2, Richard Damon wrote:
> >>>> On 1/9/22 5:35 PM, Öö Tiib wrote:
> >>>>> On Sunday, 9 January 2022 at 18:34:25 UTC+2, Manfred wrote:
> >>>>>> On 1/9/2022 1:44 PM, Öö Tiib wrote:
> >>>>>>> On Sunday, 9 January 2022 at 03:17:10 UTC+2, Manfred wrote:
> >>>>>>>> On 1/8/2022 11:50 PM, Öö Tiib wrote:
> >>>>>>>>> On Saturday, 8 January 2022 at 20:49:01 UTC+2, james...@alumni.caltech.edu wrote:
> >>>>>>>>>> On Saturday, January 8, 2022 at 11:17:53 AM UTC-5, Öö Tiib wrote:
> >>>>>>>>>>> On Saturday, 8 January 2022 at 06:52:33 UTC+2, james...@alumni.caltech.edu wrote:
> >>>>> ...
> >>>>>>>
> >>>>>>>>>>> FILE *f = fopen( "Foo😀Bar.txt", "w");
> >>>>>>>>>>> That should work unless underlying file system does not support files
> >>>>>>>>>>> named "Foo😀Bar.txt" If it supports but the code does not work then it indicates
> >>>>>>>>>>> bad standard that allows implementations to weasel away. No garbage like
> >>>>>>>>>>> u8fopen( u8"Foo😀Bar.txt", "w") coming somewhere maybe in C35 or so is
> >>>>>>>>>>> needed as it already works like in my example on vast majority of things.
> >>>>>>>>>>
> >>>>>>>>>> Nothing in the standard prevents an implementation from doing that. If one doesn't
> >>>>>>>>>> already do so, that's a choice made by the implementors, and you should ask them
> >>>>>>>>>> about it. Your real beef is with the implementors, not the standard.
> >>>>>>>>>
> >>>>>>>>> My beef is with standards. Adding garbage that does not work to standard is wrong
> >>>>>>>>> and not adding what everybody at least half sane does use to standard is also wrong.
> >>>>>>>>>
> >>>>>>>> Also agreed, but since utf-8 is transparent to ascii functions, what
> >>>>>>>> should have been added?
> >>>>>>>
> >>>>>>> Something that makes it clear that it is defect when "Foo😀Bar.txt" is silently opened
> >>>>>>> on file-system that fully supports files named "Foo😀Bar.txt" I suppose.
> >>>>>>>
> >>>>>> Assuming that "Foo≡ƒÿÇBar.txt" "Foo😀Bar.txt" have the same binary
> >>>>>> representation, what's the difference? One form or the other shows up
> >>>>>> only when it is displayed in some UI - the filesystem isn't one, which
> >>>>>> leads to the implementation's runtime behavior.
> >>>>>
> >>>>> How you mean same binary representation? Both "Foo😀Bar.txt" and
> >>>>> "Foo😀Bar.txt" files can be in same directory. Both have Unicode
> >>>>> names in underlying file system precisely as posted.
> >>>>>
> >>>>>> If they are actually different in their binary sequence, and this is the
> >>>>>> result of the utf-8 string being wrongly converted multiple times, this
> >>>>>> looks like a bad implementation, rather than a problem with the standard.
> >>>>>> IIUC you are advocating for some statement in the standard that prevents
> >>>>>> implementations from messing up with "character sets" in null terminated
> >>>>>> char strings?
> >>>>>
> >>>>> I mean that standard should require that all char* texts are treated as
> >>>>> UTF-8 by standard library unless said otherwise. If implementation needs
> >>>>> some other encoding of such byte sequence then it provides
> >>>>> platform-specific functions or compiler switches and/or extends language
> >>>>> with implementation-defined char_iso8859_1_t character types and
> >>>>> prefixes. If it is noteworthy handy type then add it to standards too, I
> >>>>> don't care.
> >>>>>
> >>>>> If standard can define that overflow in signed atomics is well defined
> >>>>> and two's complement is mandated there then it also can define that all
> >>>>> char* texts are UTF-8. The only question is if what I suggest is reasonable
> >>>>> or not. From viewpoint of implementer of standard library or users it
> >>>>> is likely blessing ... so I think it is question of business/politics/religions.
> >>>>
> >>>> The difference is that in these days, the existence of computers that
> >>>> aren't going to be able to support two's complement that will still want
> >>>> to support modern 'C' is effectively non-existent.
> >>>>
> >>>> The existance of machines that might still want to be able to support
> >>>> non-UTF-8 strings is not.
> >>>
> >>> But do there exist machines that do want to support texts as char* but
> >>> do not want to support UTF-8? Describe those machines, give examples.
> >> Small embedded micros with no need for large character sets.
> >
> > You diligently avoid giving examples?
> > You mean if it displays only Arabic numbers then it needs only 10
> > characters, if minuses and dots too then 12. ASCII and UTF-8 are identical
> > in that processing. So UTF-8 adds no extra bytes to such system.
>
> The problem is that once you define that your 'Character Set' is UTF-8,
> and thus wide characters are wider than 8 bits, then a number of
> mechanizism need to be provided by the library, and it can be hard to
> keep that down to zero cost if not used.

You must be specific.
> A big issue is locales. An ASCII only system can easily just define very
> crude locale support that is very cheap. Once you introduce UTF-8, it
> becomes a very slippery slope that can make it hard to keep the size of
> the library code brought in under control.
>
> Remember, even simple things like printf pulls in some locale code, even
> if you don't actaully ever set a locale.

No, its different topic. Fully implementation defined. Conformant
implementation may implement only locale named "C" and be done with it.
The setlocale() localeconv() and lconv() can be trivial stubs behaving by
letter of standard and not worth calling ever. Would be nice from
implementer to support localization but not really required and not
something I complain about.

> >
> >>>
> >>>> Perhaps the biggest is the embedded market where needing to support
> >>>
> >>> You mean tiny things like SD cards or flash sticks? I can store there
> >>> "Foo😀Bar.txt" just fine. Either embedded system does not need to
> >>> communicate in char* text at all, can fully ignore its encoding or has
> >>> to deal with UTF-8 anyway. I know of no other examples, despite I've
> >>> participated in programming whole pile of embedded systems over
> >>> the decades.
> >>
> >> Many such system communicate in command strings, maybe even with a
> >> minimal TCP/IP but have no need for processing data beyond pure ASCII.
> >
> > Same as with numbers, if no need to show the degree in 74.3°F so no
> > need for to process anything beyond pure ASCII. Otherwise the software
> > needs to detect that there are bytes C2 B0 for to show ° also no biggie.
>
> Again, the problem is that once you have defined that Multi-byte
> characters exist, things like printf will use locale support that might
> pull in classifaction routines that might needs to classify what
> characters are 'digits' or 'letters' in the full Unicode range.

Stay with UTF-8? It can keep locale "en_US". It has to show □ for each
missing symbol in font and � for illegal UTF-8 byte sequence (that is
trivial to detect). There are likely 0 fonts in existence with all Unicode
symbols so a font with 20 symbols is fully conformant for embedded
device that does not need to analyze Hebrew manuscripts but to
show temperature for desperate housewife.

On Sunday, January 9, 2022 at 5:35:42 PM UTC-5, Öö Tiib wrote:
> On Sunday, 9 January 2022 at 18:34:25 UTC+2, Manfred wrote:
> > On 1/9/2022 1:44 PM, Öö Tiib wrote:
....
> > > Something that makes it clear that it is defect when "Foo😀Bar.txt" is silently opened
> > > on file-system that fully supports files named "Foo😀Bar.txt" I suppose.
> > >
> > Assuming that "Foo≡ƒÿÇBar.txt" "Foo😀Bar.txt" have the same binary
> > representation, what's the difference? One form or the other shows up
> > only when it is displayed in some UI - the filesystem isn't one, which
> > leads to the implementation's runtime behavior.
> How you mean same binary representation? Both "Foo😀Bar.txt" and
> "Foo😀Bar.txt" files can be in same directory. Both have Unicode
> names in underlying file system precisely as posted.

Have you checked to make sure? Any system where passing the UTF-8 string
"Foo😀Bar.txt" to fopen() opens a file whose name displays as Foo≡ƒÿÇBar.txt
is likely to be a system where the file names are displayed using some single-
byte encoding. The UTF-8 encoding of "Foo😀Bar.txt" is

0X46 0X6F 0X6F 0XF0 0X9F 0X98 0X80 0X42 0X61 0X72 0X2E 0X74 0X78 0X74

After quite a bit of searching, I found Code page 865 (MS-DOS Nordic), which
has 0XF0 = '≡', 0x9F = 'ƒ' and 0X98 = 'ÿ'. If the utilities that you use to display file
names used that encoding to interpret the file name, that would explain your results.

On Monday, 10 January 2022 at 07:24:01 UTC+2, james...@alumni.caltech.edu wrote:
> On Sunday, January 9, 2022 at 5:35:42 PM UTC-5, Öö Tiib wrote:
> > On Sunday, 9 January 2022 at 18:34:25 UTC+2, Manfred wrote:
> > > On 1/9/2022 1:44 PM, Öö Tiib wrote:
> ...
> > > > Something that makes it clear that it is defect when "Foo😀Bar.txt" is silently opened
> > > > on file-system that fully supports files named "Foo😀Bar.txt" I suppose.
> > > >
> > > Assuming that "Foo≡ƒÿÇBar.txt" "Foo😀Bar.txt" have the same binary
> > > representation, what's the difference? One form or the other shows up
> > > only when it is displayed in some UI - the filesystem isn't one, which
> > > leads to the implementation's runtime behavior.
> > How you mean same binary representation? Both "Foo😀Bar.txt" and
> > "Foo😀Bar.txt" files can be in same directory. Both have Unicode
> > names in underlying file system precisely as posted.
> Have you checked to make sure? Any system where passing the UTF-8 string
> "Foo😀Bar.txt" to fopen() opens a file whose name displays as Foo≡ƒÿÇBar.txt
> is likely to be a system where the file names are displayed using some single-
> byte encoding. The UTF-8 encoding of "Foo😀Bar.txt" is
>
> 0X46 0X6F 0X6F 0XF0 0X9F 0X98 0X80 0X42 0X61 0X72 0X2E 0X74 0X78 0X74
>
> After quite a bit of searching, I found Code page 865 (MS-DOS Nordic), which
> has 0XF0 = '≡', 0x9F = 'ƒ' and 0X98 = 'ÿ'. If the utilities that you use to display file
> names used that encoding to interpret the file name, that would explain your results.

Need a screenshot with files named "Foo😀Bar.txt" and "Foo≡ƒÿÇBar.txt"
side-by-side? Just that for "Foo😀Bar.txt" one needs to use non-standard
_wfopen( L"Foo😀Bar.txt", L"w"). That is default both with MSVC and MinGW
gcc.

People have apparently lamented about it so the behavior of fopen can be
repaired with some butt-ugly xml file linked in the program in specific way
or by providing one next to your program with yourprogramname.xml as
name. That trick works from Windows 10 May 2019 Update.

But reading UTF-8 (for example password) from console is still impossible.
One has to write platform-specific code about like that:

SIZE_T wbuf_len = (len - 1 + 2)*sizeof(*wbuf);
WCHAR *wbuf = HeapAlloc(GetProcessHeap(), 0, wbuf_len);
DWORD nread;
ReadConsoleW(hi, wbuf, len - 1 + 2, &nread, 0);
wbuf[nread-2] = 0; // truncate "\r\n"
int r = WideCharToMultiByte(CP_UTF8, 0, wbuf, -1, buf, len, 0, 0);
SecureZeroMemory(wbuf, wbuf_len);
HeapFree(GetProcessHeap(), 0, wbuf);
That way we have UTF-8 read into buf.

On 10/01/2022 03:33, Ben Bacarisse wrote:
> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>
>> On Sunday, 9 January 2022 at 17:52:50 UTC, Richard Damon wrote:
>>>
>>> On the other hand, some languages add things like 'vowel points' to
>>> characters, and those are seperate graphemes even though they are added
>>> by a similar manner. This comes down to what the original language
>>> though of as a 'character', which just makes things even more complicated.
>>>
>> In Hebrew the "vowel points" are optional. They are used in beginners' and
>> religious texts, but not in general use. So if we take a text from scripture,
>> and represent it with and without vowels, is that the same text or a different
>> text? Almost all Hebrew speakers would say "It's the same text". So
>> strcmp() doesn't necessarily work in a Hebrew context.
>
> strcmp fails must closer to home (at least closer to my geographic home)
> because, in Spanish, ch and ll are, transitionally, considered separate
> letters. All c* words collate before any ch* words, and all l* words
> before and ll* ones.
>
> This has proved so inconvenient that I believe that the Real Academia
> Española has ruled that, now, only ñ must be considered separately.
>

In a more extreme case, in Norwegian "aa" is sometimes sorted very early
alphabetically, and sometimes very late as it is a transliteration of
the Norwegian letter "å", which is the last letter in our alphabet.

Sorting for human use (as distinct from, say, making a binary tree for
lookups, in which case pure data-based sorting is fine) is complicated
business!

2022-01-10 at 02:18 +0000, Ben Bacarisse wrote:
> Hmmm... Since UTF-8 is a multi-byte encoding of Unicode code points,
> I would interpret a "UTF-8 strlen" as being a function the counts the
> number of encoded code points, and that's simple enough.
> (...)
> Obviously, for some uses, this is too simple as it does not detect
> incorrect encodings.

I'm not sure what your point was supposed to be. "It's simple to write
non-practical, prototype-grade code"? Yes, it is.

Now, I am not saying that writing a utf-8 strlen() is incredibly
difficult of course. I am only saying it is an extra layer of
complexity compared to UCS-2 or UTF-32. And that is why I understand
why people often choose to internally store strings in one of these
encodings instead of utf-8 (esp. if dealing with fixed-width character
outputs). It's simply easier to deal with an array of values that maps
directly to codepoints rather than parse a utf-8 string taking care not
to explode on encoding errors or edge cases.

Mateusz

On Monday, 10 January 2022 at 05:26:33 UTC+2, james...@alumni.caltech.edu wrote:
> On 1/9/22 5:41 AM, Po Lu wrote:
> > "james...@alumni.caltech.edu" <james...@alumni.caltech.edu> writes:
> >
> >> No, I am quite accurately and honestly expressing my confusion. You
> >> object to something being prohibited by the standards that is, to the
> >> best of my understanding, allowed. It would make more sense if you
> >> were objecting the fact that it isn't mandatory, and if you were
> >> making such claims, I would disagree with you about whether it would
> >> be a good idea to make it mandatory - but as far as I can tell, you're
> >> claiming it isn't allowed.
> >
> > AFAICT, he's complaining about Microsoft's specific implementations of
> > some standards.
> He's repeatedly asserted that it's the standards themselves that he's
> complaining about. His actual complaints, however, seem to be about
> Microsoft-specific behavior. That's not a contradiction - he's
> complaining about the fact that the standards allow that behavior.

My complaints are because of the events somehow falling
together.

On one hand the issues with electronic components caused
shortage of prototype devices to run tests on. So some
cooperation partners decided to run unit tests on Windows
boxes. Since on Windows the C standard library is crappy these
unit tests now mostly test ad-hoc hacks of simulating proper
standard library and pointless man-months wasted into those.

On the other hand C++20 broke that u8 prefix indicating
dedication to push UTF-8 into that char8_t* garbage that no
standard library function is using. The char8_t would also
break constexpr processing of it as casts to char are illegal in
constexpr context.

It smells like some next family of non-standard functions soon
in style of _u8fopen(u8"Foo😀Bar.txt", u8"w") . Story of annex
K repeated. Oh I hope being paranoid and UCRT with Visual
Studio 2022 coming plain great, but what is the chance of
that?

Mateusz Viste <mateusz@xyz.invalid> writes:

> 2022-01-10 at 02:18 +0000, Ben Bacarisse wrote:
>> Hmmm... Since UTF-8 is a multi-byte encoding of Unicode code points,
>> I would interpret a "UTF-8 strlen" as being a function the counts the
>> number of encoded code points, and that's simple enough.
>> (...)
>> Obviously, for some uses, this is too simple as it does not detect
>> incorrect encodings.
>
> I'm not sure what your point was supposed to be. "It's simple to write
> non-practical, prototype-grade code"? Yes, it is.

Yes, that was my point. A lot of people think UTF-8 is more complex
than it is so I think it helps to demystify it a bit.

--
Ben.

Richard Damon <Richard@Damon-Family.org> writes:

> On 1/9/22 9:18 PM, Ben Bacarisse wrote:
>> Mateusz Viste <mateusz@xyz.invalid> writes:
>>
>>> While UTF-8 is neat, it is also complex to decode. Even a simple
>>> strlen() can be challenging.
>> Hmmm... Since UTF-8 is a multi-byte encoding of Unicode code points, I
>> would interpret a "UTF-8 strlen" as being a function the counts the
>> number of encoded code points, and that's simple enough. Every byte,
>> before the null, that does not have 10xxxx it it's top two bits is the
>> start of a code point:
>> size_t ustrlen(char *s)
>> {
>> size_t len = 0;
>> while (*s) len += (*s++ & 0xc0) != 0x80;
>> return len;
>> }
>> Obviously, for some uses, this is too simple as it does not detect
>> incorrect encodings.
>
> My understanding is that for MBCS the function strlen returns the
> number of BYTES in the string, not the number of Multi-Byte Characters
> in the string.
>
> This means that strlen can be used to determine how much space is
> needs to store the string.

Yes. But that's now what "a simple strlen()" for UTF-8 appeared to be
referring to. After all, as you say, strlen /is/ strlen for UTF-8
strings.

--
Ben.

On Monday, 10 January 2022 at 11:47:27 UTC, Ben Bacarisse wrote:
> Richard Damon <Ric...@Damon-Family.org> writes:
>
> > On 1/9/22 9:18 PM, Ben Bacarisse wrote:
> >> Mateusz Viste <mat...@xyz.invalid> writes:
> >>
> >>> While UTF-8 is neat, it is also complex to decode. Even a simple
> >>> strlen() can be challenging.
> >> Hmmm... Since UTF-8 is a multi-byte encoding of Unicode code points, I
> >> would interpret a "UTF-8 strlen" as being a function the counts the
> >> number of encoded code points, and that's simple enough. Every byte,
> >> before the null, that does not have 10xxxx it it's top two bits is the
> >> start of a code point:
> >> size_t ustrlen(char *s)
> >> {
> >> size_t len = 0;
> >> while (*s) len += (*s++ & 0xc0) != 0x80;
> >> return len;
> >> }
> >> Obviously, for some uses, this is too simple as it does not detect
> >> incorrect encodings.
> >
> > My understanding is that for MBCS the function strlen returns the
> > number of BYTES in the string, not the number of Multi-Byte Characters
> > in the string.
> >
> > This means that strlen can be used to determine how much space is
> > needs to store the string.
> Yes. But that's now what "a simple strlen()" for UTF-8 appeared to be
> referring to. After all, as you say, strlen /is/ strlen for UTF-8
> strings.
>
Yes. It's not obvious from the name "strlen" what it should do when fed
UTF-8.

On Monday, 10 January 2022 at 11:02:54 UTC+2, Mateusz Viste wrote:
>
> Now, I am not saying that writing a utf-8 strlen() is incredibly
> difficult of course. I am only saying it is an extra layer of
> complexity compared to UCS-2 or UTF-32. And that is why I understand
> why people often choose to internally store strings in one of these
> encodings instead of utf-8 (esp. if dealing with fixed-width character
> outputs). It's simply easier to deal with an array of values that maps
> directly to codepoints rather than parse a utf-8 string taking care not
> to explode on encoding errors or edge cases.

That argument feels like result of misinterpretation of formation of
UCS-2 and UTF-32 glyphs. Both encodings contain variety of combining
characters, modifiers, accents and tabulators. So even with monospaced
font (in world where proportional fonts are more frequently used) one
can't decide the width of result on screen without examining all
characters in sequence. But if to examine all characters of sequence
anyway then UTF-8 is often.just bit less memory to examine. Somehow
it does not look like the other options are "simply easier".

2022-01-10 at 05:15 -0800, Öö Tiib wrote:
> That argument feels like result of misinterpretation of formation of
> UCS-2 and UTF-32 glyphs. Both encodings contain variety of combining
> characters, modifiers, accents and tabulators. So even with
> monospaced font (in world where proportional fonts are more
> frequently used) one can't decide the width of result on screen
> without examining all characters in sequence.

That is true in a perfect world, yes. In practice, terminal-based
implementations often naively assume that 1 codepoint = 1 character on
screen. And this works fairly well, as far as I can tell, even if it's
not a 100% correct implementation.

I agree that a full-blown unicode implementation can be quite complex
(handling text-direction, combining characters, separators, control
characters, etc). In such context the extra complexity of parsing utf-8
strings may seem irrelevant. All depends what is the implementation's
goal I guess.

Mateusz

On Monday, 10 January 2022 at 13:25:33 UTC, Mateusz Viste wrote:
> 2022-01-10 at 05:15 -0800, Öö Tiib wrote:
> > That argument feels like result of misinterpretation of formation of
> > UCS-2 and UTF-32 glyphs. Both encodings contain variety of combining
> > characters, modifiers, accents and tabulators. So even with
> > monospaced font (in world where proportional fonts are more
> > frequently used) one can't decide the width of result on screen
> > without examining all characters in sequence.
> That is true in a perfect world, yes. In practice, terminal-based
> implementations often naively assume that 1 codepoint = 1 character on
> screen. And this works fairly well, as far as I can tell, even if it's
> not a 100% correct implementation.
>
> I agree that a full-blown unicode implementation can be quite complex
> (handling text-direction, combining characters, separators, control
> characters, etc). In such context the extra complexity of parsing utf-8
> strings may seem irrelevant. All depends what is the implementation's
> goal I guess.
>
You can't really separate Unicode handling from font handling. And that
is notoriously difficult, even if you restrict yourself to English.

On 1/10/22 8:15 AM, Öö Tiib wrote:
> On Monday, 10 January 2022 at 11:02:54 UTC+2, Mateusz Viste wrote:
>>
>> Now, I am not saying that writing a utf-8 strlen() is incredibly
>> difficult of course. I am only saying it is an extra layer of
>> complexity compared to UCS-2 or UTF-32. And that is why I understand
>> why people often choose to internally store strings in one of these
>> encodings instead of utf-8 (esp. if dealing with fixed-width character
>> outputs). It's simply easier to deal with an array of values that maps
>> directly to codepoints rather than parse a utf-8 string taking care not
>> to explode on encoding errors or edge cases.
>
> That argument feels like result of misinterpretation of formation of
> UCS-2 and UTF-32 glyphs. Both encodings contain variety of combining
> characters, modifiers, accents and tabulators. So even with monospaced
> font (in world where proportional fonts are more frequently used) one
> can't decide the width of result on screen without examining all
> characters in sequence. But if to examine all characters of sequence
> anyway then UTF-8 is often.just bit less memory to examine. Somehow
> it does not look like the other options are "simply easier".
>

And this is the reason that Unicode doesn't really meet the requirements
of a C 'Wide Character Type'. Wide characters are supposed to be 1
character = 1 storage unit. Because of combining characters Unicode
doesn't meet this requirement.

Ultimately, we have to live with it and accept that programming in the
face of full compliance with the rules of the character set are going to
add complexity.

On Monday, January 10, 2022 at 1:11:08 AM UTC-5, Öö Tiib wrote:
> On Monday, 10 January 2022 at 07:24:01 UTC+2, james...@alumni.caltech.edu wrote:
> > On Sunday, January 9, 2022 at 5:35:42 PM UTC-5, Öö Tiib wrote:
> > > On Sunday, 9 January 2022 at 18:34:25 UTC+2, Manfred wrote:
> > > > On 1/9/2022 1:44 PM, Öö Tiib wrote:
> > ...
> > > > > Something that makes it clear that it is defect when "Foo😀Bar.txt" is silently opened
> > > > > on file-system that fully supports files named "Foo😀Bar.txt" I suppose.
> > > > >
> > > > Assuming that "Foo≡ƒÿÇBar.txt" "Foo😀Bar.txt" have the same binary
> > > > representation, what's the difference? One form or the other shows up
> > > > only when it is displayed in some UI - the filesystem isn't one, which
> > > > leads to the implementation's runtime behavior.
> > > How you mean same binary representation? Both "Foo😀Bar.txt" and
> > > "Foo😀Bar.txt" files can be in same directory. Both have Unicode
> > > names in underlying file system precisely as posted.
> > Have you checked to make sure? Any system where passing the UTF-8 string
> > "Foo😀Bar.txt" to fopen() opens a file whose name displays as Foo≡ƒÿÇBar.txt
> > is likely to be a system where the file names are displayed using some single-
> > byte encoding. The UTF-8 encoding of "Foo😀Bar.txt" is
> >
> > 0X46 0X6F 0X6F 0XF0 0X9F 0X98 0X80 0X42 0X61 0X72 0X2E 0X74 0X78 0X74
> >
> > After quite a bit of searching, I found Code page 865 (MS-DOS Nordic), which
> > has 0XF0 = '≡', 0x9F = 'ƒ' and 0X98 = 'ÿ'. If the utilities that you use to display file
> > names used that encoding to interpret the file name, that would explain your results.
> Need a screenshot with files named "Foo😀Bar.txt" and "Foo≡ƒÿÇBar.txt"
> side-by-side? ...

No, that wouldn't be relevant. I realized shortly after I posted my message that you
might not understand what I meant by "Have you checked to make sure?" I started
composing a message in my head explaining in more detail. However, it was very late,
I had to get to bed, and when I checked this morning you'd already confirmed that you
didn't realize what I meant.

As I understand it, you've opened a file using "Foo😀Bar.txt" as the file name, and
somehow determined that the "actual" name of the file that got opened was
"Foo😀Bar.txt". You didn't specify, but I presume you reached that conclusion by
doing something like get a directory listing at the command line or using a GUI file
browser to look at the directory.

The UTF-8 encoding of "Foo😀Bar.txt" is

0X46 0X6F 0X6F 0XE2 0X89 0XA1 0XC6 0X92 0XC3 0XBF 0XC3 0X87 0X42 0X61 0X72 0X2E 0X74 0X78 0X7

Those same bytes, if interpreted using Code page 865, represent the string
"Foo😀Bar.txt". I know very little about Windows internals, so I'm not sure why that
might be relevant. It could be just a coincidence, but that seems excessively
unlikely. What I was suggestinging, and what I think Manfred was hinting at, is that the
string you provide as the file name is stored by the file system using UTF-8 encoding.
Whatever method you used to determine the "actual" file name interpreted those
bytes using a single-byte encoding, which could be Code Page 865, or possibly some
other encoding that encodes those particular characters the same way as Code Pag
865. There's a lot of different code pages out there, so I couldn't check them all, but of
the dozen or so I checked, that is the only one where 0xF0 represents '≡'. The "MS
DOS Nordic" code page was one of the first ones I checked, based upon your e-mail
address ootiib@hot.ee, where I presume "ee" refers to Estonia.

If that is indeed the case, consider what should happen if you try to open a file using
the name "Foo😀Bar.txt". If I understand you correctly, I believe that you would
expect it to open the same file that got opened when you specified "Foo😀Bar.txt".
However, the UTF-8 encoding of "Foo😀Bar.txt" is

0X46 0X6F 0X6F 0XE2 0X89 0XA1 0XC6 0X92 0XC3 0XBF 0XC3 0X87 0X42 0X61 0X72 0X2E 0X74 0X78 0X74

I expect that you will end up opening a different file. If the file name is being displayed
using a single-byte encoding, it should have 19 characters. If that encoding is in fact
Code Page 865, then that name should be "Foo😀Bar.txt". So, what result do
you get?

If the problem is in fact that the file name is being interpreted using a single byte
encoding by whatever utility you're using determine what the actual name is, then
there's absolutely nothing the standards can do about that - the behavior of any such
utility is completely outside the scope of either standard.

> ... Just that for "Foo😀Bar.txt" one needs to use non-standard
> _wfopen( L"Foo😀Bar.txt", L"w"). That is default both with MSVC and MinGW
> gcc.

As you say, it's non-standard. Therefore, nothing the C standard says could do
anything to constrain it's behavior. If your complaint is indeed about the behavior of
_wfopen(), it's not relevant to either the C or C++ standards, and should be posted to a
Windows-specific forum.

On 10/01/2022 00:51, Öö Tiib wrote:
> But do there exist machines that do want to support texts as char* but
> do not want to support UTF-8? Describe those machines, give examples.

All the mainframes that run EBCDIC. There are a lot of them still.

Andy

Vir Campestris <vir.campestris@invalid.invalid> writes:
>On 10/01/2022 00:51, Öö Tiib wrote:
>> But do there exist machines that do want to support texts as char* but
>> do not want to support UTF-8? Describe those machines, give examples.
>
>All the mainframes that run EBCDIC. There are a lot of them still.

Here's the implementation guide for one:

https://public.support.unisys.com/aseries/docs/ClearPath-MCP-18.0/86002268-207.pdf

See appendix E for I18N.

On Tuesday, 11 January 2022 at 00:09:48 UTC+2, Scott Lurndal wrote:
> Vir Campestris <vir.cam...@invalid.invalid> writes:
> >On 10/01/2022 00:51, Öö Tiib wrote:
> >> But do there exist machines that do want to support texts as char* but
> >> do not want to support UTF-8? Describe those machines, give examples.
> >
> >All the mainframes that run EBCDIC. There are a lot of them still.
> Here's the implementation guide for one:
>
> https://public.support.unisys.com/aseries/docs/ClearPath-MCP-18.0/86002268-207.pdf
>
> See appendix E for I18N.

Seems out of context as these guys do not look like wanting to upgrade to C99 or
something. But maybe at 2060 or so they start to think about usefulness of
UTF-8 too. The EBCDIC is usual, but irrelevant red herring in discussions like this.