On Saturday, 8 January 2022 at 19:17:19 UTC+2, Malcolm McLean wrote:
> On Saturday, 8 January 2022 at 16:17:53 UTC, Öö Tiib wrote:
> > On Saturday, 8 January 2022 at 06:52:33 UTC+2, james...@alumni.caltech.edu wrote:
> >
> > > So, you are arguing that it should be mandatory to have UTF-8 as the encoding for
> > > unprefixed string literals, even for implementations targeting platforms where that's
> > > contrary to the conventions for that platform?
> > Yes, and vast majority would be happy. What other char* text is needed than UTF-8?
> > Why? On what? For what? Must be odd corner case.
> >
> Where you've got an 8 bit character-mapped display that supports ascii plus some
> extended characters. That used to be almost every microcomputer, and it still lives
> on to a bit in modern PCs.

When? How long ago? In eighties?
I have had the fun to participate in programming such panel more than decade ago
to support showing whatever text including 8,105 "simplified" Chinese characters if
needed. Wasn't that big a project and the panel was dirt cheap. I disliked that it
was pointlessly required to use UTF-16 as UTF-8 could make it even simpler. What
is the reason to use for character-mapping in our current world anything but UTF-8?
In current PCs it lives on as deliberate sabotage by the platform vendor as there are
no reasons others than desire to make their proprietary programming language to
look better than C.

2022-01-08 at 10:20 -0800, Öö Tiib wrote:
> I disliked that it was pointlessly required to use UTF-16 as UTF-8
> could make it even simpler. What is the reason to use for
> character-mapping in our current world anything but UTF-8?

While UTF-8 is neat, it is also complex to decode. Even a simple
strlen() can be challenging. That's where UTF-16 or UTF-32 are handy
since, there is no decoding required and every glyph has a fixed
byte length.

Mateusz

On Saturday, January 8, 2022 at 11:17:53 AM UTC-5, Öö Tiib wrote:
> On Saturday, 8 January 2022 at 06:52:33 UTC+2, james...@alumni.caltech.edu wrote:
....
> > So, you are arguing that it should be mandatory to have UTF-8 as the encoding for
> > unprefixed string literals, even for implementations targeting platforms where that's
> > contrary to the conventions for that platform?
> Yes, and vast majority would be happy. What other char* text is needed than UTF-8?

Well, let me ask you - does the implementation you use most often use UTF-8
encoding for unprefixed string literals? Since you're complaining about the difficulty of
using UTF-8, I presume that it doesn't. If not, why not? The standard doesn't say
anything to prevent that implementation from doing so. If they don't, it can only be
because they don't want to. So why don't you ask the implementors why they made
that decision? They've got a reason that seemed sufficiently good for them, find out
what it is.

....
> FILE *f = fopen( "Foo😀Bar.txt", "w");
> That should work unless underlying file system does not support files
> named "Foo😀Bar.txt" If it supports but the code does not work then it indicates
> bad standard that allows implementations to weasel away. No garbage like
> u8fopen( u8"Foo😀Bar.txt", "w") coming somewhere maybe in C35 or so is
> needed as it already works like in my example on vast majority of things.

Nothing in the standard prevents an implementation from doing that. If one doesn't
already do so, that's a choice made by the implementors, and you should ask them
about it. Your real beef is with the implementors, not the standard.

....
> You never answered why should they use obscure extensions for what they need on
> majority of cases. Why UTF-8 must be obscure extension?

They shouldn't. It isn't. It's an implementation-defined choice, and if an
implementation you want to use forces you to use an obscure extension, in order to
work with UTF-8, you should ask them why - it's nothing the standard forced them to
do. I don't need to use any extensions to work with UTF-8 on my desktop. I also don't
need to use UTF-8, but that's a separate matter.

....
> It is used on close to 100% of cases anyway. I am objecting that it is deliberately
> standardized (or more like pseudo-standardized/non-standardized) to be
> inconvenient to use.

How would you like it to be more convenient? printf("Foo😀Bar.txt") works fine on my
system. If it doesn't work on yours, talk with your implementor.

....
> Agreed. So do you have numbers how many C programmers *want* to use UTF-16? I
> think that it is little, but I do not have any sources. They may *need* to for legacy
> reasons I already mentioned but even there it is most likely small number.. Their
> pain with support to their u"string", L"string" and \x \u \U character references
> might need relieving too but is bit different topic.

If they don't want it, and in particular, don't need it, then it should be pretty easy to
convince implementors to use UTF-8 instead. Have you tried? If they refuse, their
response is likely to give you far more relevant information than I could give you, since
I don't work on that platform any more, and didn't work on it very long.

The fundamental problem is - why should people who want to use some other
encoding for unprefixed string literals be forced to use UTF-8 instead, just because
you disagree? How does the existence of implementation catering to their needs hurt
you? Those implementations aren't the reason why using UTF-8 is complicated on the
implementations you use - that's entirely due to decisions made by your implementor
- so talk to the implementor and try to convince them to change.

If it seems unreasonable to you that you should have to convince one implementor to
adopt UTF-8, keep this in mind: however hard it is to convince a single implementor to
change, it would be much harder to convince the C and C++ committees to make such
a change. If you do want to convince those committees, a good way to start is by
convincing a single implmentation to change.

On Saturday, 8 January 2022 at 20:37:38 UTC+2, Mateusz Viste wrote:
> 2022-01-08 at 10:20 -0800, Öö Tiib wrote:
> > I disliked that it was pointlessly required to use UTF-16 as UTF-8
> > could make it even simpler. What is the reason to use for
> > character-mapping in our current world anything but UTF-8?
>
> While UTF-8 is neat, it is also complex to decode. Even a simple
> strlen() can be challenging. That's where UTF-16 or UTF-32 are handy
> since, there is no decoding required and every glyph has a fixed
> byte length.

That is incorrect as glyph 👌🏽 is U+1F44C U+1F3FD so all three are of
varying length. That makes UTF-8 the sole sane one.

2022-01-08 at 10:53 -0800, Öö Tiib wrote:
> That is incorrect as glyph 👌🏽 is U+1F44C U+1F3FD so all three are of
> varying length. That makes UTF-8 the sole sane one.

You are right, the implementations of UTF-16 I worked on were limited
to the BMP (ie. always 2 bytes), hence my simplified view.

Still, UTF-32 is always 4 bytes for any possible glyph, isn't it?

Mateusz

On 1/8/22 7:37 PM, Mateusz Viste wrote:
> 2022-01-08 at 10:20 -0800, Öö Tiib wrote:
>> I disliked that it was pointlessly required to use UTF-16 as UTF-8
>> could make it even simpler. What is the reason to use for
>> character-mapping in our current world anything but UTF-8?
>
> While UTF-8 is neat, it is also complex to decode. Even a simple
> strlen() can be challenging. That's where UTF-16 or UTF-32 are handy
> since, there is no decoding required and every glyph has a fixed
> byte length.

As Öö Tiib wrote, UTF-16 is variable length - its predecessor (under
Windows) UCS-2 is fixed length, but it failed to keep the promise to
accommodate for the entire universe of language glyphs.
Furthermore, the recent fashion of adding emojis to Unicode has made
UTF-32 no longer fixed length as well.

However, the problem with strlen() is most often a false problem: most
often you need to know the size of the string in memory, and that's
bytes, rather than the count of glyphs in the string. Which you still
can do, by the way, but it doesn't have to have the performance
requirements of strlen, for example.

>
> Mateusz
>

On Saturday, 8 January 2022 at 19:12:14 UTC, Mateusz Viste wrote:
> 2022-01-08 at 10:53 -0800, Öö Tiib wrote:
> > That is incorrect as glyph 👌🏽 is U+1F44C U+1F3FD so all three are of
> > varying length. That makes UTF-8 the sole sane one.
> You are right, the implementations of UTF-16 I worked on were limited
> to the BMP (ie. always 2 bytes), hence my simplified view.
>
> Still, UTF-32 is always 4 bytes for any possible glyph, isn't it?
>
The problem is that not all languages fit into the Latin mould, where
you have one letter taking up one physical rectangle of space in the
writing area.
In some languages, there are combining forms. We see this a bit
in European scripts, where you have accents. But in other languages
it can go much deeper, and you can't really provide one code per glyph.

On 08/01/2022 18:53, Öö Tiib wrote:
> On Saturday, 8 January 2022 at 20:37:38 UTC+2, Mateusz Viste wrote:
>> 2022-01-08 at 10:20 -0800, Öö Tiib wrote:
>>> I disliked that it was pointlessly required to use UTF-16 as UTF-8
>>> could make it even simpler. What is the reason to use for
>>> character-mapping in our current world anything but UTF-8?
>>
>> While UTF-8 is neat, it is also complex to decode. Even a simple
>> strlen() can be challenging. That's where UTF-16 or UTF-32 are handy
>> since, there is no decoding required and every glyph has a fixed
>> byte length.
>
> That is incorrect as glyph 👌🏽 is U+1F44C U+1F3FD so all three are of
> varying length. That makes UTF-8 the sole sane one.

This is Unicode crossing the line into typography, markup and clip-art.

The first is an actual character, or rather, a symbol (especially by
'glyph' is meant only the shape or design). The second is a modifier, I
believe of the colour, which IMO don't belong (along with font, height,
aspect and weight, among other attibutes).

I'm sure there were plenty of such schemes in ASCII too, although more
recently they take form of explicit tag strings. But it's still the case
that a linebreak in ASCII can be CR,LF or just LF; so is that one
character or two?

At the heart, though, everyone knows that a plain ASCII string of
printable characters, one - one containing no control codes, attributes
or other meta-data - can be represented by an array of bytes, one per
character.

Similarly, most such strings of full 21-bit (ie. 32 bits in practice)
Unicode codes can be represented by an indexable array of 32-bit values.

If you really, really need those multi-Unicode sequences, then you can
choose to represent a string as an array of variable-length short
strings, most of which will be one 32-bit character long.

Although there will doubtless be other special requirements that would
make that impractical too. But then, the very definition of what is a
character or word will be blurred as well.

Le 08/01/2022 à 19:37, Mateusz Viste a écrit :
> 2022-01-08 at 10:20 -0800, Öö Tiib wrote:
>> I disliked that it was pointlessly required to use UTF-16 as UTF-8
>> could make it even simpler. What is the reason to use for
>> character-mapping in our current world anything but UTF-8?
>
> While UTF-8 is neat, it is also complex to decode. Even a simple
> strlen() can be challenging. That's where UTF-16 or UTF-32 are handy
> since, there is no decoding required and every glyph has a fixed
> byte length.

Yeah. While fixed-width characters are certainly easier (and, most of
all, faster) to handle, UTF-8 is not rocket science. The encoding is
pretty simple.

The downside is more speed than complexity.

On Saturday, 8 January 2022 at 20:49:01 UTC+2, james...@alumni.caltech.edu wrote:
> On Saturday, January 8, 2022 at 11:17:53 AM UTC-5, Öö Tiib wrote:
> > On Saturday, 8 January 2022 at 06:52:33 UTC+2, james...@alumni.caltech.edu wrote:
> ...
> > > So, you are arguing that it should be mandatory to have UTF-8 as the encoding for
> > > unprefixed string literals, even for implementations targeting platforms where that's
> > > contrary to the conventions for that platform?
> > Yes, and vast majority would be happy. What other char* text is needed than UTF-8?
>
> Well, let me ask you - does the implementation you use most often use UTF-8
> encoding for unprefixed string literals? Since you're complaining about the difficulty of
> using UTF-8, I presume that it doesn't. If not, why not?

All compilers that I have used did it for some time. Or at least could be configured to.
Sometimes the configuration had to be done in inconvenient manner but these
are the easy parts of our work. I ignored that. Also I did ignore the unneeded u8
prefixes. When some confused novice added it then it did not matter. After all
these two were working same in C++17:

const char crap[] = "Öö Tiib 😀";
const char crap8[] = u8"Öö Tiib 😀";

But C++20 gives error about second line. Also cast does not compile there.
So that makes me angry opponent of the whole u8 prefix. It should be gone from
language. They may add their char_iso8859_1_t and iso8859_1"strings" if they
want to but should raise the privileges of the UTF-8 to be always supported
as char array and not fucked with.

> The standard doesn't say
> anything to prevent that implementation from doing so. If they don't, it can only be
> because they don't want to. So why don't you ask the implementors why they made
> that decision? They've got a reason that seemed sufficiently good for them, find out
> what it is.

No, that fish rots from the head, IOW from standards. MS abuses it more than others
but only because they are bit bigger assholes.
> ...
> > FILE *f = fopen( "Foo😀Bar.txt", "w");
> > That should work unless underlying file system does not support files
> > named "Foo😀Bar.txt" If it supports but the code does not work then it indicates
> > bad standard that allows implementations to weasel away. No garbage like
> > u8fopen( u8"Foo😀Bar.txt", "w") coming somewhere maybe in C35 or so is
> > needed as it already works like in my example on vast majority of things.
>
> Nothing in the standard prevents an implementation from doing that. If one doesn't
> already do so, that's a choice made by the implementors, and you should ask them
> about it. Your real beef is with the implementors, not the standard.

My beef is with standards. Adding garbage that does not work to standard is wrong
and not adding what everybody at least half sane does use to standard is also wrong.

>
> They shouldn't. It isn't. It's an implementation-defined choice, and if an
> implementation you want to use forces you to use an obscure extension, in order to
> work with UTF-8, you should ask them why - it's nothing the standard forced them to
> do. I don't need to use any extensions to work with UTF-8 on my desktop. I also don't
> need to use UTF-8, but that's a separate matter.

Oh, if I can't convince even experienced person like you that the obfuscation
around UTF-8 in standards is evil then there are no point to discuss that
position with any implementer.

On 1/8/2022 11:50 PM, Öö Tiib wrote:
> On Saturday, 8 January 2022 at 20:49:01 UTC+2, james...@alumni.caltech.edu wrote:
>> On Saturday, January 8, 2022 at 11:17:53 AM UTC-5, Öö Tiib wrote:
>>> On Saturday, 8 January 2022 at 06:52:33 UTC+2, james...@alumni.caltech.edu wrote:
>> ...
>>>> So, you are arguing that it should be mandatory to have UTF-8 as the encoding for
>>>> unprefixed string literals, even for implementations targeting platforms where that's
>>>> contrary to the conventions for that platform?
>>> Yes, and vast majority would be happy. What other char* text is needed than UTF-8?
>>
>> Well, let me ask you - does the implementation you use most often use UTF-8
>> encoding for unprefixed string literals? Since you're complaining about the difficulty of
>> using UTF-8, I presume that it doesn't. If not, why not?
>
> All compilers that I have used did it for some time. Or at least could be configured to.
> Sometimes the configuration had to be done in inconvenient manner but these
> are the easy parts of our work. I ignored that. Also I did ignore the unneeded u8
> prefixes. When some confused novice added it then it did not matter. After all
> these two were working same in C++17:
>
> const char crap[] = "Öö Tiib 😀";
> const char crap8[] = u8"Öö Tiib 😀";
>
> But C++20 gives error about second line. Also cast does not compile there.
> So that makes me angry opponent of the whole u8 prefix. It should be gone from
> language. They may add their char_iso8859_1_t and iso8859_1"strings" if they
> want to but should raise the privileges of the UTF-8 to be always supported
> as char array and not fucked with.

The argument you are making here is more than convincing to me, but let
me try the devil's advocate role here.

Granted, the Venerable Luminaries of the Holy Committee screwed up big
time, but they did it in C++17 (how surprising) rather than in C++20.

In principle, I could imagine a use for u8"" strings that are compatible
with some family of printf8() functions only, a sort of tight type
constraint for character types. This would have probably ended up like
annex K, but still it could have made sense to some Unicode purists, and
more importantly it would have made no harm to the sane world.

BUT the fact that C++17 allowed your second string, and thus people
started naively using it, and THEN C++ prohibited it, thus breaking said
naïve but so far legal code, denotes some serious dickheadedness, yes.

This is to say that the solution might be to consider C++17 a sad
parenthesis (once again), and only use char, char16_t (wchar_t?) and
char32_t where needed.
Applications where a distinct separation between utf_8 and generic char
is important can pay the price of using u8, but the majority of
applications would most probably ignore it.

>
>> The standard doesn't say
>> anything to prevent that implementation from doing so. If they don't, it can only be
>> because they don't want to. So why don't you ask the implementors why they made
>> that decision? They've got a reason that seemed sufficiently good for them, find out
>> what it is.
>
> No, that fish rots from the head, IOW from standards. MS abuses it more than others
> but only because they are bit bigger assholes.
>

Agreed.

>> ...
>>> FILE *f = fopen( "Foo😀Bar.txt", "w");
>>> That should work unless underlying file system does not support files
>>> named "Foo😀Bar.txt" If it supports but the code does not work then it indicates
>>> bad standard that allows implementations to weasel away. No garbage like
>>> u8fopen( u8"Foo😀Bar.txt", "w") coming somewhere maybe in C35 or so is
>>> needed as it already works like in my example on vast majority of things.
>>
>> Nothing in the standard prevents an implementation from doing that. If one doesn't
>> already do so, that's a choice made by the implementors, and you should ask them
>> about it. Your real beef is with the implementors, not the standard.
>
> My beef is with standards. Adding garbage that does not work to standard is wrong
> and not adding what everybody at least half sane does use to standard is also wrong.
>

Also agreed, but since utf-8 is transparent to ascii functions, what
should have been added?
I mean, if printf can't print utf-8, it is a problem of the console
rather than printf itself, right? So some way to set the console in
utf-8 mode? But that is outside the scope of the standard, isn't it?

>>
>> They shouldn't. It isn't. It's an implementation-defined choice, and if an
>> implementation you want to use forces you to use an obscure extension, in order to
>> work with UTF-8, you should ask them why - it's nothing the standard forced them to
>> do. I don't need to use any extensions to work with UTF-8 on my desktop. I also don't
>> need to use UTF-8, but that's a separate matter.
>
> Oh, if I can't convince even experienced person like you that the obfuscation
> around UTF-8 in standards is evil then there are no point to discuss that
> position with any implementer.
>

"james...@alumni.caltech.edu" <jameskuyper@alumni.caltech.edu> writes:

> No, I am quite accurately and honestly expressing my confusion. You
> object to something being prohibited by the standards that is, to the
> best of my understanding, allowed. It would make more sense if you
> were objecting the fact that it isn't mandatory, and if you were
> making such claims, I would disagree with you about whether it would
> be a good idea to make it mandatory - but as far as I can tell, you're
> claiming it isn't allowed.

AFAICT, he's complaining about Microsoft's specific implementations of
some standards.

I'm not an MS-Windows programmer, but from a Unix point-of-view their
way of doing things is indeed confusing -- at least when I looked into
porting some programs.

It's probably a matter of habit: I'm sure the distinction between wide
and ASCII system calls, code pages, and text and binary streams comes
naturally to MS-Windows programmers, who in turn find the lack of
explicit text streams in Unix confusing.

On 08/01/2022 20:12, Mateusz Viste wrote:
> 2022-01-08 at 10:53 -0800, Öö Tiib wrote:
>> That is incorrect as glyph 👌🏽 is U+1F44C U+1F3FD so all three are of
>> varying length. That makes UTF-8 the sole sane one.
>
> You are right, the implementations of UTF-16 I worked on were limited
> to the BMP (ie. always 2 bytes), hence my simplified view.
>

When Unicode was young, the intention was that every glyph was one
character, and it would all fit in 16-bits - that was UCS2, as used
originally by Windows NT, Java, Python, QT, and other systems, languages
and libraries. But it was quickly discovered that this was far from
sufficient.

> Still, UTF-32 is always 4 bytes for any possible glyph, isn't it?
>

The terminology of Unicode can be a little confusing. (And I'm sure
someone will correct me if I get it wrong.)

A "code point" is an entry in the Unicode tables. Each code point is
uniquely identified by a 32-bit number. The code points are organised
in "planes" for convenience, and designed so that the first 128 code
points match ASCII and that a wide range of languages can be covered by
the code units in the range 0x0000 .. 0xffff (excluding 0xd800 ..
0xdfff) so that 16 bits would often be enough.

A "code unit" is the container for the bits of the encoding. In UTF-8,
a code unit is an 8-bit unit. In UTF-16, it is 16-bit, in UTF-32 it is
32-bit.

UTF-8 takes up to four code units (32 bits total) per code point, UTF-16
takes up to two code units, and UTF-32 takes exactly one code unit per
code point. UTF-8 is always at least as compact as UTF-32, and will be
more or less compact than UTF-16 depending on the content. These are
just different encodings - different ways to write the code points.
There are others, such as GB18030 which is a 16-bit encoding popular in
China because it matches their traditional GB encodings in the same way
UTF-8 matches ASCII.

A "grapheme" is a written mark - a letter, punctuation, accent, etc.,
that conveys meaning. Sometimes it is useful to break them down,
sometimes it is useful to treat them separately. For example, "é" can
considered as a single grapheme, or as a grapheme "e" followed by a
combining graphene "'" acute accent. The same grapheme can match
multiple code points - a Latin alphabet capital A is the same as a Greek
alphabet capital Alpha.

A "glyph" is a rendering of a grapheme - the letter "A" in different
fonts are different glyphs of the same grapheme.

What the reader perceives as a "character" is often a single grapheme,
but might be several graphemes together.

So, with that in mind, all three UTF formats require multiple code units
to cover all graphemes. But UTF-32 always gets one code point per code
unit, making it simpler and more consistent for processing Unicode text.
As a file or transfer encoding, it has the big inconvenience of being
endian-specific as well as being bulkier than UTF-8. UTF-16 combines
the worst features of UTF-8 with the worst features of UTF-32, with none
of the benefits - it exists solely because early Unicode adopters
committed too strongly to UCS2.

People are often concerned that UTF-8 is difficult or complex to decode
or split up. It is not, in practice. It is actually quite rare that
you need to divide up a string based on characters or even find its
length in code points - for most uses of strings, you just pass them
around without bothering about the details of the contents. You need to
know how much memory the string takes, not how many code points it has.
And simply treating it as an abstract stream of data terminated by a
zero character can be enough to give you a useable sorting and
uniqueness comparison for many uses. The point where you need to decode
the code units and know what they mean is when you are doing rendering,
sorting, or other human interaction - and then you have such a vastly
bigger task that turning UTF-8 coding into UTF-32 code points is
negligible effort in comparison.

(And UTF-8 is not much harder to encode or decode than UTF-16.)

On Sunday, 9 January 2022 at 03:17:10 UTC+2, Manfred wrote:
> On 1/8/2022 11:50 PM, Öö Tiib wrote:
> > On Saturday, 8 January 2022 at 20:49:01 UTC+2, james...@alumni.caltech.edu wrote:
> >> On Saturday, January 8, 2022 at 11:17:53 AM UTC-5, Öö Tiib wrote:
> >>> On Saturday, 8 January 2022 at 06:52:33 UTC+2, james...@alumni.caltech.edu wrote:
> >> ...
> >>>> So, you are arguing that it should be mandatory to have UTF-8 as the encoding for
> >>>> unprefixed string literals, even for implementations targeting platforms where that's
> >>>> contrary to the conventions for that platform?
> >>> Yes, and vast majority would be happy. What other char* text is needed than UTF-8?
> >>
> >> Well, let me ask you - does the implementation you use most often use UTF-8
> >> encoding for unprefixed string literals? Since you're complaining about the difficulty of
> >> using UTF-8, I presume that it doesn't. If not, why not?
> >
> > All compilers that I have used did it for some time. Or at least could be configured to.
> > Sometimes the configuration had to be done in inconvenient manner but these
> > are the easy parts of our work. I ignored that. Also I did ignore the unneeded u8
> > prefixes. When some confused novice added it then it did not matter. After all
> > these two were working same in C++17:
> >
> > const char crap[] = "Öö Tiib 😀";
> > const char crap8[] = u8"Öö Tiib 😀";
> >
> > But C++20 gives error about second line. Also cast does not compile there.
> > So that makes me angry opponent of the whole u8 prefix. It should be gone from
> > language. They may add their char_iso8859_1_t and iso8859_1"strings" if they
> > want to but should raise the privileges of the UTF-8 to be always supported
> > as char array and not fucked with.
> The argument you are making here is more than convincing to me, but let
> me try the devil's advocate role here.
>
> Granted, the Venerable Luminaries of the Holy Committee screwed up big
> time, but they did it in C++17 (how surprising) rather than in C++20.
>
> In principle, I could imagine a use for u8"" strings that are compatible
> with some family of printf8() functions only, a sort of tight type
> constraint for character types. This would have probably ended up like
> annex K, but still it could have made sense to some Unicode purists, and
> more importantly it would have made no harm to the sane world.
>
> BUT the fact that C++17 allowed your second string, and thus people
> started naively using it, and THEN C++ prohibited it, thus breaking said
> naïve but so far legal code, denotes some serious dickheadedness, yes.
>
> This is to say that the solution might be to consider C++17 a sad
> parenthesis (once again), and only use char, char16_t (wchar_t?) and
> char32_t where needed.
> Applications where a distinct separation between utf_8 and generic char
> is important can pay the price of using u8, but the majority of
> applications would most probably ignore it.

Hmm. Great is that at least you are convinced. For me C++17 and
C++20 both added too big number of silently changing or silently
turning into undefined behaviors so the noisy one is even better
than the rest of it. Just that the importance of UTF-8 in software
development industry is hard to overestimate.

....

> >>> FILE *f = fopen( "Foo😀Bar.txt", "w");
> >>> That should work unless underlying file system does not support files
> >>> named "Foo😀Bar.txt" If it supports but the code does not work then it indicates
> >>> bad standard that allows implementations to weasel away. No garbage like
> >>> u8fopen( u8"Foo😀Bar.txt", "w") coming somewhere maybe in C35 or so is
> >>> needed as it already works like in my example on vast majority of things.
> >>
> >> Nothing in the standard prevents an implementation from doing that. If one doesn't
> >> already do so, that's a choice made by the implementors, and you should ask them
> >> about it. Your real beef is with the implementors, not the standard.
> >
> > My beef is with standards. Adding garbage that does not work to standard is wrong
> > and not adding what everybody at least half sane does use to standard is also wrong.
> >
> Also agreed, but since utf-8 is transparent to ascii functions, what
> should have been added?

Something that makes it clear that it is defect when "Foo😀Bar.txt" is silently opened
on file-system that fully supports files named "Foo😀Bar.txt" I suppose.

> I mean, if printf can't print utf-8, it is a problem of the console
> rather than printf itself, right? So some way to set the console in
> utf-8 mode? But that is outside the scope of the standard, isn't it?

The console output can be set to UTF-8 mode with few lines of platform specific
code ... its keyboard input can't but that is all about vendor ... I agree with James there.

On 1/9/2022 1:44 PM, Öö Tiib wrote:
> On Sunday, 9 January 2022 at 03:17:10 UTC+2, Manfred wrote:
>> On 1/8/2022 11:50 PM, Öö Tiib wrote:
>>> On Saturday, 8 January 2022 at 20:49:01 UTC+2, james...@alumni.caltech.edu wrote:
>>>> On Saturday, January 8, 2022 at 11:17:53 AM UTC-5, Öö Tiib wrote:
>>>>> On Saturday, 8 January 2022 at 06:52:33 UTC+2, james...@alumni.caltech.edu wrote:
>>>> ...
>>>>>> So, you are arguing that it should be mandatory to have UTF-8 as the encoding for
>>>>>> unprefixed string literals, even for implementations targeting platforms where that's
>>>>>> contrary to the conventions for that platform?
>>>>> Yes, and vast majority would be happy. What other char* text is needed than UTF-8?
>>>>
>>>> Well, let me ask you - does the implementation you use most often use UTF-8
>>>> encoding for unprefixed string literals? Since you're complaining about the difficulty of
>>>> using UTF-8, I presume that it doesn't. If not, why not?
>>>
>>> All compilers that I have used did it for some time. Or at least could be configured to.
>>> Sometimes the configuration had to be done in inconvenient manner but these
>>> are the easy parts of our work. I ignored that. Also I did ignore the unneeded u8
>>> prefixes. When some confused novice added it then it did not matter. After all
>>> these two were working same in C++17:
>>>
>>> const char crap[] = "Öö Tiib 😀";
>>> const char crap8[] = u8"Öö Tiib 😀";
>>>
>>> But C++20 gives error about second line. Also cast does not compile there.
>>> So that makes me angry opponent of the whole u8 prefix. It should be gone from
>>> language. They may add their char_iso8859_1_t and iso8859_1"strings" if they
>>> want to but should raise the privileges of the UTF-8 to be always supported
>>> as char array and not fucked with.
>> The argument you are making here is more than convincing to me, but let
>> me try the devil's advocate role here.
>>
>> Granted, the Venerable Luminaries of the Holy Committee screwed up big
>> time, but they did it in C++17 (how surprising) rather than in C++20.
>>
>> In principle, I could imagine a use for u8"" strings that are compatible
>> with some family of printf8() functions only, a sort of tight type
>> constraint for character types. This would have probably ended up like
>> annex K, but still it could have made sense to some Unicode purists, and
>> more importantly it would have made no harm to the sane world.
>>
>> BUT the fact that C++17 allowed your second string, and thus people
>> started naively using it, and THEN C++ prohibited it, thus breaking said
>> naïve but so far legal code, denotes some serious dickheadedness, yes.
>>
>> This is to say that the solution might be to consider C++17 a sad
>> parenthesis (once again), and only use char, char16_t (wchar_t?) and
>> char32_t where needed.
>> Applications where a distinct separation between utf_8 and generic char
>> is important can pay the price of using u8, but the majority of
>> applications would most probably ignore it.
>
> Hmm. Great is that at least you are convinced. For me C++17 and
> C++20 both added too big number of silently changing or silently
> turning into undefined behaviors so the noisy one is even better
> than the rest of it.

Yes, and the silent one is in C++17. From your example, in C++20 the
compiler doesn't allow you to pass a u8"string" to printf, does it?
If u8 was started this way from the beginning, then the problem you
mention above wouldn't exist.

Just that the importance of UTF-8 in software
> development industry is hard to overestimate.
>
> ...
>
>>>>> FILE *f = fopen( "Foo😀Bar.txt", "w");
>>>>> That should work unless underlying file system does not support files
>>>>> named "Foo😀Bar.txt" If it supports but the code does not work then it indicates
>>>>> bad standard that allows implementations to weasel away. No garbage like
>>>>> u8fopen( u8"Foo😀Bar.txt", "w") coming somewhere maybe in C35 or so is
>>>>> needed as it already works like in my example on vast majority of things.
>>>>
>>>> Nothing in the standard prevents an implementation from doing that. If one doesn't
>>>> already do so, that's a choice made by the implementors, and you should ask them
>>>> about it. Your real beef is with the implementors, not the standard.
>>>
>>> My beef is with standards. Adding garbage that does not work to standard is wrong
>>> and not adding what everybody at least half sane does use to standard is also wrong.
>>>
>> Also agreed, but since utf-8 is transparent to ascii functions, what
>> should have been added?
>
> Something that makes it clear that it is defect when "Foo😀Bar.txt" is silently opened
> on file-system that fully supports files named "Foo😀Bar.txt" I suppose.
>

Assuming that "Foo≡ƒÿÇBar.txt" "Foo😀Bar.txt" have the same binary
representation, what's the difference? One form or the other shows up
only when it is displayed in some UI - the filesystem isn't one, which
leads to the implementation's runtime behavior.

If they are actually different in their binary sequence, and this is the
result of the utf-8 string being wrongly converted multiple times, this
looks like a bad implementation, rather than a problem with the standard.
IIUC you are advocating for some statement in the standard that prevents
implementations from messing up with "character sets" in null terminated
char strings?

>> I mean, if printf can't print utf-8, it is a problem of the console
>> rather than printf itself, right? So some way to set the console in
>> utf-8 mode? But that is outside the scope of the standard, isn't it?
>
> The console output can be set to UTF-8 mode with few lines of platform specific
> code ... its keyboard input can't but that is all about vendor ... I agree with James there.
>

On 1/9/22 7:33 AM, David Brown wrote:

> A "grapheme" is a written mark - a letter, punctuation, accent, etc.,
> that conveys meaning. Sometimes it is useful to break them down,
> sometimes it is useful to treat them separately. For example, "é" can
> considered as a single grapheme, or as a grapheme "e" followed by a
> combining graphene "'" acute accent. The same grapheme can match
> multiple code points - a Latin alphabet capital A is the same as a Greek
> alphabet capital Alpha.
>
> A "glyph" is a rendering of a grapheme - the letter "A" in different
> fonts are different glyphs of the same grapheme.
>
> What the reader perceives as a "character" is often a single grapheme,
> but might be several graphemes together.
>

No, a grapheme, from my understanding, is the character as perceived by
the readed. Thus the adding of accents to a base character builds a
single grapheme from several codepoints.

The grapheme dosn't include 'font' information like which font to use,
the size, additions like bold or italics and such, which add on to make
the final glyph, but does include all the jots and tildes that are part
of the character.

On the other hand, some languages add things like 'vowel points' to
characters, and those are seperate graphemes even though they are added
by a similar manner. This comes down to what the original language
though of as a 'character', which just makes things even more complicated.

Then as you said, there are the 'look-alike' characters which are
considered (generally) to be separate, but some canonicalizations will
convert to a common character.

On Sunday, 9 January 2022 at 17:52:50 UTC, Richard Damon wrote:
>
> On the other hand, some languages add things like 'vowel points' to
> characters, and those are seperate graphemes even though they are added
> by a similar manner. This comes down to what the original language
> though of as a 'character', which just makes things even more complicated.
>
In Hebrew the "vowel points" are optional. They are used in beginners' and
religious texts, but not in general use. So if we take a text from scripture,
and represent it with and without vowels, is that the same text or a different
text? Almost all Hebrew speakers would say "It's the same text". So
strcmp() doesn't necessarily work in a Hebrew context.

On 1/9/22 1:13 PM, Malcolm McLean wrote:
> On Sunday, 9 January 2022 at 17:52:50 UTC, Richard Damon wrote:
>>
>> On the other hand, some languages add things like 'vowel points' to
>> characters, and those are seperate graphemes even though they are added
>> by a similar manner. This comes down to what the original language
>> though of as a 'character', which just makes things even more complicated.
>>
> In Hebrew the "vowel points" are optional. They are used in beginners' and
> religious texts, but not in general use. So if we take a text from scripture,
> and represent it with and without vowels, is that the same text or a different
> text? Almost all Hebrew speakers would say "It's the same text". So
> strcmp() doesn't necessarily work in a Hebrew context.

Hebrew isn't the only language to use 'vowel points', but they also
occur in a number of other languages.

The key point that I was pointing out is that in many of these
languages, to points are NOT considered a part of the letter they are
'attached' to, but a separate letter, even if typographically connected.

On 09/01/2022 18:52, Richard Damon wrote:
>
> On 1/9/22 7:33 AM, David Brown wrote:
>
>> A "grapheme" is a written mark - a letter, punctuation, accent, etc.,
>> that conveys meaning.  Sometimes it is useful to break them down,
>> sometimes it is useful to treat them separately.  For example, "é" can
>> considered as a single grapheme, or as a grapheme "e" followed by a
>> combining graphene "'" acute accent.  The same grapheme can match
>> multiple code points - a Latin alphabet capital A is the same as a Greek
>> alphabet capital Alpha.
>>
>> A "glyph" is a rendering of a grapheme - the letter "A" in different
>> fonts are different glyphs of the same grapheme.
>>
>> What the reader perceives as a "character" is often a single grapheme,
>> but might be several graphemes together.
>>
>
> No, a grapheme, from my understanding, is the character as perceived by
> the readed. Thus the adding of accents to a base character builds a
> single grapheme from several codepoints.

The letter "o" is a grapheme, and an umlaut accent " is a grapheme. The
combination ö may be considered a single grapheme, or a combination of
graphemes. A German reader might consider it two graphemes - an
accented letter "o". A Swedish reader would consider it to be one
grapheme, as "ö" is a distinct letter in Swedish.

>
> The grapheme dosn't include 'font' information like which font to use,
> the size, additions like bold or italics and such, which add on to make
> the final glyph, but does include all the jots and tildes that are part
> of the character.

Correct. These are included in the glyph - the actual ink pattern on
the page.

>
> On the other hand, some languages add things like 'vowel points' to
> characters, and those are seperate graphemes even though they are added
> by a similar manner. This comes down to what the original language
> though of as a 'character', which just makes things even more complicated.
>

Yes, I think that is correct. (And "it's complicated" is /certainly/
correct!)

> Then as you said, there are the 'look-alike' characters which are
> considered (generally) to be separate, but some canonicalizations will
> convert to a common character.

On 09/01/2022 19:24, Richard Damon wrote:
> On 1/9/22 1:13 PM, Malcolm McLean wrote:
>> On Sunday, 9 January 2022 at 17:52:50 UTC, Richard Damon wrote:
>>>
>>> On the other hand, some languages add things like 'vowel points' to
>>> characters, and those are seperate graphemes even though they are added
>>> by a similar manner. This comes down to what the original language
>>> though of as a 'character', which just makes things even more
>>> complicated.
>>>
>> In Hebrew the "vowel points" are optional. They are used in beginners'
>> and
>> religious texts, but not in general use. So if we take a text from
>> scripture,
>> and represent it with and without vowels, is that the same text or a
>> different
>> text? Almost all Hebrew speakers would say "It's the same text". So
>> strcmp() doesn't necessarily work in a Hebrew context.
>
> Hebrew isn't the only language to use 'vowel points', but they also
> occur in a number of other languages.
>
> The key point that I was pointing out is that in many of these
> languages, to points are NOT considered a part of the letter they are
> 'attached' to, but a separate letter, even if typographically connected.

That's true. The opposite is true also - a ligature like fi might be
typographically connected (depending on the font and typesetting),
despite being two independent letters and two characters. Unicode has a
number of code points for such ligatures, but there are many more that
are sometimes used in typography, especially historical documents.
(Unicode is missing an "fj" ligature, for example.)

And while "æ" is a ligature of two letters forming a diphthong (used in
Latin and related languages), it is an independent letter in Norwegian.

On Sunday, 9 January 2022 at 18:34:25 UTC+2, Manfred wrote:
> On 1/9/2022 1:44 PM, Öö Tiib wrote:
> > On Sunday, 9 January 2022 at 03:17:10 UTC+2, Manfred wrote:
> >> On 1/8/2022 11:50 PM, Öö Tiib wrote:
> >>> On Saturday, 8 January 2022 at 20:49:01 UTC+2, james...@alumni.caltech.edu wrote:
> >>>> On Saturday, January 8, 2022 at 11:17:53 AM UTC-5, Öö Tiib wrote:
> >>>>> On Saturday, 8 January 2022 at 06:52:33 UTC+2, james...@alumni.caltech.edu wrote:
....
> >
> >>>>> FILE *f = fopen( "Foo😀Bar.txt", "w");
> >>>>> That should work unless underlying file system does not support files
> >>>>> named "Foo😀Bar.txt" If it supports but the code does not work then it indicates
> >>>>> bad standard that allows implementations to weasel away. No garbage like
> >>>>> u8fopen( u8"Foo😀Bar.txt", "w") coming somewhere maybe in C35 or so is
> >>>>> needed as it already works like in my example on vast majority of things.
> >>>>
> >>>> Nothing in the standard prevents an implementation from doing that. If one doesn't
> >>>> already do so, that's a choice made by the implementors, and you should ask them
> >>>> about it. Your real beef is with the implementors, not the standard.
> >>>
> >>> My beef is with standards. Adding garbage that does not work to standard is wrong
> >>> and not adding what everybody at least half sane does use to standard is also wrong.
> >>>
> >> Also agreed, but since utf-8 is transparent to ascii functions, what
> >> should have been added?
> >
> > Something that makes it clear that it is defect when "Foo😀Bar.txt" is silently opened
> > on file-system that fully supports files named "Foo😀Bar.txt" I suppose.
> >
> Assuming that "Foo≡ƒÿÇBar.txt" "Foo😀Bar..txt" have the same binary
> representation, what's the difference? One form or the other shows up
> only when it is displayed in some UI - the filesystem isn't one, which
> leads to the implementation's runtime behavior.

How you mean same binary representation? Both "Foo😀Bar.txt" and
"Foo😀Bar.txt" files can be in same directory. Both have Unicode
names in underlying file system precisely as posted.

> If they are actually different in their binary sequence, and this is the
> result of the utf-8 string being wrongly converted multiple times, this
> looks like a bad implementation, rather than a problem with the standard.
> IIUC you are advocating for some statement in the standard that prevents
> implementations from messing up with "character sets" in null terminated
> char strings?

I mean that standard should require that all char* texts are treated as
UTF-8 by standard library unless said otherwise. If implementation needs
some other encoding of such byte sequence then it provides
platform-specific functions or compiler switches and/or extends language
with implementation-defined char_iso8859_1_t character types and
prefixes. If it is noteworthy handy type then add it to standards too, I
don't care.

If standard can define that overflow in signed atomics is well defined
and two's complement is mandated there then it also can define that all
char* texts are UTF-8. The only question is if what I suggest is reasonable
or not. From viewpoint of implementer of standard library or users it
is likely blessing ... so I think it is question of business/politics/religions.

On 1/9/22 5:35 PM, Öö Tiib wrote:
> On Sunday, 9 January 2022 at 18:34:25 UTC+2, Manfred wrote:
>> On 1/9/2022 1:44 PM, Öö Tiib wrote:
>>> On Sunday, 9 January 2022 at 03:17:10 UTC+2, Manfred wrote:
>>>> On 1/8/2022 11:50 PM, Öö Tiib wrote:
>>>>> On Saturday, 8 January 2022 at 20:49:01 UTC+2, james...@alumni.caltech.edu wrote:
>>>>>> On Saturday, January 8, 2022 at 11:17:53 AM UTC-5, Öö Tiib wrote:
>>>>>>> On Saturday, 8 January 2022 at 06:52:33 UTC+2, james...@alumni.caltech.edu wrote:
> ...
>>>
>>>>>>> FILE *f = fopen( "Foo😀Bar.txt", "w");
>>>>>>> That should work unless underlying file system does not support files
>>>>>>> named "Foo😀Bar.txt" If it supports but the code does not work then it indicates
>>>>>>> bad standard that allows implementations to weasel away. No garbage like
>>>>>>> u8fopen( u8"Foo😀Bar.txt", "w") coming somewhere maybe in C35 or so is
>>>>>>> needed as it already works like in my example on vast majority of things.
>>>>>>
>>>>>> Nothing in the standard prevents an implementation from doing that. If one doesn't
>>>>>> already do so, that's a choice made by the implementors, and you should ask them
>>>>>> about it. Your real beef is with the implementors, not the standard.
>>>>>
>>>>> My beef is with standards. Adding garbage that does not work to standard is wrong
>>>>> and not adding what everybody at least half sane does use to standard is also wrong.
>>>>>
>>>> Also agreed, but since utf-8 is transparent to ascii functions, what
>>>> should have been added?
>>>
>>> Something that makes it clear that it is defect when "Foo😀Bar.txt" is silently opened
>>> on file-system that fully supports files named "Foo😀Bar.txt" I suppose.
>>>
>> Assuming that "Foo≡ƒÿÇBar.txt" "Foo😀Bar.txt" have the same binary
>> representation, what's the difference? One form or the other shows up
>> only when it is displayed in some UI - the filesystem isn't one, which
>> leads to the implementation's runtime behavior.
>
> How you mean same binary representation? Both "Foo😀Bar.txt" and
> "Foo😀Bar.txt" files can be in same directory. Both have Unicode
> names in underlying file system precisely as posted.
>
>> If they are actually different in their binary sequence, and this is the
>> result of the utf-8 string being wrongly converted multiple times, this
>> looks like a bad implementation, rather than a problem with the standard.
>> IIUC you are advocating for some statement in the standard that prevents
>> implementations from messing up with "character sets" in null terminated
>> char strings?
>
> I mean that standard should require that all char* texts are treated as
> UTF-8 by standard library unless said otherwise. If implementation needs
> some other encoding of such byte sequence then it provides
> platform-specific functions or compiler switches and/or extends language
> with implementation-defined char_iso8859_1_t character types and
> prefixes. If it is noteworthy handy type then add it to standards too, I
> don't care.
>
> If standard can define that overflow in signed atomics is well defined
> and two's complement is mandated there then it also can define that all
> char* texts are UTF-8. The only question is if what I suggest is reasonable
> or not. From viewpoint of implementer of standard library or users it
> is likely blessing ... so I think it is question of business/politics/religions.

The difference is that in these days, the existence of computers that
aren't going to be able to support two's complement that will still want
to support modern 'C' is effectively non-existent.

The existance of machines that might still want to be able to support
non-UTF-8 strings is not.

Perhaps the biggest is the embedded market where needing to support
beyond plain ASCII isn't needed, and DEFINING that strings will follow
UTF-8 rules adds a LOT of complications for some operations that just
aren't needed on many of the systems.

The Standard does ALLOW a system to define char to be UTF-8 (at least
until you get into issues of what it requires for wide characters).

On Monday, 10 January 2022 at 01:49:07 UTC+2, Richard Damon wrote:
> On 1/9/22 5:35 PM, Öö Tiib wrote:
> > On Sunday, 9 January 2022 at 18:34:25 UTC+2, Manfred wrote:
> >> On 1/9/2022 1:44 PM, Öö Tiib wrote:
> >>> On Sunday, 9 January 2022 at 03:17:10 UTC+2, Manfred wrote:
> >>>> On 1/8/2022 11:50 PM, Öö Tiib wrote:
> >>>>> On Saturday, 8 January 2022 at 20:49:01 UTC+2, james...@alumni.caltech.edu wrote:
> >>>>>> On Saturday, January 8, 2022 at 11:17:53 AM UTC-5, Öö Tiib wrote:
> >>>>>>> On Saturday, 8 January 2022 at 06:52:33 UTC+2, james...@alumni.caltech.edu wrote:
> > ...
> >>>
> >>>>>>> FILE *f = fopen( "Foo😀Bar.txt", "w");
> >>>>>>> That should work unless underlying file system does not support files
> >>>>>>> named "Foo😀Bar.txt" If it supports but the code does not work then it indicates
> >>>>>>> bad standard that allows implementations to weasel away. No garbage like
> >>>>>>> u8fopen( u8"Foo😀Bar.txt", "w") coming somewhere maybe in C35 or so is
> >>>>>>> needed as it already works like in my example on vast majority of things.
> >>>>>>
> >>>>>> Nothing in the standard prevents an implementation from doing that.. If one doesn't
> >>>>>> already do so, that's a choice made by the implementors, and you should ask them
> >>>>>> about it. Your real beef is with the implementors, not the standard.
> >>>>>
> >>>>> My beef is with standards. Adding garbage that does not work to standard is wrong
> >>>>> and not adding what everybody at least half sane does use to standard is also wrong.
> >>>>>
> >>>> Also agreed, but since utf-8 is transparent to ascii functions, what
> >>>> should have been added?
> >>>
> >>> Something that makes it clear that it is defect when "Foo😀Bar.txt" is silently opened
> >>> on file-system that fully supports files named "Foo😀Bar.txt" I suppose.
> >>>
> >> Assuming that "Foo≡ƒÿÇBar.txt" "Foo😀Bar.txt" have the same binary
> >> representation, what's the difference? One form or the other shows up
> >> only when it is displayed in some UI - the filesystem isn't one, which
> >> leads to the implementation's runtime behavior.
> >
> > How you mean same binary representation? Both "Foo😀Bar.txt" and
> > "Foo😀Bar.txt" files can be in same directory. Both have Unicode
> > names in underlying file system precisely as posted.
> >
> >> If they are actually different in their binary sequence, and this is the
> >> result of the utf-8 string being wrongly converted multiple times, this
> >> looks like a bad implementation, rather than a problem with the standard.
> >> IIUC you are advocating for some statement in the standard that prevents
> >> implementations from messing up with "character sets" in null terminated
> >> char strings?
> >
> > I mean that standard should require that all char* texts are treated as
> > UTF-8 by standard library unless said otherwise. If implementation needs
> > some other encoding of such byte sequence then it provides
> > platform-specific functions or compiler switches and/or extends language
> > with implementation-defined char_iso8859_1_t character types and
> > prefixes. If it is noteworthy handy type then add it to standards too, I
> > don't care.
> >
> > If standard can define that overflow in signed atomics is well defined
> > and two's complement is mandated there then it also can define that all
> > char* texts are UTF-8. The only question is if what I suggest is reasonable
> > or not. From viewpoint of implementer of standard library or users it
> > is likely blessing ... so I think it is question of business/politics/religions.
>
> The difference is that in these days, the existence of computers that
> aren't going to be able to support two's complement that will still want
> to support modern 'C' is effectively non-existent.
>
> The existance of machines that might still want to be able to support
> non-UTF-8 strings is not.

But do there exist machines that do want to support texts as char* but
do not want to support UTF-8? Describe those machines, give examples.

> Perhaps the biggest is the embedded market where needing to support

You mean tiny things like SD cards or flash sticks? I can store there
"Foo😀Bar.txt" just fine. Either embedded system does not need to
communicate in char* text at all, can fully ignore its encoding or has
to deal with UTF-8 anyway. I know of no other examples, despite I've
participated in programming whole pile of embedded systems over
the decades.

> beyond plain ASCII isn't needed, and DEFINING that strings will follow
> UTF-8 rules adds a LOT of complications for some operations that just
> aren't needed on many of the systems.

WHAT complications? Give examples? Both ASCII and UTF-8 are row of
bytes that end with zero. ASCII is proper subset of UTF-8. Tell about
use-case where UTF-8 hurts? Human languages and typography
are horribly complicated but UTF-8 is genially trivial. Either embedded
system does not do linguistic analyses of poems or if it does then
it needs to use Unicode anyway. But commonly if it can't display
something then it shows � and done.
> The Standard does ALLOW a system to define char to be UTF-8 (at least
> until you get into issues of what it requires for wide characters).

Allowing is apparently not enough as the support rots in standards.
Wide characters are wchar_t, char16_t and char32_t. These are
in horrible state too but I ignore it for now. Not related to issues
with char* and far less important in industry.

On 1/9/2022 11:35 PM, Öö Tiib wrote:
> On Sunday, 9 January 2022 at 18:34:25 UTC+2, Manfred wrote:
>> On 1/9/2022 1:44 PM, Öö Tiib wrote:
>>> On Sunday, 9 January 2022 at 03:17:10 UTC+2, Manfred wrote:
>>>> On 1/8/2022 11:50 PM, Öö Tiib wrote:
>>>>> On Saturday, 8 January 2022 at 20:49:01 UTC+2, james...@alumni.caltech.edu wrote:
>>>>>> On Saturday, January 8, 2022 at 11:17:53 AM UTC-5, Öö Tiib wrote:
>>>>>>> On Saturday, 8 January 2022 at 06:52:33 UTC+2, james...@alumni.caltech.edu wrote:
> ...
>>>
>>>>>>> FILE *f = fopen( "Foo😀Bar.txt", "w");
>>>>>>> That should work unless underlying file system does not support files
>>>>>>> named "Foo😀Bar.txt" If it supports but the code does not work then it indicates
>>>>>>> bad standard that allows implementations to weasel away. No garbage like
>>>>>>> u8fopen( u8"Foo😀Bar.txt", "w") coming somewhere maybe in C35 or so is
>>>>>>> needed as it already works like in my example on vast majority of things.
>>>>>>
>>>>>> Nothing in the standard prevents an implementation from doing that. If one doesn't
>>>>>> already do so, that's a choice made by the implementors, and you should ask them
>>>>>> about it. Your real beef is with the implementors, not the standard.
>>>>>
>>>>> My beef is with standards. Adding garbage that does not work to standard is wrong
>>>>> and not adding what everybody at least half sane does use to standard is also wrong.
>>>>>
>>>> Also agreed, but since utf-8 is transparent to ascii functions, what
>>>> should have been added?
>>>
>>> Something that makes it clear that it is defect when "Foo😀Bar.txt" is silently opened
>>> on file-system that fully supports files named "Foo😀Bar.txt" I suppose.
>>>
>> Assuming that "Foo≡ƒÿÇBar.txt" "Foo😀Bar.txt" have the same binary
>> representation, what's the difference? One form or the other shows up
>> only when it is displayed in some UI - the filesystem isn't one, which
>> leads to the implementation's runtime behavior.
>
> How you mean same binary representation? Both "Foo😀Bar.txt" and
> "Foo😀Bar.txt" files can be in same directory. Both have Unicode
> names in underlying file system precisely as posted.

I mean the same byte sequence in their name, but different UI
representation, e.g. when decoded as utf-8 or w-1252 or whatever.

What you are saying assumes a Unicode-aware filesystem, that's not free
from the point of view of the standard.
But, in order to support utf-8, it would be enough to have a char based
filesystem that treats names as plain 0-terminated char[]. That's
easier, probably free on most platforms, but it's different from
Unicode-aware (which could be UTF-16 like Windows, and there you have
your problems).

>
>> If they are actually different in their binary sequence, and this is the
>> result of the utf-8 string being wrongly converted multiple times, this
>> looks like a bad implementation, rather than a problem with the standard.
>> IIUC you are advocating for some statement in the standard that prevents
>> implementations from messing up with "character sets" in null terminated
>> char strings?
>
> I mean that standard should require that all char* texts are treated as
> UTF-8 by standard library unless said otherwise. If implementation needs
> some other encoding of such byte sequence then it provides
> platform-specific functions or compiler switches and/or extends language
> with implementation-defined char_iso8859_1_t character types and
> prefixes. If it is noteworthy handy type then add it to standards too, I
> don't care.

I see this hard to win, and probably not ideal - suppose in 10 years
some better encoding than utf-8 shows up, then you are screwed again.

I'd rather stick to the fact that utf-8 is compatible with 0-terminated
char[], and so a plausible wish would be that such strings are not
screwed by the implementation; for example when you store a file name in
a filesystem with fopen() and the name is given as char[], then the
standard could mandate that reading back that same name as char[] gives
back the same byte sequence.

Currently I guess one could use a utf-8 string as a name to fopen() on
Windows, then the OS assumes it is W-1252 and converts it into UTF-16,
at which point it is screwed, and when you read it back into char[] it
is garbage.

>
> If standard can define that overflow in signed atomics is well defined
> and two's complement is mandated there then it also can define that all
> char* texts are UTF-8. The only question is if what I suggest is reasonable
> or not. From viewpoint of implementer of standard library or users it
> is likely blessing ... so I think it is question of business/politics/religions.

I agree with Richard here. Two's complement is not like utf-8.
I still think it's technical rather than business/politics/religions in
this case - as I said above I'm not sure it would even be ideal.

On 1/9/22 7:51 PM, Öö Tiib wrote:
> On Monday, 10 January 2022 at 01:49:07 UTC+2, Richard Damon wrote:
>> On 1/9/22 5:35 PM, Öö Tiib wrote:
>>> On Sunday, 9 January 2022 at 18:34:25 UTC+2, Manfred wrote:
>>>> On 1/9/2022 1:44 PM, Öö Tiib wrote:
>>>>> On Sunday, 9 January 2022 at 03:17:10 UTC+2, Manfred wrote:
>>>>>> On 1/8/2022 11:50 PM, Öö Tiib wrote:
>>>>>>> On Saturday, 8 January 2022 at 20:49:01 UTC+2, james...@alumni.caltech.edu wrote:
>>>>>>>> On Saturday, January 8, 2022 at 11:17:53 AM UTC-5, Öö Tiib wrote:
>>>>>>>>> On Saturday, 8 January 2022 at 06:52:33 UTC+2, james...@alumni.caltech.edu wrote:
>>> ...
>>>>>
>>>>>>>>> FILE *f = fopen( "Foo😀Bar.txt", "w");
>>>>>>>>> That should work unless underlying file system does not support files
>>>>>>>>> named "Foo😀Bar.txt" If it supports but the code does not work then it indicates
>>>>>>>>> bad standard that allows implementations to weasel away. No garbage like
>>>>>>>>> u8fopen( u8"Foo😀Bar.txt", "w") coming somewhere maybe in C35 or so is
>>>>>>>>> needed as it already works like in my example on vast majority of things.
>>>>>>>>
>>>>>>>> Nothing in the standard prevents an implementation from doing that. If one doesn't
>>>>>>>> already do so, that's a choice made by the implementors, and you should ask them
>>>>>>>> about it. Your real beef is with the implementors, not the standard.
>>>>>>>
>>>>>>> My beef is with standards. Adding garbage that does not work to standard is wrong
>>>>>>> and not adding what everybody at least half sane does use to standard is also wrong.
>>>>>>>
>>>>>> Also agreed, but since utf-8 is transparent to ascii functions, what
>>>>>> should have been added?
>>>>>
>>>>> Something that makes it clear that it is defect when "Foo😀Bar.txt" is silently opened
>>>>> on file-system that fully supports files named "Foo😀Bar.txt" I suppose.
>>>>>
>>>> Assuming that "Foo≡ƒÿÇBar.txt" "Foo😀Bar.txt" have the same binary
>>>> representation, what's the difference? One form or the other shows up
>>>> only when it is displayed in some UI - the filesystem isn't one, which
>>>> leads to the implementation's runtime behavior.
>>>
>>> How you mean same binary representation? Both "Foo😀Bar.txt" and
>>> "Foo😀Bar.txt" files can be in same directory. Both have Unicode
>>> names in underlying file system precisely as posted.
>>>
>>>> If they are actually different in their binary sequence, and this is the
>>>> result of the utf-8 string being wrongly converted multiple times, this
>>>> looks like a bad implementation, rather than a problem with the standard.
>>>> IIUC you are advocating for some statement in the standard that prevents
>>>> implementations from messing up with "character sets" in null terminated
>>>> char strings?
>>>
>>> I mean that standard should require that all char* texts are treated as
>>> UTF-8 by standard library unless said otherwise. If implementation needs
>>> some other encoding of such byte sequence then it provides
>>> platform-specific functions or compiler switches and/or extends language
>>> with implementation-defined char_iso8859_1_t character types and
>>> prefixes. If it is noteworthy handy type then add it to standards too, I
>>> don't care.
>>>
>>> If standard can define that overflow in signed atomics is well defined
>>> and two's complement is mandated there then it also can define that all
>>> char* texts are UTF-8. The only question is if what I suggest is reasonable
>>> or not. From viewpoint of implementer of standard library or users it
>>> is likely blessing ... so I think it is question of business/politics/religions.
>>
>> The difference is that in these days, the existence of computers that
>> aren't going to be able to support two's complement that will still want
>> to support modern 'C' is effectively non-existent.
>>
>> The existance of machines that might still want to be able to support
>> non-UTF-8 strings is not.
>
> But do there exist machines that do want to support texts as char* but
> do not want to support UTF-8? Describe those machines, give examples.

Small embedded micros with no need for large character sets.

>
>> Perhaps the biggest is the embedded market where needing to support
>
> You mean tiny things like SD cards or flash sticks? I can store there
> "Foo😀Bar.txt" just fine. Either embedded system does not need to
> communicate in char* text at all, can fully ignore its encoding or has
> to deal with UTF-8 anyway. I know of no other examples, despite I've
> participated in programming whole pile of embedded systems over
> the decades.
>

Many such system communicate in command strings, maybe even with a
minimal TCP/IP but have no need for processing data beyond pure ASCII.

>> beyond plain ASCII isn't needed, and DEFINING that strings will follow
>> UTF-8 rules adds a LOT of complications for some operations that just
>> aren't needed on many of the systems.
>
> WHAT complications? Give examples? Both ASCII and UTF-8 are row of
> bytes that end with zero. ASCII is proper subset of UTF-8. Tell about
> use-case where UTF-8 hurts? Human languages and typography
> are horribly complicated but UTF-8 is genially trivial. Either embedded
> system does not do linguistic analyses of poems or if it does then
> it needs to use Unicode anyway. But commonly if it can't display
> something then it shows � and done.

Once you have your char as being defined as a Multi-Byte Character Set,
then wchar_t must be big enough to hold any of them. If you just support
ASCII, then wchar_t can be just 8 Bits if you want, and (almost?) all
the wchar_t stuff can just be alias for the char stuff.

Thus forcing char to be UTF-8 adds a lot of complexity to the system.

>
>> The Standard does ALLOW a system to define char to be UTF-8 (at least
>> until you get into issues of what it requires for wide characters).
>
> Allowing is apparently not enough as the support rots in standards.
> Wide characters are wchar_t, char16_t and char32_t. These are
> in horrible state too but I ignore it for now. Not related to issues
> with char* and far less important in industry.
>

But that is part of the problem with supporting UTF-8, as that by
definiition brings in all the wide character issues into play.

If you define that your character set is ASCII, then wchar_t becomes
trivial.

A big part of the issue with char16_t is that it is fundamentally broken
with Unicode, but lives on due to trying to maintain the backwards
bandaids that basically can't be removed without admitting that a large
segment of code just will live as being openly non-complient.

Too much legacy code assumes that 16 bit characters are 'big enough' for
most people, and pretty much do work if you aren't being a stickler for
full conformance to the rules, which no one is because you can't be.