"Some sanity for C and C++ development on Windows" by Chris Wellons
https://nullprogram.com/blog/2021/12/30/

Lynn

On Tuesday, 4 January 2022 at 08:36:13 UTC+2, Lynn McGuire wrote:
> "Some sanity for C and C++ development on Windows" by Chris Wellons
> https://nullprogram.com/blog/2021/12/30/

The whole difference that std::string on other platforms is UTF-8.
It is something that standard of C or C++ do not support in any way.
On the contrary, the standards add obfuscation bullshit like:

const char crap[] = u8"Öö Tiib 😀";

And when to ask why then oh but maybe there is EBCDIK character
set. Shove that ebc-dick where sun doesn't shine, morons.
Let MS add its w1252 prefix if it likes that character set too lot?
But no language lawyer in committee does have balls, and there it
ends.

On 04/01/2022 18:39, Öö Tiib wrote:
> On Tuesday, 4 January 2022 at 08:36:13 UTC+2, Lynn McGuire wrote:
>> "Some sanity for C and C++ development on Windows" by Chris Wellons
>> https://nullprogram.com/blog/2021/12/30/
>
> The whole difference that std::string on other platforms is UTF-8.
> It is something that standard of C or C++ do not support in any way.
> On the contrary, the standards add obfuscation bullshit like:
>
> const char crap[] = u8"Öö Tiib 😀";
>
> And when to ask why then oh but maybe there is EBCDIK character
> set. Shove that ebc-dick where sun doesn't shine, morons.
> Let MS add its w1252 prefix if it likes that character set too lot?
> But no language lawyer in committee does have balls, and there it
> ends.
>
My first job was writing assembler on a mainframe with an EBCDIC
character set. The operating system I worked on has been dead for 40
years now, but I dare say IBM's mainframes still use it.

Andy

Vir Campestris <vir.campestris@invalid.invalid> writes:
>On 04/01/2022 18:39, Öö Tiib wrote:
>> On Tuesday, 4 January 2022 at 08:36:13 UTC+2, Lynn McGuire wrote:
>>> "Some sanity for C and C++ development on Windows" by Chris Wellons
>>> https://nullprogram.com/blog/2021/12/30/
>>
>> The whole difference that std::string on other platforms is UTF-8.
>> It is something that standard of C or C++ do not support in any way.
>> On the contrary, the standards add obfuscation bullshit like:
>>
>> const char crap[] = u8"Öö Tiib 😀";
>>
>> And when to ask why then oh but maybe there is EBCDIK character
>> set. Shove that ebc-dick where sun doesn't shine, morons.
>> Let MS add its w1252 prefix if it likes that character set too lot?
>> But no language lawyer in committee does have balls, and there it
>> ends.
>>
>My first job was writing assembler on a mainframe with an EBCDIC
>character set. The operating system I worked on has been dead for 40
>years now, but I dare say IBM's mainframes still use it.

The Unisys mainframes (from the Burroughs side) still use EBCDIC;
I believe the sperry side systems support EBCDIC, if not use it natively.

On 2022-01-05, Scott Lurndal <scott@slp53.sl.home> wrote:
> Vir Campestris <vir.campestris@invalid.invalid> writes:
>>My first job was writing assembler on a mainframe with an EBCDIC
>>character set. The operating system I worked on has been dead for 40
>>years now, but I dare say IBM's mainframes still use it.
>
> The Unisys mainframes (from the Burroughs side) still use EBCDIC;
> I believe the sperry side systems support EBCDIC, if not use it natively.

My EBCDIC experience is that over the years, I had at least one
manager who was one.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal

On Thursday, 6 January 2022 at 00:57:09 UTC+2, Scott Lurndal wrote:
> Vir Campestris <vir.cam...@invalid.invalid> writes:
> >On 04/01/2022 18:39, Öö Tiib wrote:
> >> On Tuesday, 4 January 2022 at 08:36:13 UTC+2, Lynn McGuire wrote:
> >>> "Some sanity for C and C++ development on Windows" by Chris Wellons
> >>> https://nullprogram.com/blog/2021/12/30/
> >>
> >> The whole difference that std::string on other platforms is UTF-8.
> >> It is something that standard of C or C++ do not support in any way.
> >> On the contrary, the standards add obfuscation bullshit like:
> >>
> >> const char crap[] = u8"Öö Tiib 😀";
> >>
> >> And when to ask why then oh but maybe there is EBCDIK character
> >> set. Shove that ebc-dick where sun doesn't shine, morons.
> >> Let MS add its w1252 prefix if it likes that character set too lot?
> >> But no language lawyer in committee does have balls, and there it
> >> ends.
> >>
> >My first job was writing assembler on a mainframe with an EBCDIC
> >character set. The operating system I worked on has been dead for 40
> >years now, but I dare say IBM's mainframes still use it.
>
> The Unisys mainframes (from the Burroughs side) still use EBCDIC;
> I believe the sperry side systems support EBCDIC, if not use it natively.

Even programmer of such system might want to upgrade to compiler
whose standard library supports UTF-8. But it is not possible as
standard library is defined not to.

=?UTF-8?B?w5bDtiBUaWli?= <ootiib@hot.ee> writes:
>On Thursday, 6 January 2022 at 00:57:09 UTC+2, Scott Lurndal wrote:
>> Vir Campestris <vir.cam...@invalid.invalid> writes:=20
>> >On 04/01/2022 18:39, =C3=96=C3=B6 Tiib wrote:=20
>> >> On Tuesday, 4 January 2022 at 08:36:13 UTC+2, Lynn McGuire wrote:=20
>> >>> "Some sanity for C and C++ development on Windows" by Chris Wellons=
>=20
>> >>> https://nullprogram.com/blog/2021/12/30/=20
>> >>=20
>> >> The whole difference that std::string on other platforms is UTF-8.=20
>> >> It is something that standard of C or C++ do not support in any way.=
>=20
>> >> On the contrary, the standards add obfuscation bullshit like:=20
>> >>=20
>> >> const char crap[] =3D u8"=C3=96=C3=B6 Tiib =F0=9F=98=80";=20
>> >>=20
>> >> And when to ask why then oh but maybe there is EBCDIK character=20
>> >> set. Shove that ebc-dick where sun doesn't shine, morons.=20
>> >> Let MS add its w1252 prefix if it likes that character set too lot?=20
>> >> But no language lawyer in committee does have balls, and there it=20
>> >> ends.=20
>> >>=20
>> >My first job was writing assembler on a mainframe with an EBCDIC=20
>> >character set. The operating system I worked on has been dead for 40=20
>> >years now, but I dare say IBM's mainframes still use it.
>>=20
>> The Unisys mainframes (from the Burroughs side) still use EBCDIC;=20
>> I believe the sperry side systems support EBCDIC, if not use it natively.
>
>Even programmer of such system might want to upgrade to compiler
>whose standard library supports UTF-8. But it is not possible as=20
>standard library is defined not to. =20

Well, the Burroughs programmers use COBOL and ALGOL, not C, which
have supported I18N and L10N since the 1980s.

On Wednesday, January 5, 2022 at 11:35:38 PM UTC-5, Öö Tiib wrote:
> On Thursday, 6 January 2022 at 00:57:09 UTC+2, Scott Lurndal wrote:
....
> > The Unisys mainframes (from the Burroughs side) still use EBCDIC;
> > I believe the sperry side systems support EBCDIC, if not use it natively.
> Even programmer of such system might want to upgrade to compiler
> whose standard library supports UTF-8. But it is not possible as
> standard library is defined not to.

Could you cite the text from the C and C++ standards that prohibits the
standard library from supporting UTF-8?
Are you saying that it's prohibited specifically on platforms that normally
use EBCDIC? I'm not aware of any requirement that an implementation of
C follow the conventions for the target platform: an implementation that
emulates working on a completely different platform (such as one where
UTF-8 is the norm) is always allowed.

To the best of my understanding:
* The encoding of source files and the execution character narrow
character set are both implementation-defined multibyte encodings, and
there's nothing that prohibits either encoding from being UTF-8.
* Implementations are explicitly permitted to allow extended characters in
source files for identifiers, string literals, character constants, comments
and header names.
* The current versions of both standards allow string literals and character
constants prefixed with u8, which are required to have UTF-8 encoding.
* Even the latest version of C doesn't mandate any conversion routines for
UTF-8, but on a platform which makes UTF-8 the encoding for it's
execution character set, the conversion routines that contain "mb" in their
name will interpret plain char as having UTF-8 encoding.
* C++ mandates mbrtoc8() and c8rtomb(), which convert between the
native encoding and UTF-8. It also mandates mbrtowc() and wcrtomb(),
as well as the C standard library routines for convertiong betwen the
native encodings and char16_t or char32_t in <uchar>; the C standard does
not require that those types have UTF-16 and UTF-32 encoding
respectively, but the C++ standard does. It also mandates codecvt facets
for converting between UTF-8, UTF-16, and UTF-32, so conversion between
UTF-8 and any of the other four character encodings are mandated,
though conversion to and from wchar_t is a two-step process unless the
native narrow character set uses UTF-8 encoding.

While C, in particular, doesn't mandate quite as much support for UTF-8 as
I'd like, both standards allow the fullest possible support for UTF-8 that I
could imagine. Why do you think otherwise?

Le 06/01/2022 à 18:28, james...@alumni.caltech.edu a écrit :
> While C, in particular, doesn't mandate quite as much support for UTF-8 as
> I'd like, both standards allow the fullest possible support for UTF-8 that I
> could imagine. Why do you think otherwise?

Agreed.

And yes, as support is not that complete, we usually have to use
third-party (or our own) libraries just for that. And to be fair, UTF-8
support, overall, is still a bit shaky.

Sure on Windows, where MS had focused on using Unicode UCS2 instead of
UTF-8, things are no better, even if you use the Windows API instead of
the C standard library.

On Thursday, 6 January 2022 at 19:28:46 UTC+2, james...@alumni.caltech.edu wrote:
> On Wednesday, January 5, 2022 at 11:35:38 PM UTC-5, Öö Tiib wrote:
> > On Thursday, 6 January 2022 at 00:57:09 UTC+2, Scott Lurndal wrote:
> ...
> > > The Unisys mainframes (from the Burroughs side) still use EBCDIC;
> > > I believe the sperry side systems support EBCDIC, if not use it natively.
> > Even programmer of such system might want to upgrade to compiler
> > whose standard library supports UTF-8. But it is not possible as
> > standard library is defined not to.
> Could you cite the text from the C and C++ standards that prohibits the
> standard library from supporting UTF-8?

Are you pretending that you did not understand what I meant?
AFAIK you have rather decent knowledge of standards. The
standards allow implementations to have wide array of whatever
obscure extensions.

However some essential things, like say adding 128 bit integers or
UTF-8 support or even to stop that nonsense with newline characters
is made tricky. Plus it is obscured with random half backed extensions,
prefixes and types that promise not much, tend to be deprecated and
confuse people. That in world where 98% of plain text in internet is UTF-8..

> Are you saying that it's prohibited specifically on platforms that normally
> use EBCDIC? I'm not aware of any requirement that an implementation of
> C follow the conventions for the target platform: an implementation that
> emulates working on a completely different platform (such as one where
> UTF-8 is the norm) is always allowed.

I am saying that when to ask why default string can't be UTF-8 then that
EBCDIC is usually mentioned. Despite there probably are no much C, let
alone C++ used on few alive EBCDIC platforms.

>
> To the best of my understanding:
> * The encoding of source files and the execution character narrow
> character set are both implementation-defined multibyte encodings, and
> there's nothing that prohibits either encoding from being UTF-8.
> * Implementations are explicitly permitted to allow extended characters in
> source files for identifiers, string literals, character constants, comments
> and header names.
> * The current versions of both standards allow string literals and character
> constants prefixed with u8, which are required to have UTF-8 encoding.
> * Even the latest version of C doesn't mandate any conversion routines for
> UTF-8, but on a platform which makes UTF-8 the encoding for it's
> execution character set, the conversion routines that contain "mb" in their
> name will interpret plain char as having UTF-8 encoding.
> * C++ mandates mbrtoc8() and c8rtomb(), which convert between the
> native encoding and UTF-8. It also mandates mbrtowc() and wcrtomb(),
> as well as the C standard library routines for convertiong betwen the
> native encodings and char16_t or char32_t in <uchar>; the C standard does
> not require that those types have UTF-16 and UTF-32 encoding
> respectively, but the C++ standard does.

The UTF-16 and UTF-32 are also among those 2% of obscure text encodings.
OK, UTF-16 can be useful to communicate with mis-designed programming
languages like Java or C# or operating systems like Windows. But UTF-32
is rather exotic.
> It also mandates codecvt facets
> for converting between UTF-8, UTF-16, and UTF-32, so conversion between
> UTF-8 and any of the other four character encodings are mandated,
> though conversion to and from wchar_t is a two-step process unless the
> native narrow character set uses UTF-8 encoding.

The most useful of those facets that converted to-from UTF-8 in char
array were deprecated by C++17.

> While C, in particular, doesn't mandate quite as much support for UTF-8 as
> I'd like, both standards allow the fullest possible support for UTF-8 that I
> could imagine. Why do you think otherwise?

I think that having all text streams and plain non-prefixed "strings" as
UTF-8 is both possible and most logical. UTF-8 code unit is guaranteed
to fit into char by both standards so it is possible. UTF-8 is most
widespread text format so it is logical. Other, obscure encodings like
Windows1252 or EBCDIC (and functions using or filling those) should
have weirdo prefixes and special character types and what not.
Why do you think otherwise?

On Friday, 7 January 2022 at 12:06:26 UTC, Öö Tiib wrote:
>
> OK, UTF-16 can be useful to communicate with mis-designed programming
> languages like Java or C# or operating systems like Windows. But UTF-32
> is rather exotic.
>
Yes and no. You can pass strings about as utf-8. But it's hard to manipulate them.
Often it's easier to convert to utf-32 and back to actually access the content of
a string and use it.

On Friday, January 7, 2022 at 7:06:26 AM UTC-5, Öö Tiib wrote:
> On Thursday, 6 January 2022 at 19:28:46 UTC+2, james...@alumni.caltech.edu wrote:
> > On Wednesday, January 5, 2022 at 11:35:38 PM UTC-5, Öö Tiib wrote:
....
> > > Even programmer of such system might want to upgrade to compiler
> > > whose standard library supports UTF-8. But it is not possible as
> > > standard library is defined not to.
> > Could you cite the text from the C and C++ standards that prohibits the
> > standard library from supporting UTF-8?
> Are you pretending that you did not understand what I meant?

No, I am quite accurately and honestly expressing my confusion. You object to
something being prohibited by the standards that is, to the best of my understanding,
allowed. It would make more sense if you were objecting the fact that it isn't
mandatory, and if you were making such claims, I would disagree with you about
whether it would be a good idea to make it mandatory - but as far as I can tell, you're
claiming it isn't allowed.
If you could, as requested, cite the relevant text that prohibits such compilers, you
might convince me that I'm wrong. If not, the citation would at least enable me to try
to convince you that you're misinterpreting that text. Neither possibility can happen
until you actually honor that request.

....
> However some essential things, like say adding 128 bit integers or
> UTF-8 support or even to stop that nonsense with newline characters
> is made tricky.

Neither 128 bits integer nor UTF-8 support are essential. Lots of people have no
need of either (I've never needed either one, though I have used UTF-8 since it was
available). If you need such things on a platform where no existing implementation
of C provides them, your complaint is with the implementors, not the standard,
because the standard says nothing to prohibit such things.

....
> I am saying that when to ask why default string can't be UTF-8 then that
> EBCDIC is usually mentioned. Despite there probably are no much C, let
> alone C++ used on few alive EBCDIC platforms.

That seems reasonable to me. The only places where that logic applies are
implementations of C targeting EBCDIC platforms, and regardless of how rare such
implementations, they would become substantially rarer because users would abandon
them if they switched to UTF-8.

....
> The UTF-16 and UTF-32 are also among those 2% of obscure text encodings.
> OK, UTF-16 can be useful to communicate with mis-designed programming
> languages like Java or C# or operating systems like Windows. But UTF-32
> is rather exotic.

UTF-16 is, as I understand it, the default in the Windows world. I'm not that familiar
with it, I've only done a few years of programming targeting that platform, and all of
the text that came up in the work I was doing there was simple English, with no need
to make any use of extended characters, so I can't vouch for any of the details about
how text was encoded. However, WIndows is a very common platform, whether or not
you approve of it's design (I share your disapproval for it), so calling UTF-16 obscure
seems very odd.

UTF-32 shares an important property with the implementation-defined encoding used
for wchar_t: every character takes up one and only one element in the array.. I have
written a lot of code over several decades for parsing strings that assumes that every
character takes up one char, a valid assumption in the contexts in which I wrote it. When
I think about how I would have to re-write such code to work with multi-byte encodings
such as UTF-8, then the simplicity of replacing char with wchar_t or char32_t, leaving my
logic unchanged, starts looking pretty attractive. However, since I have little experience
writing code to work with extended characters using any encoding, my preferences don't
carry much weight.

> > While C, in particular, doesn't mandate quite as much support for UTF-8 as
> > I'd like, both standards allow the fullest possible support for UTF-8 that I
> > could imagine. Why do you think otherwise?
> I think that having all text streams and plain non-prefixed "strings" as
> UTF-8 is both possible and most logical.

Yes, and that is allowed by both standards, and is the norm, not the exception, on most
Unix-like platforms that I'm familiar with. That's why your claim that it's not allowed
confuses me. If you don't work on Unix systems where the encoding of non-prefixed
strings is UTF-8, and you don't work on Windows systems where UTF-16 is the norm, and
you don't work on systems where EBCDIC is the norm, what kinds of systems do you
work on? I'm not saying that there aren't any other types of systems, there's a great
many, but most of the others are substantially less common, so I am just curious which
one(s) you use.

On Friday, 7 January 2022 at 17:34:16 UTC, james...@alumni.caltech.edu wrote:
>
> UTF-16 is, as I understand it, the default in the Windows world. I'm not that familiar
> with it, I've only done a few years of programming targeting that platform, and all of
> the text that came up in the work I was doing there was simple English, with no need
> to make any use of extended characters, so I can't vouch for any of the details about
> how text was encoded. However, WIndows is a very common platform, whether or not
> you approve of it's design (I share your disapproval for it), so calling UTF-16 obscure
> seems very odd.
>
Every Windows API call that takes text comes in an A-suffix or a W-suffix call.
The A-suffix takes ascii strings, the W-suffix takes near UTF-16, actually Microsoft's
slightly incompatible version. If you don't provide a suffix at all, the compiler
selects a version which depends on how you have set it up. I'm not sure what the
default is or exactly how you play with the settings.
>
> > I think that having all text streams and plain non-prefixed "strings" as
> > UTF-8 is both possible and most logical.
> Yes, and that is allowed by both standards, and is the norm, not the exception, on most
> Unix-like platforms that I'm familiar with. That's why your claim that it's not allowed
> confuses me.

My experience is that passing utf-8 to printf() or fopen() doesn't work. But I rarely
need to do so, and the situation might have changed recently.

On 07/01/2022 18:44, Malcolm McLean wrote:
> On Friday, 7 January 2022 at 17:34:16 UTC, james...@alumni.caltech.edu wrote:
>>
>> UTF-16 is, as I understand it, the default in the Windows world. I'm not that familiar
>> with it, I've only done a few years of programming targeting that platform, and all of
>> the text that came up in the work I was doing there was simple English, with no need
>> to make any use of extended characters, so I can't vouch for any of the details about
>> how text was encoded. However, WIndows is a very common platform, whether or not
>> you approve of it's design (I share your disapproval for it), so calling UTF-16 obscure
>> seems very odd.
>>
> Every Windows API call that takes text comes in an A-suffix or a W-suffix call.
> The A-suffix takes ascii strings, the W-suffix takes near UTF-16, actually Microsoft's
> slightly incompatible version. If you don't provide a suffix at all, the compiler
> selects a version which depends on how you have set it up. I'm not sure what the
> default is or exactly how you play with the settings.
>>
>>> I think that having all text streams and plain non-prefixed "strings" as
>>> UTF-8 is both possible and most logical.
>> Yes, and that is allowed by both standards, and is the norm, not the exception, on most
>> Unix-like platforms that I'm familiar with. That's why your claim that it's not allowed
>> confuses me.
>
> My experience is that passing utf-8 to printf() or fopen() doesn't work. But I rarely
> need to do so, and the situation might have changed recently.
>

This program, which contains UTF8 sequences:

#include <stdio.h>

int main(void) {
printf("ø°PÇ€\n");
}

works OK if compiled with bcc or tcc and run with codepage 65001 active.

However it doesn't work if compiled with gcc; I don't know why.

Calling a Windows -A function (eg. MessageBoxA) with UTF8 strings
doesn't work either.

There are also wider aspects than sending output via stdout as mentioned
in the article, such as command line input.

So it's still a mess on Windows from what I can see.

On Friday, 7 January 2022 at 19:18:57 UTC, Bart wrote:
> On 07/01/2022 18:44, Malcolm McLean wrote:
> > On Friday, 7 January 2022 at 17:34:16 UTC, james...@alumni.caltech.edu wrote:
> >>
> >> UTF-16 is, as I understand it, the default in the Windows world. I'm not that familiar
> >> with it, I've only done a few years of programming targeting that platform, and all of
> >> the text that came up in the work I was doing there was simple English, with no need
> >> to make any use of extended characters, so I can't vouch for any of the details about
> >> how text was encoded. However, WIndows is a very common platform, whether or not
> >> you approve of it's design (I share your disapproval for it), so calling UTF-16 obscure
> >> seems very odd.
> >>
> > Every Windows API call that takes text comes in an A-suffix or a W-suffix call.
> > The A-suffix takes ascii strings, the W-suffix takes near UTF-16, actually Microsoft's
> > slightly incompatible version. If you don't provide a suffix at all, the compiler
> > selects a version which depends on how you have set it up. I'm not sure what the
> > default is or exactly how you play with the settings.
> >>
> >>> I think that having all text streams and plain non-prefixed "strings" as
> >>> UTF-8 is both possible and most logical.
> >> Yes, and that is allowed by both standards, and is the norm, not the exception, on most
> >> Unix-like platforms that I'm familiar with. That's why your claim that it's not allowed
> >> confuses me.
> >
> > My experience is that passing utf-8 to printf() or fopen() doesn't work.. But I rarely
> > need to do so, and the situation might have changed recently.
> >
> This program, which contains UTF8 sequences:
>
> #include <stdio.h>
>
> int main(void) {
> printf("ø°PÇ€\n");
> }
>
> works OK if compiled with bcc or tcc and run with codepage 65001 active.
>
Are you sure that's compiling to utf-8?
A better test would be to build the utf-8 sequence explictly, and see if the output
is as specified.
>
> So it's still a mess on Windows from what I can see.
>
It's not just Windows. On my platform, there are several different types and C++
classes that are supposed to hold Unicode. We need a suite of little functions
to do ad hoc conversions between them. Most of these are binary no-ops.

On 07/01/2022 20:27, Malcolm McLean wrote:
> On Friday, 7 January 2022 at 19:18:57 UTC, Bart wrote:
>> On 07/01/2022 18:44, Malcolm McLean wrote:
>>> On Friday, 7 January 2022 at 17:34:16 UTC, james...@alumni.caltech.edu wrote:
>>>>
>>>> UTF-16 is, as I understand it, the default in the Windows world. I'm not that familiar
>>>> with it, I've only done a few years of programming targeting that platform, and all of
>>>> the text that came up in the work I was doing there was simple English, with no need
>>>> to make any use of extended characters, so I can't vouch for any of the details about
>>>> how text was encoded. However, WIndows is a very common platform, whether or not
>>>> you approve of it's design (I share your disapproval for it), so calling UTF-16 obscure
>>>> seems very odd.
>>>>
>>> Every Windows API call that takes text comes in an A-suffix or a W-suffix call.
>>> The A-suffix takes ascii strings, the W-suffix takes near UTF-16, actually Microsoft's
>>> slightly incompatible version. If you don't provide a suffix at all, the compiler
>>> selects a version which depends on how you have set it up. I'm not sure what the
>>> default is or exactly how you play with the settings.
>>>>
>>>>> I think that having all text streams and plain non-prefixed "strings" as
>>>>> UTF-8 is both possible and most logical.
>>>> Yes, and that is allowed by both standards, and is the norm, not the exception, on most
>>>> Unix-like platforms that I'm familiar with. That's why your claim that it's not allowed
>>>> confuses me.
>>>
>>> My experience is that passing utf-8 to printf() or fopen() doesn't work. But I rarely
>>> need to do so, and the situation might have changed recently.
>>>
>> This program, which contains UTF8 sequences:
>>
>> #include <stdio.h>
>>
>> int main(void) {
>> printf("ø°PÇ€\n");
>> }
>>
>> works OK if compiled with bcc or tcc and run with codepage 65001 active.
>>
> Are you sure that's compiling to utf-8?
> A better test would be to build the utf-8 sequence explictly, and see if the output
> is as specified.

Notepad was told to save as UTF8. Codepage 65001 is UTF8 (and it didn't
work with 1252). And I just checked the source file to confirm the
sequences are the correct UTF8.

On Friday, 7 January 2022 at 21:41:00 UTC, Bart wrote:
> On 07/01/2022 20:27, Malcolm McLean wrote:
> > On Friday, 7 January 2022 at 19:18:57 UTC, Bart wrote:
> >> On 07/01/2022 18:44, Malcolm McLean wrote:
> >>> On Friday, 7 January 2022 at 17:34:16 UTC, james...@alumni.caltech.edu wrote:
> >>>>
> >>>> UTF-16 is, as I understand it, the default in the Windows world. I'm not that familiar
> >>>> with it, I've only done a few years of programming targeting that platform, and all of
> >>>> the text that came up in the work I was doing there was simple English, with no need
> >>>> to make any use of extended characters, so I can't vouch for any of the details about
> >>>> how text was encoded. However, WIndows is a very common platform, whether or not
> >>>> you approve of it's design (I share your disapproval for it), so calling UTF-16 obscure
> >>>> seems very odd.
> >>>>
> >>> Every Windows API call that takes text comes in an A-suffix or a W-suffix call.
> >>> The A-suffix takes ascii strings, the W-suffix takes near UTF-16, actually Microsoft's
> >>> slightly incompatible version. If you don't provide a suffix at all, the compiler
> >>> selects a version which depends on how you have set it up. I'm not sure what the
> >>> default is or exactly how you play with the settings.
> >>>>
> >>>>> I think that having all text streams and plain non-prefixed "strings" as
> >>>>> UTF-8 is both possible and most logical.
> >>>> Yes, and that is allowed by both standards, and is the norm, not the exception, on most
> >>>> Unix-like platforms that I'm familiar with. That's why your claim that it's not allowed
> >>>> confuses me.
> >>>
> >>> My experience is that passing utf-8 to printf() or fopen() doesn't work. But I rarely
> >>> need to do so, and the situation might have changed recently.
> >>>
> >> This program, which contains UTF8 sequences:
> >>
> >> #include <stdio.h>
> >>
> >> int main(void) {
> >> printf("ø°PÇ€\n");
> >> }
> >>
> >> works OK if compiled with bcc or tcc and run with codepage 65001 active.
> >>
> > Are you sure that's compiling to utf-8?
> > A better test would be to build the utf-8 sequence explictly, and see if the output
> > is as specified.
> Notepad was told to save as UTF8. Codepage 65001 is UTF8 (and it didn't
> work with 1252). And I just checked the source file to confirm the
> sequences are the correct UTF8.
>
So almost certainly the compiler is compiling it to UTF-8.
But it would have been easier to just specify one non-ascii character as raw bytes,
forcing it to use UTF-8 whatever the source or execution character set.
If it displays correctly, then you know that UTF-8 is supported by the printf / terminal
combination.

On 07/01/2022 22:03, Malcolm McLean wrote:
> On Friday, 7 January 2022 at 21:41:00 UTC, Bart wrote:
>> On 07/01/2022 20:27, Malcolm McLean wrote:
>>> On Friday, 7 January 2022 at 19:18:57 UTC, Bart wrote:
>>>> On 07/01/2022 18:44, Malcolm McLean wrote:
>>>>> On Friday, 7 January 2022 at 17:34:16 UTC, james...@alumni.caltech.edu wrote:
>>>>>>
>>>>>> UTF-16 is, as I understand it, the default in the Windows world. I'm not that familiar
>>>>>> with it, I've only done a few years of programming targeting that platform, and all of
>>>>>> the text that came up in the work I was doing there was simple English, with no need
>>>>>> to make any use of extended characters, so I can't vouch for any of the details about
>>>>>> how text was encoded. However, WIndows is a very common platform, whether or not
>>>>>> you approve of it's design (I share your disapproval for it), so calling UTF-16 obscure
>>>>>> seems very odd.
>>>>>>
>>>>> Every Windows API call that takes text comes in an A-suffix or a W-suffix call.
>>>>> The A-suffix takes ascii strings, the W-suffix takes near UTF-16, actually Microsoft's
>>>>> slightly incompatible version. If you don't provide a suffix at all, the compiler
>>>>> selects a version which depends on how you have set it up. I'm not sure what the
>>>>> default is or exactly how you play with the settings.
>>>>>>
>>>>>>> I think that having all text streams and plain non-prefixed "strings" as
>>>>>>> UTF-8 is both possible and most logical.
>>>>>> Yes, and that is allowed by both standards, and is the norm, not the exception, on most
>>>>>> Unix-like platforms that I'm familiar with. That's why your claim that it's not allowed
>>>>>> confuses me.
>>>>>
>>>>> My experience is that passing utf-8 to printf() or fopen() doesn't work. But I rarely
>>>>> need to do so, and the situation might have changed recently.
>>>>>
>>>> This program, which contains UTF8 sequences:
>>>>
>>>> #include <stdio.h>
>>>>
>>>> int main(void) {
>>>> printf("ø°PÇ€\n");
>>>> }
>>>>
>>>> works OK if compiled with bcc or tcc and run with codepage 65001 active.
>>>>
>>> Are you sure that's compiling to utf-8?
>>> A better test would be to build the utf-8 sequence explictly, and see if the output
>>> is as specified.
>> Notepad was told to save as UTF8. Codepage 65001 is UTF8 (and it didn't
>> work with 1252). And I just checked the source file to confirm the
>> sequences are the correct UTF8.
>>
> So almost certainly the compiler is compiling it to UTF-8.

Actually, my compiler at least is doing nothing at all. It knows nothing
about UTF8; it's just a sequence of bytes forming a string literal. The
E2 82 AC sequence representing € is output to the binary as a E2 82 AC
sequence, just as 41 42 43 is passed through as 41 42 43 ("ABC").

That's the advantage of UTF8.

It is the editor that needs to be aware of it, needing to deal with
input of it, display, and writing to a file with the correct encoding.

And the runtime or OS must also show the correct display for UTF8
sequences. It's that bit that is going wrong with gcc.

On Friday, 7 January 2022 at 22:16:33 UTC, Bart wrote:
> On 07/01/2022 22:03, Malcolm McLean wrote:
> > On Friday, 7 January 2022 at 21:41:00 UTC, Bart wrote:
> >> On 07/01/2022 20:27, Malcolm McLean wrote:
> >>> On Friday, 7 January 2022 at 19:18:57 UTC, Bart wrote:
> >>>> On 07/01/2022 18:44, Malcolm McLean wrote:
> >>>>> On Friday, 7 January 2022 at 17:34:16 UTC, james...@alumni.caltech.edu wrote:
> >>>>>>
> >>>>>> UTF-16 is, as I understand it, the default in the Windows world. I'm not that familiar
> >>>>>> with it, I've only done a few years of programming targeting that platform, and all of
> >>>>>> the text that came up in the work I was doing there was simple English, with no need
> >>>>>> to make any use of extended characters, so I can't vouch for any of the details about
> >>>>>> how text was encoded. However, WIndows is a very common platform, whether or not
> >>>>>> you approve of it's design (I share your disapproval for it), so calling UTF-16 obscure
> >>>>>> seems very odd.
> >>>>>>
> >>>>> Every Windows API call that takes text comes in an A-suffix or a W-suffix call.
> >>>>> The A-suffix takes ascii strings, the W-suffix takes near UTF-16, actually Microsoft's
> >>>>> slightly incompatible version. If you don't provide a suffix at all, the compiler
> >>>>> selects a version which depends on how you have set it up. I'm not sure what the
> >>>>> default is or exactly how you play with the settings.
> >>>>>>
> >>>>>>> I think that having all text streams and plain non-prefixed "strings" as
> >>>>>>> UTF-8 is both possible and most logical.
> >>>>>> Yes, and that is allowed by both standards, and is the norm, not the exception, on most
> >>>>>> Unix-like platforms that I'm familiar with. That's why your claim that it's not allowed
> >>>>>> confuses me.
> >>>>>
> >>>>> My experience is that passing utf-8 to printf() or fopen() doesn't work. But I rarely
> >>>>> need to do so, and the situation might have changed recently.
> >>>>>
> >>>> This program, which contains UTF8 sequences:
> >>>>
> >>>> #include <stdio.h>
> >>>>
> >>>> int main(void) {
> >>>> printf("ø°PÇ€\n");
> >>>> }
> >>>>
> >>>> works OK if compiled with bcc or tcc and run with codepage 65001 active.
> >>>>
> >>> Are you sure that's compiling to utf-8?
> >>> A better test would be to build the utf-8 sequence explictly, and see if the output
> >>> is as specified.
> >> Notepad was told to save as UTF8. Codepage 65001 is UTF8 (and it didn't
> >> work with 1252). And I just checked the source file to confirm the
> >> sequences are the correct UTF8.
> >>
> > So almost certainly the compiler is compiling it to UTF-8.
> Actually, my compiler at least is doing nothing at all. It knows nothing
> about UTF8; it's just a sequence of bytes forming a string literal. The
> E2 82 AC sequence representing € is output to the binary as a E2 82 AC
> sequence, just as 41 42 43 is passed through as 41 42 43 ("ABC").
>
> That's the advantage of UTF8.
>
> It is the editor that needs to be aware of it, needing to deal with
> input of it, display, and writing to a file with the correct encoding.
>
Yes, the majority of a C source file is going to be pure ascii, with only
a few extended string literals embedded. So UTF-8 is a good choice. The
compiler needs no modification, and file size remains about the same.
>
> And the runtime or OS must also show the correct display for UTF8
> sequences. It's that bit that is going wrong with gcc.
>
The terminal is the same. So gcc must be linking a printf() that doesn't
treat UTF-8 correctly, though it's hard to see what coule be going wrong
if the terminal takes UTF-8 in 8 bit bytes.

On Friday, 7 January 2022 at 19:34:16 UTC+2, james...@alumni.caltech.edu wrote:
> On Friday, January 7, 2022 at 7:06:26 AM UTC-5, Öö Tiib wrote:
> > On Thursday, 6 January 2022 at 19:28:46 UTC+2, james...@alumni.caltech.edu wrote:
> > > On Wednesday, January 5, 2022 at 11:35:38 PM UTC-5, Öö Tiib wrote:
> ...
> > > > Even programmer of such system might want to upgrade to compiler
> > > > whose standard library supports UTF-8. But it is not possible as
> > > > standard library is defined not to.
> > > Could you cite the text from the C and C++ standards that prohibits the
> > > standard library from supporting UTF-8?
> > Are you pretending that you did not understand what I meant?
> No, I am quite accurately and honestly expressing my confusion. You object to
> something being prohibited by the standards that is, to the best of my understanding,
> allowed. It would make more sense if you were objecting the fact that it isn't
> mandatory, and if you were making such claims, I would disagree with you about
> whether it would be a good idea to make it mandatory - but as far as I can tell, you're
> claiming it isn't allowed.

It is allowed. Almost whatever is allowed. But you yourself listed all that distracting
and confusing half-support, all those char8_t-s, added and deprecated codecvt-s
and u8 prefixes. Every possible thing designed to avoid adding actual support
to standard.

> If you could, as requested, cite the relevant text that prohibits such compilers, you
> might convince me that I'm wrong. If not, the citation would at least enable me to try
> to convince you that you're misinterpreting that text. Neither possibility can happen
> until you actually honor that request.

I can not possibly cite that. And now I'm confused how can be you snipped that
"The standards allow implementations to have wide array of whatever obscure
extensions." That already told it? So if I worded it unclear, then it is my fault.
Why must UTF-8 be extension?

>
> ...
> > However some essential things, like say adding 128 bit integers or
> > UTF-8 support or even to stop that nonsense with newline characters
> > is made tricky.
>
> Neither 128 bits integer nor UTF-8 support are essential. Lots of people have no
> need of either (I've never needed either one, though I have used UTF-8 since it was
> available). If you need such things on a platform where no existing implementation
> of C provides them, your complaint is with the implementors, not the standard,
> because the standard says nothing to prohibit such things.
>

In world where 98% of text communication goes with UTF-8 there of course
is some shrinking 2% of market left.
> ...
> > I am saying that when to ask why default string can't be UTF-8 then that
> > EBCDIC is usually mentioned. Despite there probably are no much C, let
> > alone C++ used on few alive EBCDIC platforms.
> That seems reasonable to me. The only places where that logic applies are
> implementations of C targeting EBCDIC platforms, and regardless of how rare such
> implementations, they would become substantially rarer because users would abandon
> them if they switched to UTF-8.

Hypothetical C programmer on EBCDIC platform (never heard of one) most likely
wants to exchange information with rest of the world. So why he would not want
to upgrade to compiler that supports UTF-8? I am purely speculating as I got no
experience with those devices.

>
> ...
> > The UTF-16 and UTF-32 are also among those 2% of obscure text encodings..
> > OK, UTF-16 can be useful to communicate with mis-designed programming
> > languages like Java or C# or operating systems like Windows. But UTF-32
> > is rather exotic.
> UTF-16 is, as I understand it, the default in the Windows world. I'm not that familiar
> with it, I've only done a few years of programming targeting that platform, and all of
> the text that came up in the work I was doing there was simple English, with no need
> to make any use of extended characters, so I can't vouch for any of the details about
> how text was encoded. However, WIndows is a very common platform, whether or not
> you approve of it's design (I share your disapproval for it), so calling UTF-16 obscure
> seems very odd.

It has usages like I confirmed already. Obscure I said because it has merged the
overhead and need for BOMs of UTF-32 with inconveniences of UTF-8 without
any benefits.

>
> UTF-32 shares an important property with the implementation-defined encoding used
> for wchar_t: every character takes up one and only one element in the array. I have
> written a lot of code over several decades for parsing strings that assumes that every
> character takes up one char, a valid assumption in the contexts in which I wrote it. When
> I think about how I would have to re-write such code to work with multi-byte encodings
> such as UTF-8, then the simplicity of replacing char with wchar_t or char32_t, leaving my
> logic unchanged, starts looking pretty attractive. However, since I have little experience
> writing code to work with extended characters using any encoding, my preferences don't
> carry much weight.

That is good point. UTF-8 converts trivially to UTF-32 and back. So where such
precisely-one guarantee helps to make some algorithm more robust and simple
there we can easily convert of course. But I see no reason why to keep (or to
transfer) UTF-32 outside of such context.

> > > While C, in particular, doesn't mandate quite as much support for UTF-8 as
> > > I'd like, both standards allow the fullest possible support for UTF-8 that I
> > > could imagine. Why do you think otherwise?
> > I think that having all text streams and plain non-prefixed "strings" as
> > UTF-8 is both possible and most logical.
> Yes, and that is allowed by both standards, and is the norm, not the exception, on most
> Unix-like platforms that I'm familiar with. That's why your claim that it's not allowed
> confuses me.

If I appeared to make that claim then it is probably my fault. I meant it is made difficult
by adding things that look like trying to support it one day but there really are no UTF-8
support in standards despite we use it everywhere for decades.

> If you don't work on Unix systems where the encoding of non-prefixed
> strings is UTF-8, and you don't work on Windows systems where UTF-16 is the norm, and
> you don't work on systems where EBCDIC is the norm, what kinds of systems do you
> work on? I'm not saying that there aren't any other types of systems, there's a great
> many, but most of the others are substantially less common, so I am just curious which
> one(s) you use.

I have written C and C++ for systems and peripherals of things like cash dispensers,
point of sale terminals, taximeters, spectrometers, thermostats, frequency converters
and mobile phones. Also I have participated in projects of writing utility software and
services on Unixes and Windowses. From systems that I've programmed if to
remove the pointless obfuscation garbage from standards and to require UTF-8
then perhaps only MS has to do something at all.

On 1/7/22 7:06 AM, Öö Tiib wrote:
> The UTF-16 and UTF-32 are also among those 2% of obscure text encodings.
> OK, UTF-16 can be useful to communicate with mis-designed programming
> languages like Java or C# or operating systems like Windows. But UTF-32
> is rather exotic.
>

One key thing to remember is that most UTF-16 (and especially Window's
use of it) goes back to a too early adoption of Unicode and UCS-2 as
'The Standard' for Text, when 16-bit 'Unicode' was claimed to be the
answer to the problem of all those code-pages.

Then, when after it got adopted and baked into ABIs/APIs it was realized
that Unicode was going to need to be bigger so UCS-2 switched to UTF-16
and UCS-4 became the real 'wide character' type (except, in some ways it
still wasn't due to combining codepoints).

For C, this basically means that the 'wide character' system is
practically broken, and is really broken on Windows machines as the
standard says its needs to be the widest type of character, and that it
expresses all the characters in one unit, but on Windows by ABI
requirements it must be 16 bits, and 32 bits isn't really correct even
on systems which do use that for wide characters due to the combining
codes issue.

Basically, for a system that want to really conform to both Unicode and
the C standard, the implementation is put in a spot where it is actually
impossible to do so, at least if you want to keep the full intent of the
standard (that wide strings can be arbitrarily split and joined without
issue).

Then you have that Unicode is actually not stateless when you add things
like the various Left-to-Right codes or things emoji modifier characters.

On Friday, January 7, 2022 at 10:32:13 PM UTC-5, Öö Tiib wrote:
> On Friday, 7 January 2022 at 19:34:16 UTC+2, james...@alumni.caltech.edu wrote:
> > On Friday, January 7, 2022 at 7:06:26 AM UTC-5, Öö Tiib wrote:
....
> > No, I am quite accurately and honestly expressing my confusion. You object to
> > something being prohibited by the standards that is, to the best of my understanding,
> > allowed. It would make more sense if you were objecting the fact that it isn't
> > mandatory, and if you were making such claims, I would disagree with you about
> > whether it would be a good idea to make it mandatory - but as far as I can tell, you're
> > claiming it isn't allowed.
> It is allowed. Almost whatever is allowed. But you yourself listed all that distracting
> and confusing half-support, all those char8_t-s, added and deprecated codecvt-s
> and u8 prefixes. Every possible thing designed to avoid adding actual support
> to standard.

So, you are arguing that it should be mandatory to have UTF-8 as the encoding for
unprefixed string literals, even for implementations targeting platforms where that's
contrary to the conventions for that platform?

> > If you could, as requested, cite the relevant text that prohibits such compilers, you
> > might convince me that I'm wrong. If not, the citation would at least enable me to try
> > to convince you that you're misinterpreting that text. Neither possibility can happen
> > until you actually honor that request.
> I can not possibly cite that. And now I'm confused how can be you snipped that
> "The standards allow implementations to have wide array of whatever obscure
> extensions." That already told it? So if I worded it unclear, then it is my fault.
> Why must UTF-8 be extension?

It didn't occur to me that you meant "obscure extensions" to refer to UTF-8 support. It
isn't an extension. "The values of the members of the execution character set are
implementation-defined." (5.2.1p1). That puts choosing UTF-8 for that encoding in the
same category as choosing 8 as the value for CHAR_BIT or setting the values for the
macros that are #defined in <limits.h>.

The term "extension" is not normally used for implementation-defined behavior. Note that
4p9 requires that "An implementation shall be accompanied by a document that defines
all implementation-defined and locale-specific characteristics and all extensions." If
implementation-defined behavior were considered to qualify as an extension, that
specification would be redundant, something the committee generally tries to avoid.

.....
> In world where 98% of text communication goes with UTF-8 there of course
> is some shrinking 2% of market left.

Do you have sources for those numbers, or are you just pulling them out of your hat?
I'm not saying you're wrong, just that I don't know of any easy way to determine what
those numbers are.

> > ...
> > > I am saying that when to ask why default string can't be UTF-8 then that
> > > EBCDIC is usually mentioned. Despite there probably are no much C, let
> > > alone C++ used on few alive EBCDIC platforms.
> > That seems reasonable to me. The only places where that logic applies are
> > implementations of C targeting EBCDIC platforms, and regardless of how rare such
> > implementations, they would become substantially rarer because users would abandon
> > them if they switched to UTF-8.
> Hypothetical C programmer on EBCDIC platform (never heard of one) most likely
> wants to exchange information with rest of the world. So why he would not want
> to upgrade to compiler that supports UTF-8? I am purely speculating as I got no
> experience with those devices.

Unlike your hypothetical C programmer on EBCDIC platform, real programmers of that type
have access to conversion routines for use when the need to communicate outside the
EBCDIC world comes up. If UTF-8 were mandatory for unprefixed string literals, an
implementations targeting such platforms that conformed to such a mandate could add an
extension to create EBCDIC-encoded string literals. If so, developers for such platforms
would have to routinely use that extension for most of their string literals. Such developers
would find that very inconvenient, and would therefore make sure that any implementation
targeting that platform had an option that would make it fail to conform to such a mandate.

Imposing that mandate would fail to make UTF-8 any more widely used. The reason that
there do exist platforms where UTF-8 is not the encoding used for unprefixed string literals
is because their users want some other encoding to be used for that purpose.. If that weren't
the case, someone would have already created a UTF-8 implementation for that platform.

....
> > how text was encoded. However, WIndows is a very common platform, whether or not
> > you approve of it's design (I share your disapproval for it), so calling UTF-16 obscure
> > seems very odd.
> It has usages like I confirmed already. Obscure I said because it has merged the
> overhead and need for BOMs of UTF-32 with inconveniences of UTF-8 without
> any benefits.

It doesn't matter how strongly you disapprove of it - what matters is how many people want
to use it despite your disapproval.

On Friday, January 7, 2022 at 1:44:39 PM UTC-5, Malcolm McLean wrote:
> On Friday, 7 January 2022 at 17:34:16 UTC, james...@alumni.caltech.edu wrote:
....
> > > I think that having all text streams and plain non-prefixed "strings" as
> > > UTF-8 is both possible and most logical.
> > Yes, and that is allowed by both standards, and is the norm, not the exception, on most
> > Unix-like platforms that I'm familiar with. That's why your claim that it's not allowed
> > confuses me.
> My experience is that passing utf-8 to printf() or fopen() doesn't work. But I rarely
> need to do so, and the situation might have changed recently.

My wife was born in Taiwan, and our kids are bilingual, so every computer in our house has
been set up to handle Chinese text properly. If yours isn't, you might not see the right
characters in the string literals below. You could try using

"\u5929\u5B89\u95E8\u5E7F\u573A"

instead - the encoding of the character arrays should be unchanged.

#include <inttypes.h>
#include <locale.h>
#include <stdio.h>
#include <stdlib.h>
#include <uchar.h>

int main(void)
{ setlocale(LC_ALL, "");
const char location[] = "天安门广场";
const char location8[] = u8"天安门广场";
const char *p;

printf("Location :");
for(p = location; *p; p++)
printf("%#X ", *(unsigned char*)p);

printf("\nLocation8:");
for(p = location8; *p; p++)
printf("%#X ", *(unsigned char*)p);

printf("\n\"%s\"\n", location);

p = location;
const char * const end = location + sizeof location;
mbstate_t state={0};
while(*p)
{
char32_t c32;
size_t bytes = mbrtoc32(&c32, p, end - p, &state);
switch(bytes)
{
case (size_t)(-3):
printf("%#" PRIXLEAST32 " ", c32);
break;
case (size_t)(-2):
break;
fprintf(stderr, "incomplete character\n");
return EXIT_FAILURE;
case (size_t)(-1):
fprintf(stderr, "%td:Encoding error\n", p-location);
return EXIT_FAILURE;
default:
printf("%#" PRIXLEAST32 " ", c32);
p += bytes;
break;
}
}

printf("\n");
return EXIT_SUCCESS;
}

That program produces the following output on my system:
Location :0XE5 0XA4 0XA9 0XE5 0XAE 0X89 0XE9 0X97 0XA8 0XE5 0XB9 0XBF 0XE5 0X9C 0XBA
Location8:0XE5 0XA4 0XA9 0XE5 0XAE 0X89 0XE9 0X97 0XA8 0XE5 0XB9 0XBF 0XE5 0X9C 0XBA
"天安门广场"
0X5929 0X5B89 0X95E8 0X5E7F 0X573A

Note that the u8 string is encoded exactly the same way as the unprefixed string literal,
confirming that UTF-8 is the encoding for unprefixed string literals.
The setlocale() call is not needed to correctly display the string, but it is needed for
mbrtoc32() to work. In the default "C" locale, mbrtoc32() reports an encoding error. The
"" locale is the implementation-defined default locale - I'm not sure what gcc defines that
default to be, but I suspect that it's the value of my LANG environment variable, which is
"en_US.UTF-8". Virtually every locale supported on my system has UTF-8 or utf8 in it's name.
The "C" locale is one of the few exceptions, but a "C.UTF-8" locale is also supported.

On Saturday, 8 January 2022 at 06:52:33 UTC+2, james...@alumni.caltech.edu wrote:
> On Friday, January 7, 2022 at 10:32:13 PM UTC-5, Öö Tiib wrote:
> > On Friday, 7 January 2022 at 19:34:16 UTC+2, james...@alumni.caltech.edu wrote:
> > > On Friday, January 7, 2022 at 7:06:26 AM UTC-5, Öö Tiib wrote:
> ...
> > > No, I am quite accurately and honestly expressing my confusion. You object to
> > > something being prohibited by the standards that is, to the best of my understanding,
> > > allowed. It would make more sense if you were objecting the fact that it isn't
> > > mandatory, and if you were making such claims, I would disagree with you about
> > > whether it would be a good idea to make it mandatory - but as far as I can tell, you're
> > > claiming it isn't allowed.
> > It is allowed. Almost whatever is allowed. But you yourself listed all that distracting
> > and confusing half-support, all those char8_t-s, added and deprecated codecvt-s
> > and u8 prefixes. Every possible thing designed to avoid adding actual support
> > to standard.
>
> So, you are arguing that it should be mandatory to have UTF-8 as the encoding for
> unprefixed string literals, even for implementations targeting platforms where that's
> contrary to the conventions for that platform?

Yes, and vast majority would be happy. What other char* text is needed than UTF-8?
Why? On what? For what? Must be odd corner case. Each trashcan, smoke sensor or
microwave oven out there wants to communicate with whatever siris, alexas,
google homes and skynets they serve. All of those use UTF-8 texts. If it has some
LCD or LED panel then it wants to show text understandable to local desperate
housewife, low salary technician or taxi driver. If it is char* then it is UTF-8 there.

> > > If you could, as requested, cite the relevant text that prohibits such compilers, you
> > > might convince me that I'm wrong. If not, the citation would at least enable me to try
> > > to convince you that you're misinterpreting that text. Neither possibility can happen
> > > until you actually honor that request.
> > I can not possibly cite that. And now I'm confused how can be you snipped that
> > "The standards allow implementations to have wide array of whatever obscure
> > extensions." That already told it? So if I worded it unclear, then it is my fault.
> > Why must UTF-8 be extension?
>
> It didn't occur to me that you meant "obscure extensions" to refer to UTF-8 support. It
> isn't an extension. "The values of the members of the execution character set are
> implementation-defined." (5.2.1p1). That puts choosing UTF-8 for that encoding in the
> same category as choosing 8 as the value for CHAR_BIT or setting the values for the
> macros that are #defined in <limits.h>.

CHAR_BIT can't be less than 8 so UTF-8 code unit is guaranteed to fit. The flexibility
to have bigger CHAR_BIT than 8 can be left there for char has to serve also as byte.

> The term "extension" is not normally used for implementation-defined behavior. Note that
> 4p9 requires that "An implementation shall be accompanied by a document that defines
> all implementation-defined and locale-specific characteristics and all extensions." If
> implementation-defined behavior were considered to qualify as an extension, that
> specification would be redundant, something the committee generally tries to avoid.

If there is implementation defined behavior or not in my experience if text is passed
with char* then it points at UTF-8 and programmer has to fight with that implementation
defined garbage because he needs it to be UTF-8. And I'm complaining against
attempts to lie to novices that UTF-8 should be uchar8_t* or something else like
that. Practical example:
FILE *f = fopen( "Foo😀Bar.txt", "w");
That should work unless underlying file system does not support files
named "Foo😀Bar.txt" If it supports but the code does not work then it indicates
bad standard that allows implementations to weasel away. No garbage like
u8fopen( u8"Foo😀Bar.txt", "w") coming somewhere maybe in C35 or so is
needed as it already works like in my example on vast majority of things.

> ....
> > In world where 98% of text communication goes with UTF-8 there of course
> > is some shrinking 2% of market left.
> Do you have sources for those numbers, or are you just pulling them out of your hat?
> I'm not saying you're wrong, just that I don't know of any easy way to determine what
> those numbers are.

There are no easy way but some organizations do diligently statistics what is
possible to monitor. Like that:
<https://w3techs.com/technologies/history_overview/character_encoding>
Legacy is there but shrinking. If whatever new C position opens where text has
to be accessed with char* then chance is close to 0 that it is something else
but UTF-8.
> > > ...
> > > > I am saying that when to ask why default string can't be UTF-8 then that
> > > > EBCDIC is usually mentioned. Despite there probably are no much C, let
> > > > alone C++ used on few alive EBCDIC platforms.
> > > That seems reasonable to me. The only places where that logic applies are
> > > implementations of C targeting EBCDIC platforms, and regardless of how rare such
> > > implementations, they would become substantially rarer because users would abandon
> > > them if they switched to UTF-8.
> > Hypothetical C programmer on EBCDIC platform (never heard of one) most likely
> > wants to exchange information with rest of the world. So why he would not want
> > to upgrade to compiler that supports UTF-8? I am purely speculating as I got no
> > experience with those devices.
>
> Unlike your hypothetical C programmer on EBCDIC platform, real programmers of that type
> have access to conversion routines for use when the need to communicate outside the
> EBCDIC world comes up. If UTF-8 were mandatory for unprefixed string literals, an
> implementations targeting such platforms that conformed to such a mandate could add an
> extension to create EBCDIC-encoded string literals. If so, developers for such platforms
> would have to routinely use that extension for most of their string literals. Such developers
> would find that very inconvenient, and would therefore make sure that any implementation
> targeting that platform had an option that would make it fail to conform to such a mandate.

You never answered why should they use obscure extensions for what they need on
majority of cases. Why UTF-8 must be obscure extension?

> Imposing that mandate would fail to make UTF-8 any more widely used. The reason that
> there do exist platforms where UTF-8 is not the encoding used for unprefixed string literals
> is because their users want some other encoding to be used for that purpose. If that weren't
> the case, someone would have already created a UTF-8 implementation for that platform.

It is used on close to 100% of cases anyway. I am objecting that it is deliberately
standardized (or more like pseudo-standardized/non-standardized) to be
inconvenient to use.

> ...
> > > how text was encoded. However, WIndows is a very common platform, whether or not
> > > you approve of it's design (I share your disapproval for it), so calling UTF-16 obscure
> > > seems very odd.
> > It has usages like I confirmed already. Obscure I said because it has merged the
> > overhead and need for BOMs of UTF-32 with inconveniences of UTF-8 without
> > any benefits.
> It doesn't matter how strongly you disapprove of it - what matters is how many people want
> to use it despite your disapproval.

Agreed. So do you have numbers how many C programmers *want* to use UTF-16? I
think that it is little, but I do not have any sources. They may *need* to for legacy
reasons I already mentioned but even there it is most likely small number. Their
pain with support to their u"string", L"string" and \x \u \U character references
might need relieving too but is bit different topic.

On Saturday, 8 January 2022 at 16:17:53 UTC, Öö Tiib wrote:
> On Saturday, 8 January 2022 at 06:52:33 UTC+2, james...@alumni.caltech.edu wrote:
>
> > So, you are arguing that it should be mandatory to have UTF-8 as the encoding for
> > unprefixed string literals, even for implementations targeting platforms where that's
> > contrary to the conventions for that platform?
> Yes, and vast majority would be happy. What other char* text is needed than UTF-8?
> Why? On what? For what? Must be odd corner case.
>
Where you've got an 8 bit character-mapped display that supports ascii plus some
extended characters. That used to be almost every microcomputer, and it still lives
on to a bit in modern PCs.