On Monday, 10 January 2022 at 19:01:31 UTC+2, james...@alumni.caltech.edu wrote:
> On Monday, January 10, 2022 at 1:11:08 AM UTC-5, Öö Tiib wrote:
> > On Monday, 10 January 2022 at 07:24:01 UTC+2, james...@alumni.caltech.edu wrote:
> > > On Sunday, January 9, 2022 at 5:35:42 PM UTC-5, Öö Tiib wrote:
> > > > On Sunday, 9 January 2022 at 18:34:25 UTC+2, Manfred wrote:
> > > > > On 1/9/2022 1:44 PM, Öö Tiib wrote:
> > > ...
> > > > > > Something that makes it clear that it is defect when "Foo😀Bar.txt" is silently opened
> > > > > > on file-system that fully supports files named "Foo😀Bar.txt" I suppose.
> > > > > >
> > > > > Assuming that "Foo≡ƒÿÇBar.txt" "Foo😀Bar.txt" have the same binary
> > > > > representation, what's the difference? One form or the other shows up
> > > > > only when it is displayed in some UI - the filesystem isn't one, which
> > > > > leads to the implementation's runtime behavior.
> > > > How you mean same binary representation? Both "Foo😀Bar.txt" and
> > > > "Foo😀Bar.txt" files can be in same directory. Both have Unicode
> > > > names in underlying file system precisely as posted.
> > > Have you checked to make sure? Any system where passing the UTF-8 string
> > > "Foo😀Bar.txt" to fopen() opens a file whose name displays as Foo≡ƒÿÇBar.txt
> > > is likely to be a system where the file names are displayed using some single-
> > > byte encoding. The UTF-8 encoding of "Foo😀Bar.txt" is
> > >
> > > 0X46 0X6F 0X6F 0XF0 0X9F 0X98 0X80 0X42 0X61 0X72 0X2E 0X74 0X78 0X74
> > >
> > > After quite a bit of searching, I found Code page 865 (MS-DOS Nordic), which
> > > has 0XF0 = '≡', 0x9F = 'ƒ' and 0X98 = 'ÿ'. If the utilities that you use to display file
> > > names used that encoding to interpret the file name, that would explain your results.
> > Need a screenshot with files named "Foo😀Bar.txt" and "Foo≡ƒÿÇBar.txt"
> > side-by-side? ...
>
> No, that wouldn't be relevant. I realized shortly after I posted my message that you
> might not understand what I meant by "Have you checked to make sure?" I started
> composing a message in my head explaining in more detail. However, it was very late,
> I had to get to bed, and when I checked this morning you'd already confirmed that you
> didn't realize what I meant.
>
> As I understand it, you've opened a file using "Foo😀Bar.txt" as the file name, and
> somehow determined that the "actual" name of the file that got opened was
> "Foo😀Bar.txt". You didn't specify, but I presume you reached that conclusion by
> doing something like get a directory listing at the command line or using a GUI file
> browser to look at the directory.
> The UTF-8 encoding of "Foo😀Bar.txt" is
> 0X46 0X6F 0X6F 0XE2 0X89 0XA1 0XC6 0X92 0XC3 0XBF 0XC3 0X87 0X42 0X61 0X72 0X2E 0X74 0X78 0X7
>
> Those same bytes, if interpreted using Code page 865, represent the string
> "Foo😀Bar.txt". I know very little about Windows internals, so I'm not sure why that
> might be relevant. It could be just a coincidence, but that seems excessively
> unlikely. What I was suggestinging, and what I think Manfred was hinting at, is that the
> string you provide as the file name is stored by the file system using UTF-8 encoding.
> Whatever method you used to determine the "actual" file name interpreted those
> bytes using a single-byte encoding, which could be Code Page 865, or possibly some
> other encoding that encodes those particular characters the same way as Code Pag
> 865. There's a lot of different code pages out there, so I couldn't check them all, but of
> the dozen or so I checked, that is the only one where 0xF0 represents '≡'. The "MS
> DOS Nordic" code page was one of the first ones I checked, based upon your e-mail
> address oot...@hot.ee, where I presume "ee" refers to Estonia.
>
> If that is indeed the case, consider what should happen if you try to open a file using
> the name "Foo😀Bar.txt". If I understand you correctly, I believe that you would
> expect it to open the same file that got opened when you specified "Foo😀Bar.txt".
> However, the UTF-8 encoding of "Foo😀Bar.txt" is
>
> 0X46 0X6F 0X6F 0XE2 0X89 0XA1 0XC6 0X92 0XC3 0XBF 0XC3 0X87 0X42 0X61 0X72 0X2E 0X74 0X78 0X74
>
> I expect that you will end up opening a different file. If the file name is being displayed
> using a single-byte encoding, it should have 19 characters. If that encoding is in fact
> Code Page 865, then that name should be "Foo😀Bar.txt". So, what result do
> you get?
>
> If the problem is in fact that the file name is being interpreted using a single byte
> encoding by whatever utility you're using determine what the actual name is, then
> there's absolutely nothing the standards can do about that - the behavior of any such
> utility is completely outside the scope of either standard.
>
> > ... Just that for "Foo😀Bar.txt" one needs to use non-standard
> > _wfopen( L"Foo😀Bar.txt", L"w"). That is default both with MSVC and MinGW
> > gcc.
> As you say, it's non-standard. Therefore, nothing the C standard says could do
> anything to constrain it's behavior. If your complaint is indeed about the behavior of
> _wfopen(), it's not relevant to either the C or C++ standards, and should be posted to a
> Windows-specific forum.

Hmm. I'm thinking you analyze correctly. To clarify wherever I refer to file
named "Foo😀Bar.txt" and file named "Foo≡ƒÿÇBar.txt" then it is how Windows
(or other operating system if I make the files reachable) displays names of the files.
The string "Foo😀Bar.txt" is what I see in text editor of UTF-8 source code. The
bytes of yours are correct. Windows C runtime translates all char* strings to
UTF-16 and so it is all about algorithm how it does that.

On Monday, 10 January 2022 at 15:38:21 UTC+2, Richard Damon wrote:
> On 1/10/22 8:15 AM, Öö Tiib wrote:
> > On Monday, 10 January 2022 at 11:02:54 UTC+2, Mateusz Viste wrote:
> >>
> >> Now, I am not saying that writing a utf-8 strlen() is incredibly
> >> difficult of course. I am only saying it is an extra layer of
> >> complexity compared to UCS-2 or UTF-32. And that is why I understand
> >> why people often choose to internally store strings in one of these
> >> encodings instead of utf-8 (esp. if dealing with fixed-width character
> >> outputs). It's simply easier to deal with an array of values that maps
> >> directly to codepoints rather than parse a utf-8 string taking care not
> >> to explode on encoding errors or edge cases.
> >
> > That argument feels like result of misinterpretation of formation of
> > UCS-2 and UTF-32 glyphs. Both encodings contain variety of combining
> > characters, modifiers, accents and tabulators. So even with monospaced
> > font (in world where proportional fonts are more frequently used) one
> > can't decide the width of result on screen without examining all
> > characters in sequence. But if to examine all characters of sequence
> > anyway then UTF-8 is often.just bit less memory to examine. Somehow
> > it does not look like the other options are "simply easier".
> >
> And this is the reason that Unicode doesn't really meet the requirements
> of a C 'Wide Character Type'. Wide characters are supposed to be 1
> character = 1 storage unit. Because of combining characters Unicode
> doesn't meet this requirement.
>
> Ultimately, we have to live with it and accept that programming in the
> face of full compliance with the rules of the character set are going to
> add complexity.

Unicode is quite successful in supporting nuances of texts that
people expect to see. So it is most popular text format in world.
If it contradicts with requirements of C then ... C has to change
as world is lot harder to change.

On Tuesday, 11 January 2022 at 09:52:19 UTC, Öö Tiib wrote:
> On Monday, 10 January 2022 at 15:38:21 UTC+2, Richard Damon wrote:
> > On 1/10/22 8:15 AM, Öö Tiib wrote:
> > > On Monday, 10 January 2022 at 11:02:54 UTC+2, Mateusz Viste wrote:
> > >>
> > >> Now, I am not saying that writing a utf-8 strlen() is incredibly
> > >> difficult of course. I am only saying it is an extra layer of
> > >> complexity compared to UCS-2 or UTF-32. And that is why I understand
> > >> why people often choose to internally store strings in one of these
> > >> encodings instead of utf-8 (esp. if dealing with fixed-width character
> > >> outputs). It's simply easier to deal with an array of values that maps
> > >> directly to codepoints rather than parse a utf-8 string taking care not
> > >> to explode on encoding errors or edge cases.
> > >
> > > That argument feels like result of misinterpretation of formation of
> > > UCS-2 and UTF-32 glyphs. Both encodings contain variety of combining
> > > characters, modifiers, accents and tabulators. So even with monospaced
> > > font (in world where proportional fonts are more frequently used) one
> > > can't decide the width of result on screen without examining all
> > > characters in sequence. But if to examine all characters of sequence
> > > anyway then UTF-8 is often.just bit less memory to examine. Somehow
> > > it does not look like the other options are "simply easier".
> > >
> > And this is the reason that Unicode doesn't really meet the requirements
> > of a C 'Wide Character Type'. Wide characters are supposed to be 1
> > character = 1 storage unit. Because of combining characters Unicode
> > doesn't meet this requirement.
> >
> > Ultimately, we have to live with it and accept that programming in the
> > face of full compliance with the rules of the character set are going to
> > add complexity.
> Unicode is quite successful in supporting nuances of texts that
> people expect to see. So it is most popular text format in world.
> If it contradicts with requirements of C then ... C has to change
> as world is lot harder to change.
>
UTF-8 is designed so that programs written in C, as well as many other
programming languages, have a good chance of working correctly
even if not UTF-8 aware.
However if you are displaying text, and start developing for an exclusively
English-speaking end user, then moving to non-English texts isn't always as
simple as changing the raster patterns of the glyphs. That's inherent in the
complexities of human languages and writing systems. Representing
strings in UTF-8 is just the start.

On 11/01/2022 02:03, Öö Tiib wrote:
> On Tuesday, 11 January 2022 at 00:09:48 UTC+2, Scott Lurndal wrote:
>> Vir Campestris <vir.cam...@invalid.invalid> writes:
>>> On 10/01/2022 00:51, Öö Tiib wrote:
>>>> But do there exist machines that do want to support texts as char* but
>>>> do not want to support UTF-8? Describe those machines, give examples.
>>>
>>> All the mainframes that run EBCDIC. There are a lot of them still.
>> Here's the implementation guide for one:
>>
>> https://public.support.unisys.com/aseries/docs/ClearPath-MCP-18.0/86002268-207.pdf
>>
>> See appendix E for I18N.
>
> Seems out of context as these guys do not look like wanting to upgrade to C99 or
> something. But maybe at 2060 or so they start to think about usefulness of
> UTF-8 too. The EBCDIC is usual, but irrelevant red herring in discussions like this.
>

I'm not going to read the whole of that spec - but it does say the
compiler by default reads EBCDIC encoded source files (with a switch for
ASCII) and it suggests text data files are EBCDIC too.

I haven't programmed for an EBCDIC machine in 30 years - in fact I
didn't learn C until I stopped using them - but that doesn't mean they
don't exist.

I don't see why they are a red herring.

It would be a perfectly valid decision to target Windows only, or
Android only, but if you want to be truly portable EBCDIC should always
be in the back of your mind.

Andy

On Wednesday, 12 January 2022 at 23:46:03 UTC+2, Vir Campestris wrote:
> On 11/01/2022 02:03, Öö Tiib wrote:
> > On Tuesday, 11 January 2022 at 00:09:48 UTC+2, Scott Lurndal wrote:
> >> Vir Campestris <vir.cam...@invalid.invalid> writes:
> >>> On 10/01/2022 00:51, Öö Tiib wrote:
> >>>> But do there exist machines that do want to support texts as char* but
> >>>> do not want to support UTF-8? Describe those machines, give examples..
> >>>
> >>> All the mainframes that run EBCDIC. There are a lot of them still.
> >> Here's the implementation guide for one:
> >>
> >> https://public.support.unisys.com/aseries/docs/ClearPath-MCP-18.0/86002268-207.pdf
> >>
> >> See appendix E for I18N.
> >
> > Seems out of context as these guys do not look like wanting to upgrade to C99 or
> > something. But maybe at 2060 or so they start to think about usefulness of
> > UTF-8 too. The EBCDIC is usual, but irrelevant red herring in discussions like this.
> >
> I'm not going to read the whole of that spec - but it does say the
> compiler by default reads EBCDIC encoded source files (with a switch for
> ASCII) and it suggests text data files are EBCDIC too.
>
> I haven't programmed for an EBCDIC machine in 30 years - in fact I the
> didn't learn C until I stopped using them - but that doesn't mean they
> don't exist.
>
> I don't see why they are a red herring.

I tried to tell. Because the vendors of those platforms do not want to migrate even to
more than 20 year old C99 standard.

> It would be a perfectly valid decision to target Windows only, or
> Android only, but if you want to be truly portable EBCDIC should always
> be in the back of your mind.

"Truly portable" is another red herring. Everything is portable only between
certain platforms & versions. Windows program that one writes now most likely
does not run on Windows 2000. Android programs released today have always
list of Android version that these run on. Portability is achieved with certain
amount of work per platform supported. Some old platforms are not worth to
invest that work in.

On 2022-01-13, Öö Tiib <ootiib@hot.ee> wrote:
> On Wednesday, 12 January 2022 at 23:46:03 UTC+2, Vir Campestris wrote:
>> It would be a perfectly valid decision to target Windows only, or
>> Android only, but if you want to be truly portable EBCDIC should always
>> be in the back of your mind.
>
> "Truly portable" is another red herring. Everything is portable only between

Indeed, "actually ported" beats "truly portable".

If you don't have a single test case in your system which covers EBCDIC,
or have written a few, but never had the opportunity to run them, then
the portability of the system to EBCDIC is only theoretical.

There could be snags in an actual porting effort that you have no clue
about.

When we write code, we have certain platforms in mind (and their
toolchains). Practical portable coding means having a few more kinds of
platforms in mind, without going overboard (such as ones that actually
exist and that some of the code might plausibly go to).

Where it is practical and easy, you make the coding decisions to be a
little more portable than actually required.

In the C language, a lot of this thinking is spent in not platform
concerns but compiler concerns. We have to worry a lot less about EBCDIC
or 36 bit pointers (which will never happen to our code) than, say,
about how deeply some future compiler makes inferences based on the
assumption of well-defined behavior. A C programmer's portability brain
cycles are much more profitably (or less unprofitably) spent on these
language semantic issues: as a strategist, you have to fortify yourself
where you expect to be attacked!

Strategically, worrying about EBCDIC is like building a fortress against
rocks and arrows, when your enemy has long abandoned those and moved on
to rockets and grenades. Well kind of. Imagine if there was a way of
fortifying that works gainst rockets and grenades, but succumbs to
arrows and rocks ... but arrows and rocks won't happen because almost
nobody remembers them. :)

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal