On Wednesday, April 13, 2022 at 7:43:21 PM UTC-3, Siri Cruise wrote:
> In article <slrnt5dvg9...@shadow.rauhala.org>,
> Mikko Rauhala <m...@iki.fi> wrote:
>
> > > This is my test program.
> > [...]
> > > setlocale(LC_ALL,"en_US.UTF - 8");
> > > FILE* f = fopen(u8"maçã", "w");
> > [...]
> > > It creates a file ma?? instead of maçã.
> >
> > As has been at least implied by others, the file showing up as that
> > probably has more to do with what you're listing the directory with
> > and what locale settings are there.
> On unix, inside the kernel a path is a string of any possible
> bytes except '\x00'; the byte '/' is given special
> interpretation. A UTF8 byte string will have non-ASCII bytes, but
> Unix kernels should have no difficulty. A kernel might convert a
> path into a normal form NFC or NFD. Those can result in
> apparently identical paths which are actually different. Outside
> of the kernel, it's about how various software, terminal drivers,
> windows text display, etc decide to do.
> --
> :-<> Siri Seal of Disavowal #000-001. Disavowed. Denied. Deleted. @
> 'I desire mercy, not sacrifice.' /|\
> Discordia: not just a religion but also a parody. This post / \
> I am an Andrea Doria sockpuppet. insults Islam. Mohammed

Lets say I put a file called maçã.txt inside a pen drive. Then I copy
to the file to the linux filesystem. I expect to see the same name. So
it is hard to understand that file names are just array of chars no matter
what.

As Windows operates natively in UTF-16 (WCHAR).
"Until recently, Windows has emphasized "Unicode" -W variants over -A APIs.
However, recent releases have used the ANSI code page and -A APIs as a means to introduce
UTF-8 support to apps. If the ANSI code page is configured for UTF-8, -A
APIs operate in UTF-8. This model has the benefit of supporting existing
code built with -A APIs without any code changes."

https://docs.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page

I was expecting with these move Windows would become more like Linux
but now it is confused. I was expecting everting utf8 inside linux.
(unicode in some way)

Siri Cruise <chine.bleu@yahoo.com> wrote:
> In article <slrnt5dvg9.1ocu7.mjr@shadow.rauhala.org>,
> Mikko Rauhala <mjr@iki.fi> wrote:
>
> > > This is my test program.
> > [...]
> > > setlocale(LC_ALL,"en_US.UTF - 8");
> > > FILE* f = fopen(u8"ma????", "w");
> > [...]
> > > It creates a file ma?? instead of ma????.
> >
> > As has been at least implied by others, the file showing up as that
> > probably has more to do with what you're listing the directory with
> > and what locale settings are there.
>
> On unix, inside the kernel a path is a string of any possible
> bytes except '\x00'; the byte '/' is given special
> interpretation. A UTF8 byte string will have non-ASCII bytes, but
> Unix kernels should have no difficulty.

Kernel proper will transparently ship bytes to filesystem code.

> A kernel might convert a
> path into a normal form NFC or NFD. Those can result in
> apparently identical paths which are actually different. Outside
> of the kernel, it's about how various software, terminal drivers,
> windows text display, etc decide to do.

This is very impercise. OP was talking about WSL. I have no
idea how WSL handles files and what it will do with filenames.
Native Linux filesystems stores bytes from filenames "as is"
(except for '/' that separate parts of name and null terminator
at the end). Other filesystem may convert bytes to whatever
encoding is in use. Normally in Linux this is under control
of system configuration (including possible descriptor on
disc). So to have bytes stored correctly in file names
system must be properly configured. AFAIK Mac filesystem
mandates some wierdness. The same is likely to hold
for Windows filesystem(s). So it may be impossible to
have clean round trip

string => filename => string

OTOH if Linux inside WSL is properly configured for Unicode
round trip

filename => string => filename

should be clean.

--
Waldek Hebisch

Thiago Adams <thiago.adams@gmail.com> writes:
[...]
> Lets say I put a file called maçã.txt inside a pen drive.

Then the name of the file on the pen drive depends on what filesystem
the drive uses. FAT32 is most common, and it restricts which characters
can appear in file names (I don't know the details, but I've definitely
run into problems myself).

> Then I copy
> to the file to the linux filesystem.

Linux-based systems support a number of different filesystem
implementations. Most of them support all byte values other than '\0'
and '/' in file names.

> I expect to see the same name. So
> it is hard to understand that file names are just array of chars no matter
> what.

If you create a file named maçã.txt on a Unix-like system, the name in
the file system is probably (I think) going to consist of the bytes 'm',
'a', 0xc3, 0xa7, 0xc3, 0xa3, '.', 't', 'x', 't', the UTF-8
representation of that string. On NTFS, The file system doesn't care
whether it's valid UTF-8 or not. Other software layers might.

Of course the C standard says very little about any of this.

> As Windows operates natively in UTF-16 (WCHAR).
> "Until recently, Windows has emphasized "Unicode" -W variants over -A APIs.
> However, recent releases have used the ANSI code page and -A APIs as a means to introduce
> UTF-8 support to apps. If the ANSI code page is configured for UTF-8, -A
> APIs operate in UTF-8. This model has the benefit of supporting existing
> code built with -A APIs without any code changes."
>
> https://docs.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page
>
> I was expecting with these move Windows would become more like Linux
> but now it is confused. I was expecting everting utf8 inside linux.
> (unicode in some way)

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips
void Void(void) { Void(); } /* The recursive call of the void */

In article <877d7sfijq.fsf@nosuchdomain.example.com>,
Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:

> If you create a file named maçã.txt on a Unix-like system, the name in
> the file system is probably (I think) going to consist of the bytes 'm',
> 'a', 0xc3, 0xa7, 0xc3, 0xa3, '.', 't', 'x', 't', the UTF-8
> representation of that string. On NTFS, The file system doesn't care
> whether it's valid UTF-8 or not. Other software layers might.

I'm not sure but I Macos normalises to NFC.

--
:-<> Siri Seal of Disavowal #000-001. Disavowed. Denied. Deleted. @
'I desire mercy, not sacrifice.' /|\
Discordia: not just a religion but also a parody. This post / \
I am an Andrea Doria sockpuppet. insults Islam. Mohammed

Am 13.04.22 um 21:52 schrieb James Kuyper:

>
> locale -a gives this result on my system:
> C
> […]
>
> That's 48 different national locales, even though I only live in one of
> them. I believe that they're standard Linux locales, and it just depends
> upon which language packs you've got installed. English is one of the
> most widely used languages, and "en_US.utf8" is probably one of
> the most popular locales even outside the US.
>

On the system on which I am typing this:

philipp@notebook6:~$ locale -a
C C.UTF-8
de_DE.utf8
POSIX

I installed Debian GNU/Linux, chose German keyboard layout, and
Germany/Berlin timezone in the installer, and never bothered to install
additional locales manually on this system.

English might be used widely, but that doesn't mean the US locale is
installed. en_GB.utf8 is likely to be a popular choice, too. Not
however, that those that just want English error messages (e.g. for bug
reports) are likely to use the C or C.UTF-8 locale.

Philipp

On Thu, 2022-04-14, Philipp Klaus Krause wrote:
> Am 13.04.22 um 21:52 schrieb James Kuyper:
>
>>
>> locale -a gives this result on my system:
>> C
>> [???]
>>
>> That's 48 different national locales, even though I only live in one of
>> them. I believe that they're standard Linux locales, and it just depends
>> upon which language packs you've got installed. English is one of the
>> most widely used languages, and "en_US.utf8" is probably one of
>> the most popular locales even outside the US.
>>
>
> On the system on which I am typing this:
>
> philipp@notebook6:~$ locale -a
> C
> C.UTF-8
> de_DE.utf8
> POSIX
>
> I installed Debian GNU/Linux, chose German keyboard layout, and
> Germany/Berlin timezone in the installer, and never bothered to install
> additional locales manually on this system.

Mine is similar. I make a point of enabling a few sv_SE and en_US
locales, in case I want them. That's when doing an "expert" install,
though; people who install their systems and just say "I want GNOME
and everything" may get more locales.

Interestingly, my OpenBSD box has ~65 national locales, and they're
all named *.UTF-8, i.e. different from the Linux names. They
have "C", "POSIX" and "C.UTF-8" in common, though.

> English might be used widely, but that doesn't mean the US locale is
> installed. en_GB.utf8 is likely to be a popular choice, too. Not
> however, that those that just want English error messages (e.g. for bug
> reports) are likely to use the C or C.UTF-8 locale.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

On 13/04/2022 00:23, Thiago Adams wrote:
> On Tuesday, April 12, 2022 at 6:59:22 PM UTC-3, Keith Thompson wrote:
>> Thiago Adams <thiago...@gmail.com> writes:
>>> This is my test program.
>>>
>>> #include <stdio.h>
>>> #include <locale.h>
>>> int main() {
>>> setlocale(LC_ALL,"en_US.UTF - 8");
>>> FILE* f = fopen(u8"maçã", "w");
>>> if (f)
>>> fclose(f);
>>> }
>>>
>>> It creates a file ma�� instead of maçã.
>> The name of the file it creates includes two occurrences of the
>> Unicode REPLACEMENT CHARACTER (fffd).
>>
>> The valid values of the second argument to setlocale() are not
>> specified by the C standard. On my system, "en_US.UTF-8" is a valid
>> locale name. Why do you have spaces in yours?
>>
>> What does setlocale() return? It returns a char* that is either a
>> pointer to a string or NULL if it was given an invalid locale
>> specification.
>>
>> (I haven't been able to reproduce the behavior you describe on my
>> system, Ubuntu 20.04.)
>
> Considering your answer I tried some combinations and it worked.
> Thanks.
>
> I am using WSL. (Windows Subsystem for Linux.)
>
> I saved the file using utf8 encoding.
>
> Some locales didn't work on linux.. like ".UTF-8". (the locale with spaces where wrong when I copy pasted)
> (This one ".UTF-8" works on windows when I compile with VC++.)
>

Why are you setting the locale at all? For most purposes, strings are
just lists of characters ending in a null - whatever you put in the
string is used for the filename without any kind of translation or
re-encoding. The locale is only relevant for whatever program or
terminal you are using later to look at the name of the created file.

On Thursday, April 14, 2022 at 8:52:22 AM UTC-3, David Brown wrote:
> On 13/04/2022 00:23, Thiago Adams wrote:
> > On Tuesday, April 12, 2022 at 6:59:22 PM UTC-3, Keith Thompson wrote:
> >> Thiago Adams <thiago...@gmail.com> writes:
> >>> This is my test program.
> >>>
> >>> #include <stdio.h>
> >>> #include <locale.h>
> >>> int main() {
> >>> setlocale(LC_ALL,"en_US.UTF - 8");
> >>> FILE* f = fopen(u8"maçã", "w");
> >>> if (f)
> >>> fclose(f);
> >>> }
> >>>
> >>> It creates a file ma�� instead of maçã.
> >> The name of the file it creates includes two occurrences of the
> >> Unicode REPLACEMENT CHARACTER (fffd).
> >>
> >> The valid values of the second argument to setlocale() are not
> >> specified by the C standard. On my system, "en_US.UTF-8" is a valid
> >> locale name. Why do you have spaces in yours?
> >>
> >> What does setlocale() return? It returns a char* that is either a
> >> pointer to a string or NULL if it was given an invalid locale
> >> specification.
> >>
> >> (I haven't been able to reproduce the behavior you describe on my
> >> system, Ubuntu 20.04.)
> >
> > Considering your answer I tried some combinations and it worked.
> > Thanks.
> >
> > I am using WSL. (Windows Subsystem for Linux.)
> >
> > I saved the file using utf8 encoding.
> >
> > Some locales didn't work on linux.. like ".UTF-8". (the locale with spaces where wrong when I copy pasted)
> > (This one ".UTF-8" works on windows when I compile with VC++.)
> >
> Why are you setting the locale at all? For most purposes, strings are
> just lists of characters ending in a null - whatever you put in the
> string is used for the filename without any kind of translation or
> re-encoding. The locale is only relevant for whatever program or
> terminal you are using later to look at the name of the created file.

I am still trying to understand Linux... I understand Windows.
In windows file names are Unicode (encoded utf16). So using fopen to open a file
saved as "maçã" using fopen instead of _wfopen we need a code page or utf8 locale.
I was expecting Linux filenames to be unicode as well but maybe encoded with utf8
instead of utf16. Now I don't know how it works. If the files in linux are not unicode
just arrays of chars this would make each file name be different between computers
depending of linux internals!

On 4/14/2022 3:10 PM, Thiago Adams wrote:
> If the files in linux are not unicode
> just arrays of chars this would make each file name be different between computers
> depending of linux internals!

No it wouldn't, and it doesn't.
The "name" that identifies the file would be identical across the Linux
filesystems in use (say e.g. EXT-X variants).
What /might/ change is how that name is shown on a display (be it a
console, a window, widget or whatever) depending on the configuration of
such display - not on linux internals.
However, as many have repeated, the vast majority of Linux installations
default to UTF-8 for their UI configuration.
The filename representation in EXT filesystems (all bytes except '\0'
and '/') is inherently compatible with UTF-8.

You seem to consider a WSL installation equivalent to a native Linux
distribution.
Although I am not familiar with the details of WSL, this doesn't seem to
be correct.
For starter, a standard Linux distribution would definitely not default
to NTFS for their filesystem. Other technologies like EXT2,3,4 and XFS
are dominant in this area.
A major difference here is that NTFS uses UTF-16 (appearently unchecked)
for its filename representation, while EXT*, XFS and others use, as
said, all bytes except '\0' and '/', which is transparent to UTF-8 and
ASCII.

Thiago Adams <thiago.adams@gmail.com> writes:
[...]
> I am still trying to understand Linux... I understand Windows.
> In windows file names are Unicode (encoded utf16). So using fopen to open a file
> saved as "maçã" using fopen instead of _wfopen we need a code page or utf8 locale.
> I was expecting Linux filenames to be unicode as well but maybe encoded with utf8
> instead of utf16. Now I don't know how it works. If the files in linux are not unicode
> just arrays of chars this would make each file name be different between computers
> depending of linux internals!

The encoding of file names is (usually?) determined by the file system,
not by the operating system, though of course they interact with each
other. Windows uses NTFS by default these days, but it also supports
FAT32 and probably others.

If you copy a file from a Windows system to a Linux system, you're
creating a new file on the Linux system's file system (often ext4, but
there are other possibilities). Typically the code that copies the file
will take the Windows name (probably UTF-16), *translate* it to UTF-8
and use the resulting byte sequence as the name of the newly created
file.

It's also possible to share a file system across operating systems.
Both OS's are responsible for making that work consistently.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips
void Void(void) { Void(); } /* The recursive call of the void */

On Thursday, April 14, 2022 at 2:48:06 PM UTC-3, Keith Thompson wrote:
> Thiago Adams <thiago...@gmail.com> writes:
> [...]
> > I am still trying to understand Linux... I understand Windows.
> > In windows file names are Unicode (encoded utf16). So using fopen to open a file
> > saved as "maçã" using fopen instead of _wfopen we need a code page or utf8 locale.
> > I was expecting Linux filenames to be unicode as well but maybe encoded with utf8
> > instead of utf16. Now I don't know how it works. If the files in linux are not unicode
> > just arrays of chars this would make each file name be different between computers
> > depending of linux internals!
> The encoding of file names is (usually?) determined by the file system,
> not by the operating system, though of course they interact with each
> other. Windows uses NTFS by default these days, but it also supports
> FAT32 and probably others.
>
> If you copy a file from a Windows system to a Linux system, you're
> creating a new file on the Linux system's file system (often ext4, but
> there are other possibilities). Typically the code that copies the file
> will take the Windows name (probably UTF-16), *translate* it to UTF-8
> and use the resulting byte sequence as the name of the newly created
> file.

Right...but in this case can everyone agree that Linux uses Unicode and
it is encoded with utf8?

On 14/04/2022 18:59, Thiago Adams wrote:
> Right...but in this case can everyone agree that Linux uses Unicode and
> it is encoded with utf8?

Your best bet is to ask on the Linux Newsgroup. You don't have to
subscribe to that group. Just cross-post to that group and reply will
come on this newsgroup. Their newsgroup is:

"alt.os.linux"

I have done this here so hopefully somebody will reply.

Sams Lara <samlara622@gmail.com> writes:
> On 14/04/2022 18:59, Thiago Adams wrote:
>> Right...but in this case can everyone agree that Linux uses Unicode and
>> it is encoded with utf8?
>
> Your best bet is to ask on the Linux Newsgroup. You don't have to
> subscribe to that group. Just cross-post to that group and reply will
> come on this newsgroup. Their newsgroup is:
>
> "alt.os.linux"
>
> I have done this here so hopefully somebody will reply.

But you didn't provide enough context. We were talking about how file
names are encoded in a file system.

My understanding is that in most Linux file system implementations, file
names are uninterpreted byte sequences (excluding '/' and '\0') that are
commonly *displayed* as their UTF-8 interpretations. I *think* that you
could create a file whose name is not a valid UTF-8 string, and the file
system wouldn't have any problem with it, but commands like "ls" might
have to do something special. I haven't tried it.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips
void Void(void) { Void(); } /* The recursive call of the void */

On Thu, 14 Apr 2022 14:52:47 -0400, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
> Sams Lara <samlara622@gmail.com> writes:
>> On 14/04/2022 18:59, Thiago Adams wrote:
>>> Right...but in this case can everyone agree that Linux uses Unicode and
>>> it is encoded with utf8?
>>
>> Your best bet is to ask on the Linux Newsgroup. You don't have to
>> subscribe to that group. Just cross-post to that group and reply will
>> come on this newsgroup. Their newsgroup is:
>>
>> "alt.os.linux"
>>
>> I have done this here so hopefully somebody will reply.
>
> But you didn't provide enough context. We were talking about how file
> names are encoded in a file system.
>
> My understanding is that in most Linux file system implementations, file
> names are uninterpreted byte sequences (excluding '/' and '\0') that are
> commonly *displayed* as their UTF-8 interpretations. I *think* that you
> could create a file whose name is not a valid UTF-8 string, and the file
> system wouldn't have any problem with it, but commands like "ls" might
> have to do something special. I haven't tried it.

How they are interpreted for display depends on the locale setting of LC_CTYPE
(character type). It can be ascii, utf-8, utf-16, or utf-ebcdic.

Regards, Dave Hodgins

On Wed, 13 Apr 2022 21:58:24 -0700, Siri Cruise <chine.bleu@yahoo.com> wrote:
> I'm not sure but I Macos normalises to NFC.

It normalizes to NFD, which causes all sorts of compatibility problems.
(Except that it isn't even standard NFD, they make some exceptions for
Mac-Roman legacy support, showing that they were fully aware that their
choice was a compatibility nightmare.)

--
Mikko Rauhala - mjr@iki.fi - http://rauhala.org

On Thu, 14 Apr 2022 10:59:01 -0700 (PDT), Thiago Adams
<thiago.adams@gmail.com> wrote:
> On Thursday, April 14, 2022 at 2:48:06 PM UTC-3, Keith Thompson wrote:
>> The encoding of file names is (usually?) determined by the file system,
>> not by the operating system, though of course they interact with each
>> other. Windows uses NTFS by default these days, but it also supports
>> FAT32 and probably others.
>>
>> If you copy a file from a Windows system to a Linux system, you're
>> creating a new file on the Linux system's file system (often ext4, but
>> there are other possibilities). Typically the code that copies the file
>> will take the Windows name (probably UTF-16), *translate* it to UTF-8
>> and use the resulting byte sequence as the name of the newly created
>> file.
>
> Right...but in this case can everyone agree that Linux uses Unicode and
> it is encoded with utf8?

It's somewhat annoyingly only the de facto standard, but modern (GNU/)Linux
systems will use UTF-8 unless you wonk them to do something else yourself.
(Disclaimer: I'm not sure if distributions in CJK-using countries still
prefer their own encodings; it may well be.)

The complications here are that either your filesystem doesn't have any
opinion on the matter and will accept any (non-zero-byte-or-'/'-containing)
byte string, and then you don't have guarantees except GIGO. (Eg. ext[234].)

Or then your filesystem uses internally something well-specified, say,
UTF-16 (like NTFS/VFAT), and it'll have to be converted to something
POSIX API compatible _in practice per mount point_. Generally, by default,
this is UTF-8 these days.

But you can see how there might be complications if you have an UTF-8
mount and you try to use a non-UTF-8 locale in some software.

The current best solution to this impedance matching is indeed to just
de facto always use UTF-8 on the interface and forget the 8-bit legacy
encodings.

--
Mikko Rauhala - mjr@iki.fi - http://rauhala.org

On 2022-04-14, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
> Sams Lara <samlara622@gmail.com> writes:
>> On 14/04/2022 18:59, Thiago Adams wrote:
>>> Right...but in this case can everyone agree that Linux uses Unicode and
>>> it is encoded with utf8?
>>
>> Your best bet is to ask on the Linux Newsgroup. You don't have to
>> subscribe to that group. Just cross-post to that group and reply will
>> come on this newsgroup. Their newsgroup is:
>>
>> "alt.os.linux"
>>
>> I have done this here so hopefully somebody will reply.
>
> But you didn't provide enough context. We were talking about how file
> names are encoded in a file system.
>
> My understanding is that in most Linux file system implementations, file
> names are uninterpreted byte sequences (excluding '/' and '\0') that are
> commonly *displayed* as their UTF-8 interpretations. I *think* that you
> could create a file whose name is not a valid UTF-8 string, and the file
> system wouldn't have any problem with it, but commands like "ls" might
> have to do something special. I haven't tried it.

I just did try it. utf-8 is not a reqirement for filenames, any
sequence of bytes (except NUL and slash) will work.
the file I created shows as "zzz�@�X (invalid encoding)" in a GTK file
selector an as "'zzz'$'\377''@'$'\200''X" in ls output.

--
Jasen.