I want to pass utf8 encoded strings to the C runtime in Linux.
For instance create a file with fopen?
How to do this?

(
Windows it is working using using setlocale(LC_ALL, ".UTF8")
)

Thiago Adams <thiago.adams@gmail.com> writes:
>I want to pass utf8 encoded strings to the C runtime in Linux.

Assume that "utf8" is your UTF-8 encoded string and that "f"
is a function of the C runtime in Linux that takes a UTF-8
encoded string as its only argument. Then you need to have
this expression evaluated:

f( utf8 )

.

This is my test program.

#include <stdio.h>
#include <locale.h>
int main() {
setlocale(LC_ALL,"en_US.UTF - 8");
FILE* f = fopen(u8"maçã", "w");
if (f)
fclose(f);
}

It creates a file ma�� instead of maçã.

In article
<passing-utf8-encoded-strings-20220412214938@ram.dialup.fu-berlin
..de>,
ram@zedat.fu-berlin.de (Stefan Ram) wrote:

> Thiago Adams <thiago.adams@gmail.com> writes:
> >I want to pass utf8 encoded strings to the C runtime in Linux.
>
> Assume that "utf8" is your UTF-8 encoded string and that "f"
> is a function of the C runtime in Linux that takes a UTF-8
> encoded string as its only argument. Then you need to have
> this expression evaluated:
>
> f( utf8 )
>
> .

The point of UTF-8 is can use traditionally C functions, like
str*, without modification if you aren't interested in
identifying unicode code points. You can even do that with
minimal code ((c & 0xC0)==0x80) or something like that.

--
:-<> Siri Seal of Disavowal #000-001. Disavowed. Denied. Deleted. @
'I desire mercy, not sacrifice.' /|\
Discordia: not just a religion but also a parody. This post / \
I am an Andrea Doria sockpuppet. insults Islam. Mohammed

Thiago Adams <thiago.adams@gmail.com> writes:
>FILE* f = fopen(u8"maçã", "w");

I have no experience using UTF-8 with C, so I can only
guess:

Make sure to have your editor store this file with the
encoding your C implementation expects!

If this does not work, you might try to remove the "u8"
and store the file with the encoding UTF-8, or try

FILE * f = fopen( "ma\xc3\xa7\xc3\xa3", "w" );

and save it with US-ASCII.

On Tuesday, April 12, 2022 at 6:06:22 PM UTC-3, Siri Cruise wrote:
> In article
> <passing-utf8-encoded-...@ram.dialup.fu-berlin
> .de>,
> r...@zedat.fu-berlin.de (Stefan Ram) wrote:
> > Thiago Adams <thiago...@gmail.com> writes:
> > >I want to pass utf8 encoded strings to the C runtime in Linux.
> >
> > Assume that "utf8" is your UTF-8 encoded string and that "f"
> > is a function of the C runtime in Linux that takes a UTF-8
> > encoded string as its only argument. Then you need to have
> > this expression evaluated:
> >
> > f( utf8 )
> >
> > .
>
> The point of UTF-8 is can use traditionally C functions, like
> str*, without modification if you aren't interested in
>
I need fopen, rename, mkdir, opendir etc.. and the strings I
have are utf8 encoded.

In article <UTF-8-20220412220925@ram.dialup.fu-berlin.de>,
ram@zedat.fu-berlin.de (Stefan Ram) wrote:

> Thiago Adams <thiago.adams@gmail.com> writes:
> >FILE* f = fopen(u8"maçã", "w");
>
> I have no experience using UTF-8 with C, so I can only
> guess:
>
> Make sure to have your editor store this file with the
> encoding your C implementation expects!
>
> If this does not work, you might try to remove the "u8"
> and store the file with the encoding UTF-8, or try
>
> FILE * f = fopen( "ma\xc3\xa7\xc3\xa3", "w" );
>
> and save it with US-ASCII.

UTF8 has ASCII has a subset. If the high bit is clear, the UTF8
octet encodes a unicode character which is identical to ASCII of
the same octet. If the high bit is set, there is no simple way to
interpret the UTF8 octet as ASCII. Convert UTF8 octets to
unicode, Unicode does have some transformations that can wholly
or partially convert Unicode into pure ASCII. There are also
various translations that convert unicode subsets into single
octets such as the MacRoman encoding.

Some archaic Unix code vomitted on octets 7F - FF, but most
either internally convert to Unicode or pass through those octets
unchanged as just uninterpretted bytes.

--
:-<> Siri Seal of Disavowal #000-001. Disavowed. Denied. Deleted. @
'I desire mercy, not sacrifice.' /|\
Discordia: not just a religion but also a parody. This post / \
I am an Andrea Doria sockpuppet. insults Islam. Mohammed

On 12/04/2022 21:55, Thiago Adams wrote:
> This is my test program.
>
> #include <stdio.h>
> #include <locale.h>
> int main() {
> setlocale(LC_ALL,"en_US.UTF - 8");
> FILE* f = fopen(u8"maçã", "w");
> if (f)
> fclose(f);
> }
>
> It creates a file ma�� instead of maçã.

Try this:

#include <stdio.h>

int main(int argc, char *argv[])
{ char buf[100];

printf("Writing file...\n");
FILE *fp = fopen("output.txt", "w");
if (!fp)
{
perror("Output File open error!\n");
return 1;
}
fprintf(fp, "ma��\n");
fclose(fp);

printf("Reading file...\n");
fp = fopen("output.txt", "r");
if (!fp)
{
perror("Input File open error!\n");
return 1;
}
fgets(buf, sizeof(buf), fp);
printf("buf: %s...\n", buf);
fclose(fp);

return 0;
}

Thiago Adams <thiago.adams@gmail.com> writes:
>I want to pass utf8 encoded strings to the C runtime in Linux.
>For instance create a file with fopen?
>How to do this?
>
>(
> Windows it is working using using setlocale(LC_ALL, ".UTF8")
>)
>

Why do you believe that comp.lang.c is the proper venue for
this question; perhaps you should try comp.unix.programmer
instead?

Linux doesn't interpret the string of bytes provided as a
filename, other than the ascii '/' and nul characters.
If you provide an UTF-8 string to open, it will use that
string as the filename.

You'll probably want to ensure that you compile your code
with the locale set to the same locale that you set at
runtime to ensure any UTF strings in the C program
are interpreted by the compiler into the correct string
of UTF-8 bytes.

Thiago Adams <thiago.adams@gmail.com> writes:
>This is my test program.
>
>#include <stdio.h>
>#include <locale.h>
>int main() {
> setlocale(LC_ALL,"en_US.UTF - 8");
> FILE* f =3D fopen(u8"ma=C3=A7=C3=A3", "w");
> if (f)
> fclose(f);
>}
>
>It creates a file ma=EF=BF=BD=EF=BF=BD instead of ma=C3=A7=C3=A3.

Hopefully you didn't have spaces around the hyphen symbol in the
locale name 'en_us.UTF-8', like you do in the example code you
posted above.

On Tuesday, April 12, 2022 at 6:49:29 PM UTC-3, Sams Lara wrote:
> On 12/04/2022 21:55, Thiago Adams wrote:
> > This is my test program.
> >
> > #include <stdio.h>
> > #include <locale.h>
> > int main() {
> > setlocale(LC_ALL,"en_US.UTF - 8");
> > FILE* f = fopen(u8"maçã", "w");
> > if (f)
> > fclose(f);
> > }
> >
> > It creates a file ma�� instead of maçã.
> Try this:
>
> #include <stdio.h>
>
> int main(int argc, char *argv[])
> {
> char buf[100];
>
> printf("Writing file...\n");
> FILE *fp = fopen("output.txt", "w");
> if (!fp)
> {
> perror("Output File open error!\n");
> return 1;
> }
> fprintf(fp, "ma�ã\n");

The name of the file that is utf8 encoded, in my case.

Thiago Adams <thiago.adams@gmail.com> writes:
> This is my test program.
>
> #include <stdio.h>
> #include <locale.h>
> int main() {
> setlocale(LC_ALL,"en_US.UTF - 8");
> FILE* f = fopen(u8"maçã", "w");
> if (f)
> fclose(f);
> }
>
> It creates a file ma�� instead of maçã.

The name of the file it creates includes two occurrences of the
Unicode REPLACEMENT CHARACTER (fffd).

The valid values of the second argument to setlocale() are not
specified by the C standard. On my system, "en_US.UTF-8" is a valid
locale name. Why do you have spaces in yours?

What does setlocale() return? It returns a char* that is either a
pointer to a string or NULL if it was given an invalid locale
specification.

(I haven't been able to reproduce the behavior you describe on my
system, Ubuntu 20.04.)

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips
void Void(void) { Void(); } /* The recursive call of the void */

On Tuesday, April 12, 2022 at 6:59:22 PM UTC-3, Keith Thompson wrote:
> Thiago Adams <thiago...@gmail.com> writes:
> > This is my test program.
> >
> > #include <stdio.h>
> > #include <locale.h>
> > int main() {
> > setlocale(LC_ALL,"en_US.UTF - 8");
> > FILE* f = fopen(u8"maçã", "w");
> > if (f)
> > fclose(f);
> > }
> >
> > It creates a file ma�� instead of maçã.
> The name of the file it creates includes two occurrences of the
> Unicode REPLACEMENT CHARACTER (fffd).
>
> The valid values of the second argument to setlocale() are not
> specified by the C standard. On my system, "en_US.UTF-8" is a valid
> locale name. Why do you have spaces in yours?
>
> What does setlocale() return? It returns a char* that is either a
> pointer to a string or NULL if it was given an invalid locale
> specification.
>
> (I haven't been able to reproduce the behavior you describe on my
> system, Ubuntu 20.04.)

Considering your answer I tried some combinations and it worked.
Thanks.

I am using WSL. (Windows Subsystem for Linux.)

I saved the file using utf8 encoding.

Some locales didn't work on linux.. like ".UTF-8". (the locale with spaces where wrong when I copy pasted)
(This one ".UTF-8" works on windows when I compile with VC++.)

This is the program that worked on my computer. (Gcc)

#include <stdio.h>
#include <locale.h>

int main() {
const char* locale setlocale(LC_ALL,"en_US.UTF-8");

if (locale == NULL)
printf("locale is null\n");
else
printf("locale=%s\n",locale);

FILE* f = fopen(u8"maçã", "w");
if (f) {
fclose(f);
}
} The same program also worked on windows. (but originally I was doing ".UTF-8" on windows.)

These are the docs I am following.
Windows
https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/setlocale-wsetlocale?view=msvc-170

Linux
https://linux.die.net/man/3/setlocale

On linux tried the command
locale -a
and got:
C C.UTF-8
POSIX
en_US.utf8
so I think is is clear now. "C.UTF-8" also worked.

Am 13.04.22 um 00:23 schrieb Thiago Adams:

> On linux tried the command
> locale -a
> and got:
> C
> C.UTF-8
> POSIX
> en_US.utf8
> so I think is is clear now. "C.UTF-8" also worked.
>
>

I recommend to use C.UTF-8 here. It has a high chance of being present
on the user's system. en_US-stuff is unlikely to be available outside
the US.

Philipp

On 4/12/2022 10:55 PM, Thiago Adams wrote:
> This is my test program.
>
> #include <stdio.h>
> #include <locale.h>
> int main() {
> setlocale(LC_ALL,"en_US.UTF - 8");
> FILE* f = fopen(u8"maçã", "w");
> if (f)
> fclose(f);
> }
>
> It creates a file ma�� instead of maçã.

I don't get the same result, it works for me:

$ cc -std=c11 -O2 -Wall fopen-utf8.c && ./a.out && ls ma??
maçã

What is the console you are using?
Is there any chance MS is getting in the way?

On Wednesday, April 13, 2022 at 7:57:33 AM UTC-3, Manfred wrote:
> On 4/12/2022 10:55 PM, Thiago Adams wrote:
> > This is my test program.
> >
> > #include <stdio.h>
> > #include <locale.h>
> > int main() {
> > setlocale(LC_ALL,"en_US.UTF - 8");
> > FILE* f = fopen(u8"maçã", "w");
> > if (f)
> > fclose(f);
> > }
> >
> > It creates a file ma�� instead of maçã.
> I don't get the same result, it works for me:
>
> $ cc -std=c11 -O2 -Wall fopen-utf8.c && ./a.out && ls ma??
> maçã
>
> What is the console you are using?
Windows WSL

> Is there any chance MS is getting in the way?
Check if you have saved the file as utf8. Otherwise maybe don't use u8""
and put the ut8 chars directly.
Check locale -a in your linux.

On 2022-04-13, Thiago Adams <thiago.adams@gmail.com> wrote:
> On Wednesday, April 13, 2022 at 7:57:33 AM UTC-3, Manfred wrote:
>> On 4/12/2022 10:55 PM, Thiago Adams wrote:
>> > This is my test program.
>> >
>> > #include <stdio.h>
>> > #include <locale.h>
>> > int main() {
>> > setlocale(LC_ALL,"en_US.UTF - 8");
>> > FILE* f = fopen(u8"maçã", "w");
>> > if (f)
>> > fclose(f);
>> > }
>> >
>> > It creates a file ma�� instead of maçã.
>> I don't get the same result, it works for me:
>>
>> $ cc -std=c11 -O2 -Wall fopen-utf8.c && ./a.out && ls ma??
>> maçã
>>
>> What is the console you are using?
> Windows WSL
>
>> Is there any chance MS is getting in the way?
> Check if you have saved the file as utf8. Otherwise maybe don't use u8""
> and put the ut8 chars directly.
> Check locale -a in your linux.
>
>

I think you need to also need to set your locale in your shell:
cc -std=c11 -O2 -Wall fopen-utf8.c && LC_ALL="en_US.UTF-8" ls

peter

On 4/13/2022 1:42 PM, Thiago Adams wrote:
> On Wednesday, April 13, 2022 at 7:57:33 AM UTC-3, Manfred wrote:
>> On 4/12/2022 10:55 PM, Thiago Adams wrote:
>>> This is my test program.
>>>
>>> #include <stdio.h>
>>> #include <locale.h>
>>> int main() {
>>> setlocale(LC_ALL,"en_US.UTF - 8");
>>> FILE* f = fopen(u8"maçã", "w");
>>> if (f)
>>> fclose(f);
>>> }
>>>
>>> It creates a file ma�� instead of maçã.
>> I don't get the same result, it works for me:
>>
>> $ cc -std=c11 -O2 -Wall fopen-utf8.c && ./a.out && ls ma??
>> maçã
>>
>> What is the console you are using?
> Windows WSL

So, MS /is/ somehow around, however your answer is about a distrubution
(default Ubuntu, possibly customized by MS). But you did not answer the
relevant question: what is the console that you are using?

Note that you are complaining that the program creates a file with the
wrong name, but you look at that name through a console, so you should
verify that your console is displaying UTF-8 correctly: you might have
the correct name on disk, and have it wrong when displayed on screen.

>
>> Is there any chance MS is getting in the way?
> Check if you have saved the file as utf8. Otherwise maybe don't use u8""
> and put the ut8 chars directly.
> Check locale -a in your linux.
>

I get the correct result on my linux distro (Fedora 32, gnome-terminal),
with the program above, that you posted earlier, no change needed.

On Wednesday, April 13, 2022 at 10:56:32 AM UTC-3, Manfred wrote:
> On 4/13/2022 1:42 PM, Thiago Adams wrote:
> > On Wednesday, April 13, 2022 at 7:57:33 AM UTC-3, Manfred wrote:
> >> On 4/12/2022 10:55 PM, Thiago Adams wrote:
> >>> This is my test program.
> >>>
> >>> #include <stdio.h>
> >>> #include <locale.h>
> >>> int main() {
> >>> setlocale(LC_ALL,"en_US.UTF - 8");
> >>> FILE* f = fopen(u8"maçã", "w");
> >>> if (f)
> >>> fclose(f);
> >>> }
> >>>
> >>> It creates a file ma�� instead of maçã.
> >> I don't get the same result, it works for me:
> >>
> >> $ cc -std=c11 -O2 -Wall fopen-utf8.c && ./a.out && ls ma??
> >> maçã
> >>
> >> What is the console you are using?
> > Windows WSL
> So, MS /is/ somehow around, however your answer is about a distrubution
> (default Ubuntu, possibly customized by MS). But you did not answer the
> relevant question: what is the console that you are using?
>
> Note that you are complaining that the program creates a file with the
> wrong name, but you look at that name through a console, so you should
> verify that your console is displaying UTF-8 correctly: you might have
> the correct name on disk, and have it wrong when displayed on screen.

Yes. I checked against this problem because it is possible to see the same
file on the windows explorer with the correct name.
When we type wsl on windows it opens the terminal. So I think the answer
is that I am using the "default" terminal.

> >> Is there any chance MS is getting in the way?
> > Check if you have saved the file as utf8. Otherwise maybe don't use u8""
> > and put the ut8 chars directly.
> > Check locale -a in your linux.
> >
> I get the correct result on my linux distro (Fedora 32, gnome-terminal),
> with the program above, that you posted earlier, no change needed.

Thiago Adams <thiago.adams@gmail.com> writes:

> This is my test program.
>
> #include <stdio.h>
> #include <locale.h>
> int main() {
> setlocale(LC_ALL,"en_US.UTF - 8");
> FILE* f = fopen(u8"maçã", "w");
> if (f)
> fclose(f);
> }
>
> It creates a file ma�� instead of maçã.

I don't use WSL but in most Linux versions one would write

#include <stdio.h>

int main(void) {
FILE* f = fopen("maçã", "w");
if (f)
fclose(f);
}

--
Ben.

On Tue, 12 Apr 2022 13:55:23 -0700 (PDT), Thiago Adams
<thiago.adams@gmail.com> wrote:
> This is my test program.
[...]
> setlocale(LC_ALL,"en_US.UTF - 8");
> FILE* f = fopen(u8"maçã", "w");
[...]
> It creates a file ma�� instead of maçã.

As has been at least implied by others, the file showing up as that
probably has more to do with what you're listing the directory with
and what locale settings are there.

I might add that I don't think setlocale() does anything for pure C
string I/O. It'll become relevant eg. if you use wchar_t and functions
using wchar_t for I/O. If you feed u8"maçã" into fopen() the filename
will be those UTF-8 encoded bytes exactly regardless of locale settings.

(Assuming of course that your editor and compiler agree on what format
the data should be between the quotes...)

--
Mikko Rauhala - mjr@iki.fi - http://rauhala.org

Mikko Rauhala <mjr@iki.fi> writes:
> On Tue, 12 Apr 2022 13:55:23 -0700 (PDT), Thiago Adams
> <thiago.adams@gmail.com> wrote:
>> This is my test program.
> [...]
>> setlocale(LC_ALL,"en_US.UTF - 8");
>> FILE* f = fopen(u8"maçã", "w");
> [...]
>> It creates a file ma�� instead of maçã.
>
> As has been at least implied by others, the file showing up as that
> probably has more to do with what you're listing the directory with
> and what locale settings are there.

I think I mentioned before that the file name *in the article* consists of
"ma" followed by two occurrences of the Unicode REPLACEMENT CHARACTER, U+fffd.

But I don't know when or how those REPLACEMENT CHARACTERs were
generated. It's possible that they were generated when the file name
was pasted into the OP's newsreader.

[...]

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips
void Void(void) { Void(); } /* The recursive call of the void */

On Wednesday, April 13, 2022 at 1:45:11 PM UTC-3, Mikko Rauhala wrote:
> On Tue, 12 Apr 2022 13:55:23 -0700 (PDT), Thiago Adams
> <thiago...@gmail.com> wrote:
> > This is my test program.
> [...]
> > setlocale(LC_ALL,"en_US.UTF - 8");
> > FILE* f = fopen(u8"maçã", "w");
> [...]
> > It creates a file ma�� instead of maçã.
> As has been at least implied by others, the file showing up as that
> probably has more to do with what you're listing the directory with
> and what locale settings are there.

Yes, you are right. I repeated the test removing setlocale.
It worked.
So since the begging what was making the ma�� instead of maçã
was the encode of the source file. I changed the encode of the source file
and the problem repeat.
So, Linux filename seams "as it is"

Then I repeated the test on windows using two types of file encoding.
windows depends on set_locale. to work. So I thought the linux was the same...

On 4/13/22 03:11, Philipp Klaus Krause wrote:
> Am 13.04.22 um 00:23 schrieb Thiago Adams:
> ex
>> On linux tried the command
>> locale -a
>> and got:
>> C
>> C.UTF-8
>> POSIX
>> en_US.utf8
>> so I think is is clear now. "C.UTF-8" also worked.
>>
>>
>
> I recommend to use C.UTF-8 here. It has a high chance of being present
> on the user's system. en_US-stuff is unlikely to be available outside
> the US.

locale -a gives this result on my system:
C C.UTF-8
de_AT.utf8
de_BE.utf8
de_CH.utf8
de_DE.utf8
de_IT.utf8
de_LI.utf8
de_LU.utf8
en_AG
en_AG.utf8
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IL
en_IL.utf8
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZM
en_ZM.utf8
en_ZW.utf8
es_AR.utf8
es_BO.utf8
es_CL.utf8
es_CO.utf8
es_CR.utf8
es_CU
es_CU.utf8
es_DO.utf8
es_EC.utf8
es_ES.utf8
es_GT.utf8
es_HN.utf8
es_MX.utf8
es_NI.utf8
es_PA.utf8
es_PE.utf8
es_PR.utf8
es_PY.utf8
es_SV.utf8
es_US.utf8
es_UY.utf8
es_VE.utf8
POSIX
ru_RU.utf8
ru_UA.utf8
zh_CN.utf8
zh_HK.utf8
zh_SG.utf8
zh_TW.utf8

That's 48 different national locales, even though I only live in one of
them. I believe that they're standard Linux locales, and it just depends
upon which language packs you've got installed. English is one of the
most widely used languages, and "en_US.utf8" is probably one of
the most popular locales even outside the US.

In article <slrnt5dvg9.1ocu7.mjr@shadow.rauhala.org>,
Mikko Rauhala <mjr@iki.fi> wrote:

> > This is my test program.
> [...]
> > setlocale(LC_ALL,"en_US.UTF - 8");
> > FILE* f = fopen(u8"maçã", "w");
> [...]
> > It creates a file ma?? instead of maçã.
>
> As has been at least implied by others, the file showing up as that
> probably has more to do with what you're listing the directory with
> and what locale settings are there.

On unix, inside the kernel a path is a string of any possible
bytes except '\x00'; the byte '/' is given special
interpretation. A UTF8 byte string will have non-ASCII bytes, but
Unix kernels should have no difficulty. A kernel might convert a
path into a normal form NFC or NFD. Those can result in
apparently identical paths which are actually different. Outside
of the kernel, it's about how various software, terminal drivers,
windows text display, etc decide to do.

--
:-<> Siri Seal of Disavowal #000-001. Disavowed. Denied. Deleted. @
'I desire mercy, not sacrifice.' /|\
Discordia: not just a religion but also a parody. This post / \
I am an Andrea Doria sockpuppet. insults Islam. Mohammed