On Friday, 14 January 2022 at 16:18:11 UTC, Mateusz Viste wrote:
> 2022-01-14 at 15:43 +0100, Bonita Montero wrote:
> > Am 14.01.2022 um 15:10 schrieb Mateusz Viste:
> > > 2022-01-14 at 05:45 -0800, Malcolm McLean wrote:
> > >> Yes. It should compile cleanly. That's not the objection.
> > >
> > > clang with -Weverything (or -Wsign-conversion) does warn about this.
> > >
> > > warning: implicit conversion changes signedness: 'int' to 'size_t'
> > > (aka 'unsigned long') [-Wsign-conversion]
> >
> > Stupid warning. All compiler widen the data to the size of the
> > destination-type and then convert it to it.
> The warning points out a clear inconsistency. If that's *really* what
> the programmer wanted, then an explicit cast should clear out the
> warning. Otherwise it is likely to be a human error, and warning about
> that is never stupid.
>
If you are writing a utf8strlen() you have to do something is passed malformed
utf8. -1 is a standard "error return". Really you should return -2/-3. Keep -1
for "out of memory" errors, -2 or -3 for "parse errors", and -3 / -2 for
"IO errors". Those cover the vast majority of error conditions that can arise.

However size_t cannot represent -1, so instead the maximum value is returned.

2022-01-14 at 08:34 -0800, Malcolm McLean wrote:
> However size_t cannot represent -1, so instead the maximum value is
> returned.

Obviously, but it can still return "-1 casted to size_t", so the
caller may check for errors by doing the same cast. That's a convention
I have seen in actual code in the past. Whether or not it is good
practice and healthy design is another matter.

Mateusz

Am 14.01.2022 um 16:50 schrieb Bart:
> On 14/01/2022 14:43, Bonita Montero wrote:
>> Am 14.01.2022 um 15:10 schrieb Mateusz Viste:
>>> 2022-01-14 at 05:45 -0800, Malcolm McLean wrote:
>>>> Yes. It should compile cleanly. That's not the objection.
>>>
>>> clang with -Weverything (or -Wsign-conversion) does warn about this.
>>>
>>> warning: implicit conversion changes signedness: 'int' to 'size_t' (aka
>>> 'unsigned long') [-Wsign-conversion]
>>
>> Stupid warning. All compiler widen the data to the size of the
>> destination-type and then convert it to it.
>>
>
>
> If you change int -1 to size_t then you are liable to end up with
> 18446744073709551615.
> You don't want a warning about that?

No, because the upper value is intended.

Am 14.01.2022 um 17:17 schrieb Mateusz Viste:
> 2022-01-14 at 15:43 +0100, Bonita Montero wrote:
>> Am 14.01.2022 um 15:10 schrieb Mateusz Viste:
>>> 2022-01-14 at 05:45 -0800, Malcolm McLean wrote:
>>>> Yes. It should compile cleanly. That's not the objection.
>>>
>>> clang with -Weverything (or -Wsign-conversion) does warn about this.
>>>
>>> warning: implicit conversion changes signedness: 'int' to 'size_t'
>>> (aka 'unsigned long') [-Wsign-conversion]
>>
>> Stupid warning. All compiler widen the data to the size of the
>> destination-type and then convert it to it.
>
> The warning points out a clear inconsistency. If that's *really* what
> the programmer wanted, then an explicit cast should clear out the
> warning. Otherwise it is likely to be a human error, and warning about
> that is never stupid.

When s.o. casts -1 to an unsigned value he always intends that.
So this warning is superfluous.

2022-01-14 at 17:51 +0100, Bonita Montero wrote:
> Am 14.01.2022 um 17:17 schrieb Mateusz Viste:
> > The warning points out a clear inconsistency. If that's *really*
> > what the programmer wanted, then an explicit cast should clear out
> > the warning. Otherwise it is likely to be a human error, and
> > warning about that is never stupid.
>
> When s.o. casts -1 to an unsigned value he always intends that.

Look again at the original code. It states "return -1". No cast, other
than the implicit one due to the function being defined as returning an
unsigned value. The majority of situations where I have seen this were
human mistakes. Often because the function was initially supposed to
return a signed value and the programmer changed his mind later, but
forgot about the few negative returns he left behind.

> So this warning is superfluous.

You think that because you are used to working with reptilians. For us,
mere humans, such warnings help saving man-hours of code tracking.

Mateusz

Am 14.01.2022 um 18:07 schrieb Mateusz Viste:

> Look again at the original code. It states "return -1". No cast, other
> than the implicit one due to the function being defined as returning an
> unsigned value. The majority of situations where I have seen this were
> human mistakes. ...

No, this is never been done mistakenly, but intended.

On 14.01.22 15:07, Otto J. Makela wrote:
>
> How about this? It of course does not count the slightly odder
> compositions (like flags) correctly.
>

Why not just call mblen() in the loop instead?

Philipp Klaus Krause <pkk@spth.de> wrote:

> On 14.01.22 15:07, Otto J. Makela wrote:
>> How about this? It of course does not count the slightly odder
>> compositions (like flags) correctly.
>
> Why not just call mblen() in the loop instead?

Firstly, I will fully admit wasn't aware of its existance.

Secondly, on quick tests I can't seem to find the settings it needs to
be happy even with 2-byte characters. Apparently I don't understand it,
or my environment isn't correctly set up?

----

#include <stdio.h>
#include <stdlib.h>

int main() {
unsigned char str[10]="Ä";

printf("'%s' %02x %02x %02x %d\n",
str,str[0],str[1],str[3],mblen(str,10));
exit(0);
}

----

Produces output:

'Ä' c3 84 00 -1

--
/* * * Otto J. Makela <om@iki.fi> * * * * * * * * * */
/* Phone: +358 40 765 5772, ICBM: N 60 10' E 24 55' */
/* Mail: Mechelininkatu 26 B 27, FI-00100 Helsinki */
/* * * Computers Rule 01001111 01001011 * * * * * * */

On Saturday, 22 January 2022 at 09:44:52 UTC, Otto J. Makela wrote:
> Philipp Klaus Krause <p...@spth.de> wrote:
>
> > On 14.01.22 15:07, Otto J. Makela wrote:
> >> How about this? It of course does not count the slightly odder
> >> compositions (like flags) correctly.
> >
> > Why not just call mblen() in the loop instead?
> Firstly, I will fully admit wasn't aware of its existance.
>
> Secondly, on quick tests I can't seem to find the settings it needs to
> be happy even with 2-byte characters. Apparently I don't understand it,
> or my environment isn't correctly set up?
>
> ----
>
> #include <stdio.h>
> #include <stdlib.h>
>
> int main() {
> unsigned char str[10]="Ä";
>
> printf("'%s' %02x %02x %02x %d\n",
> str,str[0],str[1],str[3],mblen(str,10));
> exit(0);
> }
>
> ----
>
> Produces output:
>
> 'Ä' c3 84 00 -1
>
Something is wrong.
Try resetting the "shift state" with mblen(NULL, 0);

I doubt that's the problem, but it's worth a try.

On 1/22/22 7:48 AM, Malcolm McLean wrote:
> On Saturday, 22 January 2022 at 09:44:52 UTC, Otto J. Makela wrote:
>> Philipp Klaus Krause <p...@spth.de> wrote:
>>
>>> On 14.01.22 15:07, Otto J. Makela wrote:
>>>> How about this? It of course does not count the slightly odder
>>>> compositions (like flags) correctly.
>>>
>>> Why not just call mblen() in the loop instead?
>> Firstly, I will fully admit wasn't aware of its existance.
>>
>> Secondly, on quick tests I can't seem to find the settings it needs to
>> be happy even with 2-byte characters. Apparently I don't understand it,
>> or my environment isn't correctly set up?
>>
>> ----
>>
>> #include <stdio.h>
>> #include <stdlib.h>
>>
>> int main() {
>> unsigned char str[10]="Ä";
>>
>> printf("'%s' %02x %02x %02x %d\n",
>> str,str[0],str[1],str[3],mblen(str,10));
>> exit(0);
>> }
>>
>> ----
>>
>> Produces output:
>>
>> 'Ä' c3 84 00 -1
>>
> Something is wrong.
> Try resetting the "shift state" with mblen(NULL, 0);
>
> I doubt that's the problem, but it's worth a try.

Could it be that the library isn't in a 'Unicode' mode but an 'ASCII' mode.

I thought that mblen was locale dependent, and if the default 'C' local
defines the character set at 'ASCII', then the first byte of 0xC3 is in
fact, an invalid character.

One of the limits of the C language is that it does not default
(necessarily) to assuming Unicode, so the code may need an
implementation specific call to set the locale to one that uses Unicode,
if one exists (which isn't required).

om@iki.fi (Otto J. Makela) writes (non-ASCII characters guessed):
>#include <stdio.h>
>#include <stdlib.h>
>int main() {
> unsigned char str[10]="Ä";
> printf("'%s' %02x %02x %02x %d\n",
> str,str[0],str[1],str[3],mblen(str,10));
....
>'Ä' c3 84 00 -1

Here:

main.c

#include <stdint.h> /* SIZE_MAX */
#include <stdio.h> /* printf */
#include <stdlib.h> /* mblen */
#include <string.h> /* strlen */

int main( void )
{ char const s[ 10 ]= "Ä";
printf( "s = \"%s\"\n", s );
for( size_t i = 0; i < strlen( s )+ 1; ++i )
printf( "s[ %zu ]= %02X\n", i, 0xFF & s[ i ]);
printf( "mblen( s, SIZE_MAX )= %d\n", mblen( s, SIZE_MAX )); }

transcript

s = "Ä"
s[ 0 ]= C3
s[ 1 ]= 84
s[ 2 ]= 00
mblen( s, SIZE_MAX )= 1

On 1/22/22 04:44, Otto J. Makela wrote:
> #include <stdio.h>
> #include <stdlib.h>
>
> int main() {
> unsigned char str[10]="Ä";
>
> printf("'%s' %02x %02x %02x %d\n",
> str,str[0],str[1],str[3],mblen(str,10));
> exit(0);
> }

The set of valid extended characters is locale-dependent (5.2.1p1). In
the "C" locale on my system, there are none - only the basic character
set is supported. It's not clear to me that the C standard requires
this, but it certainly allows it.

You need to use setlocale() to select a locale that supports UTF-8.
Which locales are supported depend upon your implementation. On my
platform, the default locale (the one chosen if the second argument of
setlocale() is "") is determined by the LC_LANG environment, which I
have set to "en_US.UTF-8". When I insert setlocale(LC_CTYPE, "") in your
program before the mblen() call, I get the following results:

'Ä' c3 84 00 2

James Kuyper <jameskuyper@alumni.caltech.edu> wrote:

> The set of valid extended characters is locale-dependent (5.2.1p1). In
> the "C" locale on my system, there are none - only the basic character
> set is supported. It's not clear to me that the C standard requires
> this, but it certainly allows it.
>
> You need to use setlocale() to select a locale that supports UTF-8.
> Which locales are supported depend upon your implementation. On my
> platform, the default locale (the one chosen if the second argument of
> setlocale() is "") is determined by the LC_LANG environment, which I
> have set to "en_US.UTF-8". When I insert setlocale(LC_CTYPE, "") in
> your program before the mblen() call, I get the following results:
>
> 'Ä' c3 84 00 2

Yep, that setlocale() call did it. I actually have the semi-joke value
of LANG=en_DK.UTF-8 in my environment, produces good results for me.

https://unix.stackexchange.com/questions/62316/why-is-there-no-euro-english-locale
--
/* * * Otto J. Makela <om@iki.fi> * * * * * * * * * */
/* Phone: +358 40 765 5772, ICBM: N 60 10' E 24 55' */
/* Mail: Mechelininkatu 26 B 27, FI-00100 Helsinki */
/* * * Computers Rule 01001111 01001011 * * * * * * */

ram@zedat.fu-berlin.de (Stefan Ram) wrote:

> Here:

The end result is exactly the same as with my program,
mblen() returns -1.

James Kuyper <jameskuyper@alumni.caltech.edu> understood it, I need to
ask for the default locale. I'm a bit stumped as to why so.

--
/* * * Otto J. Makela <om@iki.fi> * * * * * * * * * */
/* Phone: +358 40 765 5772, ICBM: N 60 10' E 24 55' */
/* Mail: Mechelininkatu 26 B 27, FI-00100 Helsinki */
/* * * Computers Rule 01001111 01001011 * * * * * * */

On 1/27/22 06:33, Otto J. Makela wrote:
> ram@zedat.fu-berlin.de (Stefan Ram) wrote:
>
>> Here:
>
> The end result is exactly the same as with my program,
> mblen() returns -1.
>
> James Kuyper <jameskuyper@alumni.caltech.edu> understood it, I need to
> ask for the default locale. I'm a bit stumped as to why so.

Note: while I referred to it as the "default" locale, that's a bad term
for it, since the "C" locale is the true default:

"At program startup, the equivalent of
setlocale(LC_ALL, "C");
is executed." (7.11.1.1p4)

"... a value of "" for locale specifies the locale-specific native
environment." (7.11.1.1p3), so it would be better to call it the
"native" locale.

"In the "C" locale, isalpha returns true only for the characters for
which isupper or islower is true." (7.4.1.2p2)

"In the "C" locale, islower returns true only for the lowercase letters
(as defined in 5.2.1)." (7.4.1.7p2)

"In the "C" locale, isupper returns true only for the uppercase letters
(as defined in 5.2.1)." (7.4.1.11p2)

"... the 26 _uppercase letters_ of the Latin alphabet
A B C D E F G H I J K L M
N O P Q R S T U V W X Y Z

the 26 _lowercase letters_ of the Latin alphabet
a b c d e f g h i j k l m
n o p q r s t u v w x y z" (5.2.1p2)

Note that the phrases "uppercase letters" and "lowercase letters" are in
italics, an ISO convention indicating that the clause in which they
appear constitutes the official definition for those terms within this
document.

Thus, isalpha() can only be true in the "C" locale for the 26 letters of
the latin alphabet. That's not quite the same as requiring that those be
the only valid character encodings. Note, in particular, that it's not
possible to pass a member of the extended character set to isalpha() if
that member's encoding takes more than 1 byte, so nothing the standard
says about isalpha() can constrain how such characters are treated. Such
a character could be converted to wchar_t and passed to iswalpha(), but
that function has no special specifications for the "C" locale. However,
those facts certainly don't encourage support for additional valid
character encodings in the "C" locale.

James Kuyper <jameskuyper@alumni.caltech.edu> writes:

> On 1/27/22 06:33, Otto J. Makela wrote:
>
>> ram@zedat.fu-berlin.de (Stefan Ram) wrote:
>>
>>> Here:
>>
>> The end result is exactly the same as with my program,
>> mblen() returns -1.
>>
>> James Kuyper <jameskuyper@alumni.caltech.edu> understood it, I need to
>> ask for the default locale. I'm a bit stumped as to why so.
>
> Note: while I referred to it as the "default" locale, that's a bad
> term for it, since the "C" locale is the true default:
>
> "At program startup, the equivalent of
> setlocale(LC_ALL, "C");
> is executed." (7.11.1.1p4)
>
> "... a value of "" for locale specifies the locale-specific native
> environment." (7.11.1.1p3), so it would be better to call it the
> "native" locale.
>
> "In the "C" locale, isalpha returns true only for the characters for
> which isupper or islower is true." (7.4.1.2p2)
>
> "In the "C" locale, islower returns true only for the lowercase
> letters (as defined in 5.2.1)." (7.4.1.7p2)
>
> "In the "C" locale, isupper returns true only for the uppercase
> letters (as defined in 5.2.1)." (7.4.1.11p2)
>
> "... the 26 _uppercase letters_ of the Latin alphabet
> A B C D E F G H I J K L M
> N O P Q R S T U V W X Y Z
>
> the 26 _lowercase letters_ of the Latin alphabet
> a b c d e f g h i j k l m
> n o p q r s t u v w x y z" (5.2.1p2)
>
> Note that the phrases "uppercase letters" and "lowercase letters"
> are in italics, an ISO convention indicating that the clause in
> which they appear constitutes the official definition for those
> terms within this document.
>
> Thus, isalpha() can only be true in the "C" locale for the 26
> letters of the latin alphabet. [...]

What I think you mean is that in the "C" locale isalpha() can be
true only for the 26 letters of the latin alphabet.

Mateusz Viste <mateusz@xyz.invalid> writes:

> 2022-01-14 at 15:43 +0100, Bonita Montero wrote:
>
>> Am 14.01.2022 um 15:10 schrieb Mateusz Viste:
>>
>>> 2022-01-14 at 05:45 -0800, Malcolm McLean wrote:
>>>
>>>> Yes. It should compile cleanly. That's not the objection.
>>>
>>> clang with -Weverything (or -Wsign-conversion) does warn about
>>> this.
>>>
>>> warning: implicit conversion changes signedness: 'int' to 'size_t'
>>> (aka 'unsigned long') [-Wsign-conversion]
>>
>> Stupid warning. All compiler widen the data to the size of the
>> destination-type and then convert it to it.
>
> The warning points out a clear inconsistency. If that's *really*
> what the programmer wanted, then an explicit cast should clear out
> the warning. Otherwise it is likely to be a human error, and
> warning about that is never stupid.

Using a cast to avoid a warning for mixing signed and unsigned
values is a fairly common practice. I believe that practice
leads to worse results than simply leaving the code alone (unless
of course the warning does reveal a true problem, in which case
the problem should be fixed).

First, warnings for signed/unsigned mismatch yield more false
positives than true positives IME.

Second, in most cases a cast is an accident waiting to happen.
If subsequently a type is changed there is a chance that the cast
will convert to a type that is not appropriate for the context.

Third, if a type change does cause a cast to go wrong, then
having used a cast virtually guarantees that no warning will
be generated. Using a cast is like putting black electrical
tape over a blinking red warning light - you don't see the light
any more, but that doesn't mean there is no danger.

Fourth, in some cases mixing (or potentially mixing) signed and
unsigned types is almost inevitable. To give an example,
recently I had occasion to write code something like this:

Count r = -1;

while( r++, ..something.. ){
..whatever needed doing..
}

return r;

As it happens 'Count' was a typedef for an unsigned type, but it
may be a different type in different environments, including the
possibility of being a signed type at some point in the future.
But this code doesn't need to know that: the same code works
whether Count is a signed type or an unsigned type. Furthermore
that result is not just happenstance.

It's important to distinguish between unacceptable warnings and
advisory warnings. Some warning conditions give so few false
positives that it is advisable to treat them as absolute, just
the same as though they were errors. Other warning conditions
are more an indication that there might be a problem, but also
there is a good chance that there isn't. Mixing signed values and
unsigned values definitely falls into the latter category, and
should be treated accordingly.

Corollary: any case where a cast is used to avoid a warning should
be viewed with suspicion and a skeptical eye.