Mateusz Viste <mateusz@xyz.invalid> writes:
>While UTF-8 is neat, it is also complex to decode. Even a simple
>strlen() can be challenging.

In the case of US-ASCII strings that do not contain a NUL
character, strlen gives both the number of characters and
the size of the memory required (minus one).

Even the handling of US-ASCII strings can already become
difficult in C when they contain a NUL character.

In the case of UTF-8, the two aspects "memory size" and
"character count" might differ. It even becomes difficult
two define "character" (when several code points can be
combined to form one, er, glyph).

In C, "strlen" sometimes is used to merely learn about the
memory requirements, not the number of characters. In this
case, strlen still can be used as is because except for the
encoding of NUL, UTF-8 does not use a zero byte anywhere.

ram@zedat.fu-berlin.de (Stefan Ram) writes:

> It even becomes difficult two define "character" (when several code
> points can be combined to form one, er, glyph).

The distinction between character and glyph (text shaping, compositions,
and maybe even bidirectional reordering) is not really important for
most programmers.

IMO, it's best to use the usual definition of character in a multibyte
encoding (a single code point) unless you're specifically into what I
mentioned above.

On 1/9/22 12:59 AM, Po Lu wrote:
> ram@zedat.fu-berlin.de (Stefan Ram) writes:
>
>> It even becomes difficult two define "character" (when several code
>> points can be combined to form one, er, glyph).
>
> The distinction between character and glyph (text shaping, compositions,
> and maybe even bidirectional reordering) is not really important for
> most programmers.
>
> IMO, it's best to use the usual definition of character in a multibyte
> encoding (a single code point) unless you're specifically into what I
> mentioned above.

The issue is that with Unicode 'Combining Characters', where a glyph can
be represented as two code points, one being the 'Basic Character', and
the second being an accent added to it, means that almost all of the
advantages of using 'wide' characters is gone. You can no longer just
break the string at any character boundary and not need to worry about
affecting the presentation of the string.

If you aren't wanting to do that sort of operation, then multi-byte
encodings tend to be better than what the wide (and supposedly single
unit) encodings give you as it tends to be more compact.

Richard Damon <Richard@Damon-Family.org> writes:

> The issue is that with Unicode 'Combining Characters', where a glyph
> can be represented as two code points, one being the 'Basic
> Character', and the second being an accent added to it, means that
> almost all of the advantages of using 'wide' characters is gone. You
> can no longer just break the string at any character boundary and not
> need to worry about affecting the presentation of the string.

I would only worry about that if the string was going to be displayed to
the user at some point, and it was significantly likely to contain such
sequences. For most use cases, they aren't important enough to worry
about, IME.

> If you aren't wanting to do that sort of operation, then multi-byte
> encodings tend to be better than what the wide (and supposedly single
> unit) encodings give you as it tends to be more compact.

Agreed.

On Sunday, 9 January 2022 at 13:44:38 UTC, Po Lu wrote:
> Richard Damon <Ric...@Damon-Family.org> writes:
>
> > The issue is that with Unicode 'Combining Characters', where a glyph
> > can be represented as two code points, one being the 'Basic
> > Character', and the second being an accent added to it, means that
> > almost all of the advantages of using 'wide' characters is gone. You
> > can no longer just break the string at any character boundary and not
> > need to worry about affecting the presentation of the string.
> I would only worry about that if the string was going to be displayed to
> the user at some point, and it was significantly likely to contain such
> sequences. For most use cases, they aren't important enough to worry
> about, IME.
>
The problem with that approach is that things seem to work perfectly,
then break unexepectedly when someone passes a rare character like
ash (a and e combined, you see it mainly in archaic texts in English).

Computer text processing is full of these little glitches. Usually they
are just an irriation, but occasionally the costs can be high.

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:

> The problem with that approach is that things seem to work perfectly,
> then break unexepectedly when someone passes a rare character like
> ash (a and e combined, you see it mainly in archaic texts in English).

[...]

> Computer text processing is full of these little glitches. Usually they
> are just an irriation, but occasionally the costs can be high.

I agree with what you said above, but usually the irritation is rare
enough to be bearable.

I think your specific example is wrong, however: æ here shows up here as
U+00E6, a single code point.

Or did I miss something? Thanks.

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
> On Sunday, 9 January 2022 at 13:44:38 UTC, Po Lu wrote:
>> Richard Damon <Ric...@Damon-Family.org> writes:
>> > The issue is that with Unicode 'Combining Characters', where a glyph
>> > can be represented as two code points, one being the 'Basic
>> > Character', and the second being an accent added to it, means that
>> > almost all of the advantages of using 'wide' characters is gone. You
>> > can no longer just break the string at any character boundary and not
>> > need to worry about affecting the presentation of the string.
>> I would only worry about that if the string was going to be displayed to
>> the user at some point, and it was significantly likely to contain such
>> sequences. For most use cases, they aren't important enough to worry
>> about, IME.
>>
> The problem with that approach is that things seem to work perfectly,
> then break unexepectedly when someone passes a rare character like
> ash (a and e combined, you see it mainly in archaic texts in English).

I don't think that's a particularly good example. Unicode encodes 'Æ'
as 0xc6 and 'æ' as 0xe6. As far as I know they can't even be specified
using combining characters.

> Computer text processing is full of these little glitches. Usually they
> are just an irriation, but occasionally the costs can be high.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips
void Void(void) { Void(); } /* The recursive call of the void */

Then write your own UTF-8 strlen().

On 10/01/2022 02:59, Keith Thompson wrote:
> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>> On Sunday, 9 January 2022 at 13:44:38 UTC, Po Lu wrote:
>>> Richard Damon <Ric...@Damon-Family.org> writes:
>>>> The issue is that with Unicode 'Combining Characters', where a glyph
>>>> can be represented as two code points, one being the 'Basic
>>>> Character', and the second being an accent added to it, means that
>>>> almost all of the advantages of using 'wide' characters is gone. You
>>>> can no longer just break the string at any character boundary and not
>>>> need to worry about affecting the presentation of the string.
>>> I would only worry about that if the string was going to be displayed to
>>> the user at some point, and it was significantly likely to contain such
>>> sequences. For most use cases, they aren't important enough to worry
>>> about, IME.
>>>
>> The problem with that approach is that things seem to work perfectly,
>> then break unexepectedly when someone passes a rare character like
>> ash (a and e combined, you see it mainly in archaic texts in English).
>
> I don't think that's a particularly good example. Unicode encodes 'Æ'
> as 0xc6 and 'æ' as 0xe6. As far as I know they can't even be specified
> using combining characters.
>

It is also not that rare in some languages - it's a normal letter (the
27th in the alphabet) in Norwegian.

In English, you typically only see it some imported Latin terms, such as
"curriculum vitæ" or "dæmon" - and then usually only in more archaic
texts, as Malcolm said.

A more interesting example is the "i" with diaeresis, "ï". The common
example in English is the word "naïve", which is typically spelt with a
normal "i" these days since most US and UK keyboards make it extremely
inconvenient to type non-ASCII letters. (Both spellings are, I think,
valid forms in British English.)

What makes this particularly fun is that when made with combining
characters of "i" plus double-dot accent, the "i" glyph must be changed
significantly.

David Brown:

> A more interesting example is the "i" with diaeresis, ï".
> The common example in English is the word "naïve", which
> is typically spelt with a normal "i" these days since most
> US and UK keyboards make it extremely inconvenient to type
> non-ASCII letters. (Both spellings are, I think, valid
> forms in British English.)

I think these non-ASCII letters look out of place in
English. See paragraph I The Naturalization of Foreign
Words in:

Tract No. III of the Society for Pure English:
https://www.gutenberg.org/files/12390/12390-h/12390-h.htm

--
() ascii ribbon campaign - against html e-mail
/\ http://preview.tinyurl.com/qcy6mjc [archived]

2022-01-10 at 07:17 +0100, Bonita Montero wrote:
> Then write your own UTF-8 strlen().

That wouldn't be CPU efficient. At this point it might be saner to
implement a special utf-8 string type that keeps the string's
render-width in a separate field.

Mateusz

Am 10.01.2022 um 09:32 schrieb Mateusz Viste:
> 2022-01-10 at 07:17 +0100, Bonita Montero wrote:
>> Then write your own UTF-8 strlen().
>
> That wouldn't be CPU efficient. ...

I'll bet it woudln't be relevant slower than strlen()
because the memory-latency here is the limit.

Am 10.01.2022 um 11:26 schrieb Bonita Montero:
> Am 10.01.2022 um 09:32 schrieb Mateusz Viste:
>> 2022-01-10 at 07:17 +0100, Bonita Montero wrote:
>>> Then write your own UTF-8 strlen().
>>
>> That wouldn't be CPU efficient. ...

Ok, it would be slower, but I think this is the fastest possible
implementation (C++20):

size_t utf8Strlen( char const *str )
{ struct encode_t
{
size_t lenIncr, strIncr;
};
static encode_t const encodes[] =
{
{ 1, 1 },
{ 0, 0 },
{ 1, 2 },
{ 1, 3 },
{ 1, 4 },
{ 0, 0 },
{ 0, 0 },
{ 0, 0 },
{ 0, 0 }
};
size_t len = 0;
for( unsigned char c; (c = *str); )
{
encode_t const &enc = encodes[(size_t)countl_zero<unsigned char>( ~c )];
if( !enc.lenIncr ) [[unlikely]]
return -1;
len += enc.lenIncr;
str += enc.strIncr;
}
return len;
}

Am 10.01.2022 um 11:55 schrieb Bonita Montero:
> Am 10.01.2022 um 11:26 schrieb Bonita Montero:
>> Am 10.01.2022 um 09:32 schrieb Mateusz Viste:
>>> 2022-01-10 at 07:17 +0100, Bonita Montero wrote:
>>>> Then write your own UTF-8 strlen().
>>>
>>> That wouldn't be CPU efficient. ...
>
> Ok, it would be slower, but I think this is the fastest possible
> implementation (C++20):

This is more correct since it completely detects encoding-errors:

size_t utf8Strlen( char const *str )
{ struct encode_t { size_t lenIncr, strIncr; };
static encode_t const encodes[] =
{
{ 1, 1 },
{ 0, 0 },
{ 1, 2 },
{ 1, 3 },
{ 1, 4 },
{ 0, 0 },
{ 0, 0 },
{ 0, 0 },
{ 0, 0 }
};
size_t len = 0;
for( unsigned char c; (c = *str); )
{
encode_t const &enc = encodes[(size_t)countl_zero<unsigned
char>( ~c )];
if( !enc.lenIncr ) [[unlikely]]
return -1;
len += enc.lenIncr;
for( char const *cpEnd = str + enc.strIncr; ++str != cpEnd; )
if( ((unsigned char)*str & 0x0C0) != 0x080 ) [[unlikely]]
return -1;
}
return len;
}

Bonita Montero <Bonita.Montero@gmail.com> wrote:

> This is more correct since it completely detects encoding-errors:
>
> size_t utf8Strlen( char const *str )
> {
> struct encode_t { size_t lenIncr, strIncr; };
> static encode_t const encodes[] =
> {
> { 1, 1 },
> { 0, 0 },
> { 1, 2 },
> { 1, 3 },
> { 1, 4 },
> { 0, 0 },
> { 0, 0 },
> { 0, 0 },
> { 0, 0 }
> };
> size_t len = 0;
> for( unsigned char c; (c = *str); )
> {
> encode_t const &enc = encodes[(size_t)countl_zero<unsigned
> char>( ~c )];
> if( !enc.lenIncr ) [[unlikely]]
> return -1;
> len += enc.lenIncr;
> for( char const *cpEnd = str + enc.strIncr; ++str != cpEnd; )
> if( ((unsigned char)*str & 0x0C0) != 0x080 ) [[unlikely]]
> return -1;
> }
> return len;
> }

I must be getting a bit old here, I don't seem to understand
what that countl_zero stuff (and the syntax thereabouts) is?

PS. Also, shouldn't the first struct line be a typedef,
at least you seem to use it at bit later like it was?
PPS. And shouldn't const be before type, not after?
--
/* * * Otto J. Makela <om@iki.fi> * * * * * * * * * */
/* Phone: +358 40 765 5772, ICBM: N 60 10' E 24 55' */
/* Mail: Mechelininkatu 26 B 27, FI-00100 Helsinki */
/* * * Computers Rule 01001111 01001011 * * * * * * */

Bonita Montero <Bonita.Montero@gmail.com> wrote:

> This is more correct since it completely detects encoding-errors:
>
> size_t utf8Strlen( char const *str )
> {
> struct encode_t { size_t lenIncr, strIncr; };
> static encode_t const encodes[] =
> {
> { 1, 1 },
> { 0, 0 },
> { 1, 2 },
> { 1, 3 },
> { 1, 4 },
> { 0, 0 },
> { 0, 0 },
> { 0, 0 },
> { 0, 0 }
> };
> size_t len = 0;
> for( unsigned char c; (c = *str); )
> {
> encode_t const &enc = encodes[(size_t)countl_zero<unsigned
> char>( ~c )];
> if( !enc.lenIncr ) [[unlikely]]
> return -1;
> len += enc.lenIncr;
> for( char const *cpEnd = str + enc.strIncr; ++str != cpEnd; )
> if( ((unsigned char)*str & 0x0C0) != 0x080 ) [[unlikely]]
> return -1;
> }
> return len;
> }

Took me a while to realize the countl_zero stuff was a C++ library call,
with the accompanying syntax.

However, this is comp.lang.c, so C solutions are appreciated.
--
/* * * Otto J. Makela <om@iki.fi> * * * * * * * * * */
/* Phone: +358 40 765 5772, ICBM: N 60 10' E 24 55' */
/* Mail: Mechelininkatu 26 B 27, FI-00100 Helsinki */
/* * * Computers Rule 01001111 01001011 * * * * * * */

On Friday, 14 January 2022 at 12:18:42 UTC, Otto J. Makela wrote:
> Bonita Montero <Bonita....@gmail.com> wrote:
>
> > This is more correct since it completely detects encoding-errors:
> >
> > size_t utf8Strlen( char const *str )
> > {
> > struct encode_t { size_t lenIncr, strIncr; };
> > static encode_t const encodes[] =
> > {
> > { 1, 1 },
> > { 0, 0 },
> > { 1, 2 },
> > { 1, 3 },
> > { 1, 4 },
> > { 0, 0 },
> > { 0, 0 },
> > { 0, 0 },
> > { 0, 0 }
> > };
> > size_t len = 0;
> > for( unsigned char c; (c = *str); )
> > {
> > encode_t const &enc = encodes[(size_t)countl_zero<unsigned
> > char>( ~c )];
> > if( !enc.lenIncr ) [[unlikely]]
> > return -1;
> > len += enc.lenIncr;
> > for( char const *cpEnd = str + enc.strIncr; ++str != cpEnd; )
> > if( ((unsigned char)*str & 0x0C0) != 0x080 ) [[unlikely]]
> > return -1;
> > }
> > return len;
> > }
> I must be getting a bit old here, I don't seem to understand
> what that countl_zero stuff (and the syntax thereabouts) is?
>
> PS. Also, shouldn't the first struct line be a typedef,
> at least you seem to use it at bit later like it was?
> PPS. And shouldn't const be before type, not after?
>
Also, it would be safer to make the function return an int.
size_t is unsigned, and the return is -1 for error. So code like

size_t Nchars = utf8Strlen(utf8str);
size_t i;

for (i =0; i < Nchars; i++)
{ }

will run through quintillions of iterations, instead of being
a no-op, if the utf8 is malformed.

Am 14.01.2022 um 13:41 schrieb Malcolm McLean:
> On Friday, 14 January 2022 at 12:18:42 UTC, Otto J. Makela wrote:
>> Bonita Montero <Bonita....@gmail.com> wrote:
>>
>>> This is more correct since it completely detects encoding-errors:
>>>
>>> size_t utf8Strlen( char const *str )
>>> {
>>> struct encode_t { size_t lenIncr, strIncr; };
>>> static encode_t const encodes[] =
>>> {
>>> { 1, 1 },
>>> { 0, 0 },
>>> { 1, 2 },
>>> { 1, 3 },
>>> { 1, 4 },
>>> { 0, 0 },
>>> { 0, 0 },
>>> { 0, 0 },
>>> { 0, 0 }
>>> };
>>> size_t len = 0;
>>> for( unsigned char c; (c = *str); )
>>> {
>>> encode_t const &enc = encodes[(size_t)countl_zero<unsigned
>>> char>( ~c )];
>>> if( !enc.lenIncr ) [[unlikely]]
>>> return -1;
>>> len += enc.lenIncr;
>>> for( char const *cpEnd = str + enc.strIncr; ++str != cpEnd; )
>>> if( ((unsigned char)*str & 0x0C0) != 0x080 ) [[unlikely]]
>>> return -1;
>>> }
>>> return len;
>>> }
>> I must be getting a bit old here, I don't seem to understand
>> what that countl_zero stuff (and the syntax thereabouts) is?
>>
>> PS. Also, shouldn't the first struct line be a typedef,
>> at least you seem to use it at bit later like it was?
>> PPS. And shouldn't const be before type, not after?
>>
> Also, it would be safer to make the function return an int.
> size_t is unsigned, and the return is -1 for error. ...

All compilers support this because it is very common.

>
> size_t Nchars = utf8Strlen(utf8str);
> size_t i;
>
> for (i =0; i < Nchars; i++)
> {
> }
>
> will run through quintillions of iterations, instead of being
> a no-op, if the utf8 is malformed.

On Friday, 14 January 2022 at 13:43:16 UTC, Bonita Montero wrote:
> Am 14.01.2022 um 13:41 schrieb Malcolm McLean:
> > On Friday, 14 January 2022 at 12:18:42 UTC, Otto J. Makela wrote:
> >> Bonita Montero <Bonita....@gmail.com> wrote:
> >>
> >>> This is more correct since it completely detects encoding-errors:
> >>>
> >>> size_t utf8Strlen( char const *str )
> >>> {
> >>> struct encode_t { size_t lenIncr, strIncr; };
> >>> static encode_t const encodes[] =
> >>> {
> >>> { 1, 1 },
> >>> { 0, 0 },
> >>> { 1, 2 },
> >>> { 1, 3 },
> >>> { 1, 4 },
> >>> { 0, 0 },
> >>> { 0, 0 },
> >>> { 0, 0 },
> >>> { 0, 0 }
> >>> };
> >>> size_t len = 0;
> >>> for( unsigned char c; (c = *str); )
> >>> {
> >>> encode_t const &enc = encodes[(size_t)countl_zero<unsigned
> >>> char>( ~c )];
> >>> if( !enc.lenIncr ) [[unlikely]]
> >>> return -1;
> >>> len += enc.lenIncr;
> >>> for( char const *cpEnd = str + enc.strIncr; ++str != cpEnd; )
> >>> if( ((unsigned char)*str & 0x0C0) != 0x080 ) [[unlikely]]
> >>> return -1;
> >>> }
> >>> return len;
> >>> }
> >> I must be getting a bit old here, I don't seem to understand
> >> what that countl_zero stuff (and the syntax thereabouts) is?
> >>
> >> PS. Also, shouldn't the first struct line be a typedef,
> >> at least you seem to use it at bit later like it was?
> >> PPS. And shouldn't const be before type, not after?
> >>
> > Also, it would be safer to make the function return an int.
> > size_t is unsigned, and the return is -1 for error. ...
>
> All compilers support this because it is very common.
>
Yes. It should compile cleanly. That's not the objection.

om@iki.fi (Otto J. Makela) wrote:

> Took me a while to realize the countl_zero stuff was a C++ library
> call, with the accompanying syntax.
>
> However, this is comp.lang.c, so C solutions are appreciated.

How about this? It of course does not count the slightly odder
compositions (like flags) correctly.

----

int utf8Strlen( char const *str ) {
int length = 0;
int multibyte = 0;
unsigned char c;

while(c = *str++) {
if(multibyte) {
multibyte--;
if((c & 0xC0) != 0x80 ) // UTF8 encoding error
return -1;
} else {
length++;
if (c < 0x80) // Normal 7-bit ASCII
multibyte = 0;
else if ((c & 0xE0) == 0xC0) // Single following byte
multibyte = 1;
else if ((c & 0xF0) == 0xE0) // Two following bytes
multibyte = 2;
else if ((c & 0xF8) == 0xF0) // Three following bytes
multibyte = 3;
else // UTF8 encoding error
return -1;
}
}

if(multibyte) // Ran out of string in the middle of multibyte character
return -1;
return length;
}

--
/* * * Otto J. Makela <om@iki.fi> * * * * * * * * * */
/* Phone: +358 40 765 5772, ICBM: N 60 10' E 24 55' */
/* Mail: Mechelininkatu 26 B 27, FI-00100 Helsinki */
/* * * Computers Rule 01001111 01001011 * * * * * * */

2022-01-14 at 05:45 -0800, Malcolm McLean wrote:
> Yes. It should compile cleanly. That's not the objection.

clang with -Weverything (or -Wsign-conversion) does warn about this.

warning: implicit conversion changes signedness: 'int' to 'size_t' (aka
'unsigned long') [-Wsign-conversion]

Mateusz

Am 14.01.2022 um 15:07 schrieb Otto J. Makela:
> om@iki.fi (Otto J. Makela) wrote:
>
>> Took me a while to realize the countl_zero stuff was a C++ library
>> call, with the accompanying syntax.
>>
>> However, this is comp.lang.c, so C solutions are appreciated.
>
> How about this? It of course does not count the slightly odder
> compositions (like flags) correctly.
>
> ----
>
> int utf8Strlen( char const *str ) {
> int length = 0;
> int multibyte = 0;
> unsigned char c;
>
> while(c = *str++) {
> if(multibyte) {
> multibyte--;
> if((c & 0xC0) != 0x80 ) // UTF8 encoding error
> return -1;
> } else {
> length++;
> if (c < 0x80) // Normal 7-bit ASCII
> multibyte = 0;
> else if ((c & 0xE0) == 0xC0) // Single following byte
> multibyte = 1;
> else if ((c & 0xF0) == 0xE0) // Two following bytes
> multibyte = 2;
> else if ((c & 0xF8) == 0xF0) // Three following bytes
> multibyte = 3;
> else // UTF8 encoding error
> return -1;
> }
> }
>
> if(multibyte) // Ran out of string in the middle of multibyte character
> return -1;
> return length;
> }
>

A lot of unpredictible branches.
That's what I want to prevent with the table.

Am 14.01.2022 um 15:10 schrieb Mateusz Viste:
> 2022-01-14 at 05:45 -0800, Malcolm McLean wrote:
>> Yes. It should compile cleanly. That's not the objection.
>
> clang with -Weverything (or -Wsign-conversion) does warn about this.
>
> warning: implicit conversion changes signedness: 'int' to 'size_t' (aka
> 'unsigned long') [-Wsign-conversion]

Stupid warning. All compiler widen the data to the size of the
destination-type and then convert it to it.

On 14/01/2022 14:43, Bonita Montero wrote:
> Am 14.01.2022 um 15:10 schrieb Mateusz Viste:
>> 2022-01-14 at 05:45 -0800, Malcolm McLean wrote:
>>> Yes. It should compile cleanly. That's not the objection.
>>
>> clang with -Weverything (or -Wsign-conversion) does warn about this.
>>
>> warning: implicit conversion changes signedness: 'int' to 'size_t' (aka
>> 'unsigned long') [-Wsign-conversion]
>
> Stupid warning. All compiler widen the data to the size of the
> destination-type and then convert it to it.
>

If you change int -1 to size_t then you are liable to end up with
18446744073709551615.

You don't want a warning about that?

2022-01-14 at 15:43 +0100, Bonita Montero wrote:
> Am 14.01.2022 um 15:10 schrieb Mateusz Viste:
> > 2022-01-14 at 05:45 -0800, Malcolm McLean wrote:
> >> Yes. It should compile cleanly. That's not the objection.
> >
> > clang with -Weverything (or -Wsign-conversion) does warn about this.
> >
> > warning: implicit conversion changes signedness: 'int' to 'size_t'
> > (aka 'unsigned long') [-Wsign-conversion]
>
> Stupid warning. All compiler widen the data to the size of the
> destination-type and then convert it to it.

The warning points out a clear inconsistency. If that's *really* what
the programmer wanted, then an explicit cast should clear out the
warning. Otherwise it is likely to be a human error, and warning about
that is never stupid.

Mateusz