On Saturday, 29 July 2023 at 19:20:10 UTC+1, fir wrote:
> czwartek, 27 lipca 2023 o 20:50:09 UTC+2 Kaz Kylheku napisał(a):
> > On 2023-07-27, Malcolm McLean <malcolm.ar...@gmail.com> wrote:
> > > Lynn's comment inspired me to add Unicode support to the Baby X
> > > resource compiler. But despite searching for quite a long time, I
> > > can't find a test suite of Unicode files in various formats. In fact
> > > it's hard to find any Unicode files at all which are not UTF-8.
> > It's a fool's errand to support Unicode formats other than UTF-8.
> >
> dont know what fools errand mean but probbaly seems so, i dont remember waht
> i was sayin back then on this but probably if utf8 become the standard in use it is
> probably firtunate to stay with it, wchich is kinda fortunate (most comapatible in
> ascii
>
> btw if utf8 become de facto standard there is also defacto standard of csolo haracter?
>
> is this just 32 bit integer which is ascii below 127 and ofical number from unicode tables
> in above? i mean does things liek arrow left has one oficiall integer value?
>
There's an almost standard, which is the Unicode "code point". However whilst saying that
a code point is a character is good enough for most alphabets and texts, you do have
problems, like "pointing" in Hebrew texts (vowels and hard / soft signs are optional,
and indicated by small annotations around the letters). Unicode has a concept of
"combining charcters" to handle this sort of situation. Unfortunately it means that
Unicode-aware routines are quite hard to write is they are to handle everything correctly.

Bart <bc@freeuk.com> writes:

> In C11, "u8" is a valid user identifier, so that you can do this:
>
> typedef unsigned byte u8;
>
> But it can also be a prefix to a string literal. I believe it's called a
> 'contextual keyword'.

No, it's not called that. For the sake of simplicity, C does not give
any special name to something that is simply part of some other lexical
entity. Likewise, the L in L"abc" and 0x1L have no special name.

--
Ben.

sobota, 29 lipca 2023 o 20:55:42 UTC+2 Malcolm McLean napisał(a):
> On Saturday, 29 July 2023 at 19:20:10 UTC+1, fir wrote:
> > czwartek, 27 lipca 2023 o 20:50:09 UTC+2 Kaz Kylheku napisał(a):
> > > On 2023-07-27, Malcolm McLean <malcolm.ar...@gmail.com> wrote:
> > > > Lynn's comment inspired me to add Unicode support to the Baby X
> > > > resource compiler. But despite searching for quite a long time, I
> > > > can't find a test suite of Unicode files in various formats. In fact
> > > > it's hard to find any Unicode files at all which are not UTF-8.
> > > It's a fool's errand to support Unicode formats other than UTF-8.
> > >
> > dont know what fools errand mean but probbaly seems so, i dont remember waht
> > i was sayin back then on this but probably if utf8 become the standard in use it is
> > probably firtunate to stay with it, wchich is kinda fortunate (most comapatible in
> > ascii
> >
> > btw if utf8 become de facto standard there is also defacto standard of csolo haracter?
> >
> > is this just 32 bit integer which is ascii below 127 and ofical number from unicode tables
> > in above? i mean does things liek arrow left has one oficiall integer value?
> >
> There's an almost standard, which is the Unicode "code point". However whilst saying that
> a code point is a character is good enough for most alphabets and texts, you do have
> problems, like "pointing" in Hebrew texts (vowels and hard / soft signs are optional,
> and indicated by small annotations around the letters). Unicode has a concept of
> "combining charcters" to handle this sort of situation. Unfortunately it means that
> Unicode-aware routines are quite hard to write is they are to handle everything correctly.

so you mean that in most cases there is "odpowiedniosc" (how it is ine english? strict corespondence:
character <---> some 32 bit integer value ?

On Saturday, 29 July 2023 at 21:38:30 UTC+1, fir wrote:
> so you mean that in most cases there is "odpowiedniosc" (how it is ine english? strict corespondence:
> character <---> some 32 bit integer value ?
>
I think it depends which language you speak. For many people in the world,
Hebrew is something they would never use. For many Israelis, the pointing
system is something they rarely use and can do without. But for people interested
in Hebrew religious texts, it's essential, and they will need it every day.

But for a program written in a Latin alphabet based language, and with the
assumption that it might be translated as long as the burden isn't too high,
then saying that codepoints are C "characters" and UI "glyphs" is reasonable
enough.

sobota, 29 lipca 2023 o 22:48:46 UTC+2 Malcolm McLean napisał(a):
> On Saturday, 29 July 2023 at 21:38:30 UTC+1, fir wrote:
> > so you mean that in most cases there is "odpowiedniosc" (how it is ine english? strict corespondence:
> > character <---> some 32 bit integer value ?
> >
> I think it depends which language you speak. For many people in the world,
> Hebrew is something they would never use. For many Israelis, the pointing
> system is something they rarely use and can do without. But for people interested
> in Hebrew religious texts, it's essential, and they will need it every day.
>
> But for a program written in a Latin alphabet based language, and with the
> assumption that it might be translated as long as the burden isn't too high,
> then saying that codepoints are C "characters" and UI "glyphs" is reasonable
> enough.

this yeilds probably to a concept of encoded aray, if i is index on character and
if this haracters are 32 but int but the array is partialy 8bit and sometimes encoded

the question is if language should offer this type of array for unicode

unicode string textExample;
wide char = textExample[i];

then some iperators for this..

possibly this should be built in..once ago i think i opted for using 32 bit unicode
for simplicity - then no need to introduce such encoded arrays liek above,
but in fact maybe utf8 is better then some my cover it by introducing such kind of encoded
arrays

this btw shows that such acces as txt[i] is not optimal in case of such encoded arrays
and better to use something like txt.next txt.prev or something

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:

> On Saturday, 29 July 2023 at 21:38:30 UTC+1, fir wrote:
>> so you mean that in most cases there is "odpowiedniosc" (how it is ine english? strict corespondence:
>> character <---> some 32 bit integer value ?
>>
> I think it depends which language you speak. For many people in the
> world, Hebrew is something they would never use. For many Israelis,
> the pointing system is something they rarely use and can do
> without. But for people interested in Hebrew religious texts, it's
> essential, and they will need it every day.
>
> But for a program written in a Latin alphabet based language, and with
> the assumption that it might be translated as long as the burden isn't
> too high, then saying that codepoints are C "characters" and UI
> "glyphs" is reasonable enough.

I suppose it depends on what too high a burden is, but the routines
might still have to cope with things like combining accents. A simple
test. How many newsreaders get these two words right:

café café

Some software might need these two to compare equal, and some might want
them both the match "cafe" when a user does a search.

--
Ben.

On Saturday, 29 July 2023 at 22:22:08 UTC+1, Ben Bacarisse wrote:
> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>
> > On Saturday, 29 July 2023 at 21:38:30 UTC+1, fir wrote:
> >> so you mean that in most cases there is "odpowiedniosc" (how it is ine english? strict corespondence:
> >> character <---> some 32 bit integer value ?
> >>
> > I think it depends which language you speak. For many people in the
> > world, Hebrew is something they would never use. For many Israelis,
> > the pointing system is something they rarely use and can do
> > without. But for people interested in Hebrew religious texts, it's
> > essential, and they will need it every day.
> >
> > But for a program written in a Latin alphabet based language, and with
> > the assumption that it might be translated as long as the burden isn't
> > too high, then saying that codepoints are C "characters" and UI
> > "glyphs" is reasonable enough.
> I suppose it depends on what too high a burden is, but the routines
> might still have to cope with things like combining accents. A simple
> test. How many newsreaders get these two words right:
>
> café café
>
> Some software might need these two to compare equal, and some might want
> them both the match "cafe" when a user does a search.
>
A naive routine will match "café" and "cafe". But not "élan" and "elan". If you use
combining accents. And it will match neither if you use codepoint U-00E9. Whilst
if you speak French then a routine which will work for French is relatively simple
to write, the problem is that other languages have subtly different rules, and
it's a huge undertaking to support them all, or even a subset of likely customer
languages.
(We considered transalting our software, but rejected the idea as far too expensive
and problematic. Plus even Japanese customers don't seem to mind UIs with
small amounts of English text).

Bart <bc@freeuk.com> writes:
[...]
> In C11, "u8" is a valid user identifier, so that you can do this:
>
> typedef unsigned byte u8;
>
> But it can also be a prefix to a string literal. I believe it's called
> a 'contextual keyword'.

No, as Ben pointed out, it's just part of the syntax of a string
literal. The L in L"abc" doesn't conflict with L as an identifier.

[...]
> Actually, I can't see the point of such a prefix in C. All you have to
> do is to decree that string literals are UTF8 anyway.

Bad idea. It would restrict C's portability to systems that support
UTF-8, excluding both small embedded systems with limited character set
support and systems that still use EBCDIC.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

On 29/07/2023 22:52, Keith Thompson wrote:
> Bart <bc@freeuk.com> writes:
> [...]
>> In C11, "u8" is a valid user identifier, so that you can do this:
>>
>> typedef unsigned byte u8;
>>
>> But it can also be a prefix to a string literal. I believe it's called
>> a 'contextual keyword'.
>
> No, as Ben pointed out, it's just part of the syntax of a string
> literal. The L in L"abc" doesn't conflict with L as an identifier.
>
> [...]
>> Actually, I can't see the point of such a prefix in C. All you have to
>> do is to decree that string literals are UTF8 anyway.
>
> Bad idea. It would restrict C's portability to systems that support
> UTF-8, excluding both small embedded systems with limited character set
> support

It what way would be restrict those small systems?

Support for UTF8 means merely that characters in a string can have
values outside of 0 to 127, which they could have anyway whether UTF8 is
involved or not.

How exactly is a u8"€" string different from "€" anyway?

If your text editor saves as UTF8, then you will have a 3-byte sequence
plus nul in either case.

and systems that still use EBCDIC.
>

On Saturday, 29 July 2023 at 22:52:26 UTC+1, Keith Thompson wrote:
> Bart <b...@freeuk.com> writes:
> [...]
> > In C11, "u8" is a valid user identifier, so that you can do this:
> >
> > typedef unsigned byte u8;
> >
> > But it can also be a prefix to a string literal. I believe it's called
> > a 'contextual keyword'.
> No, as Ben pointed out, it's just part of the syntax of a string
> literal. The L in L"abc" doesn't conflict with L as an identifier.
>
> [...]
> > Actually, I can't see the point of such a prefix in C. All you have to
> > do is to decree that string literals are UTF8 anyway.
> Bad idea. It would restrict C's portability to systems that support
> UTF-8, excluding both small embedded systems with limited character set
> support and systems that still use EBCDIC.
>
By coincidence, fortunate or unfortunate, computers were first developed
in the United States, and English lends itself to representation by a sequence
of octets. But English is the exception. Most languages have some features
that make things difficult, whether that is accents or special forms for letters
in combination or at initial or end positions in words. (English does have
capitals, but that's the only real complication in English orthography).

But you can't win. Allow UTF-8 and the system breaks on small machines
or where the programmers haven't handled the special rules of every single
language out there. Ban it and the system breaks when you try to sell it to
the French.

The other thing is that most programmers can read English text, but not
most other languages. They might have a native language or studied one
foreign language at school. But the vast majority of foreign text might as
well be Linear A for all the use it is to most programmers. By restriciting
things to ASCII, you are discouraging the use of non-English in comments
and identifiers.

On 2023-07-29, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
> Bart <bc@freeuk.com> writes:
> [...]
>> In C11, "u8" is a valid user identifier, so that you can do this:
>>
>> typedef unsigned byte u8;
>>
>> But it can also be a prefix to a string literal. I believe it's called
>> a 'contextual keyword'.
>
> No, as Ben pointed out, it's just part of the syntax of a string
> literal. The L in L"abc" doesn't conflict with L as an identifier.
>
> [...]
>> Actually, I can't see the point of such a prefix in C. All you have to
>> do is to decree that string literals are UTF8 anyway.
>
> Bad idea. It would restrict C's portability to systems that support
> UTF-8, excluding both small embedded systems with limited character set
> support and systems that still use EBCDIC.

The downside if we define a string literal as UTF-8 is this: there are
invalid UTF-8 sequences, so we have lost the ability to dump arbitrary
bytes between the quotes. (Which we don't exactly have, since we can't
dump a double quote, backslash or newline, but I mean those bytes that
don't require escaping.)

If a string literals is defined as UTF-8, it means the compiler can
terminate with a diagnostic if there are invalid bytes, like an overlong
form, or a continuation byte without a start byte or whatever.

This is a problem for programs that have, say, ISO-Latin-1 string
literals; they have to be converted. Converting the code to UTF-8
may not be an option if the literal data is deliberately ISO-Latin-1,
because that's what the target system understands (e.g. can display
on its display).

If a string literal is restricted to UTF-8, then even numeric hex escape
sequences have to constituted valid UTF-8. (Otherwise what is the point
of the definition?)

That's going to cause a problem in any situation (small embedded
system or not) in which string literals are used for defining
special byte sequences for serial communication protocols and whatnot.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

On 7/29/23 5:52 PM, Keith Thompson wrote:
> Bart <bc@freeuk.com> writes:
> [...]
>> In C11, "u8" is a valid user identifier, so that you can do this:
>>
>> typedef unsigned byte u8;
>>
>> But it can also be a prefix to a string literal. I believe it's called
>> a 'contextual keyword'.
>
> No, as Ben pointed out, it's just part of the syntax of a string
> literal. The L in L"abc" doesn't conflict with L as an identifier.
>
> [...]
>> Actually, I can't see the point of such a prefix in C. All you have to
>> do is to decree that string literals are UTF8 anyway.
>
> Bad idea. It would restrict C's portability to systems that support
> UTF-8, excluding both small embedded systems with limited character set
> support and systems that still use EBCDIC.
>

Actually, it would give problems on todays computers, with files that
were written awhile ago using some "code-page" as its encoding (like
some version of Latin-1)

As I understand it, a u8"string" will convert the string from its source
character set (which might be set to Latin-1) into UTF-8, while a plain
"string" will convert it to what is defined as the "execution" character
set, which might be defined as the same as the input set of Latin-1.

Thus u8 strings MUST use UTF-8 encoding, while regular strings use
whatever encoding the implementation defines as its narrow character set.

There are reasons to allow then narrow character set to be something
other than a UTF-8 encoding.

On 2023-07-29, Bart <bc@freeuk.com> wrote:
> On 29/07/2023 22:52, Keith Thompson wrote:
>> Bad idea. It would restrict C's portability to systems that support
>> UTF-8, excluding both small embedded systems with limited character set
>> support
>
> It what way would be restrict those small systems?
>
> Support for UTF8 means merely that characters in a string can have
> values outside of 0 to 127, which they could have anyway whether UTF8 is
> involved or not.

If a spec says that the object specified by a string literal has to be
UTF-8, it means there cannot be non-UTF-8 sequences.

(If any byte string were allowed, what would be the point of mentioning
UTF-8.)

> How exactly is a u8"€" string different from "€" anyway?

That one probably isn't, other than the silly char8_t thing,
but I imagine u8"\x8F\xff" can be diagnosed.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

Bart <bc@freeuk.com> writes:
> On 29/07/2023 22:52, Keith Thompson wrote:
>> Bart <bc@freeuk.com> writes:
>> [...]
>>> In C11, "u8" is a valid user identifier, so that you can do this:
>>>
>>> typedef unsigned byte u8;
>>>
>>> But it can also be a prefix to a string literal. I believe it's called
>>> a 'contextual keyword'.
>> No, as Ben pointed out, it's just part of the syntax of a string
>> literal. The L in L"abc" doesn't conflict with L as an identifier.
>> [...]
>>> Actually, I can't see the point of such a prefix in C. All you have to
>>> do is to decree that string literals are UTF8 anyway.
>> Bad idea. It would restrict C's portability to systems that support
>> UTF-8, excluding both small embedded systems with limited character set
>> support
>
> It what way would be restrict those small systems?

I'm not sure. See below.

> Support for UTF8 means merely that characters in a string can have
> values outside of 0 to 127, which they could have anyway whether UTF8
> is involved or not.
>
> How exactly is a u8"€" string different from "€" anyway?

I haven't actually used UTF-8 string literals. As of C11/N1570, the
standard is a bit vague about the difference.

N1570 6.4.5p3:

A *character string literal* is a sequence of zero or more multibyte
characters enclosed in double-quotes, as in "xyz". A *UTF−8 string
literal* is the same, except prefixed by u8.

p6:

For character string literals, the array elements have type char,
and are initialized with the individual bytes of the multibyte
character sequence. For UTF−8 string literals, the array elements
have type char, and are initialized with the characters of the
multibyte character sequence, as encoded in UTF−8.

I would guess that something like u8"\xff", which specifies an invalid
UTF-8 sequence, would be invalid, but it doesn't seem to violate any
constraint or syntax rule, and neither gcc nor clang complains about it.

I guess that for a compiler that uses EBCDIC for source code, "x" would
be equivalent to "\xa7" (the EBCDIC code for 'x') and u8"x" would be
equivalent to "\x78".

Either the standard is insufficiently clear, or I'm missing something.
I definitely wouldn't bet against the latter.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

On 2023-07-30, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
> I would guess that something like u8"\xff", which specifies an invalid
> UTF-8 sequence, would be invalid, but it doesn't seem to violate any
> constraint or syntax rule, and neither gcc nor clang complains about it.

The requirement "encoded in UTF-8" could be regarded as a syntax rule.

> I guess that for a compiler that uses EBCDIC for source code, "x" would
> be equivalent to "\xa7" (the EBCDIC code for 'x') and u8"x" would be
> equivalent to "\x78".

If the u8 literal maps translation characters like 'x' to UTF-8,
that is actually very useful to the EBCDC people; they can directly
encode strings needed for communicating with ASCII systems.

E.g.

char *dial_prefix = u8"ATDT";

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

On 7/29/23 9:00 PM, Keith Thompson wrote:
> Bart <bc@freeuk.com> writes:
>> On 29/07/2023 22:52, Keith Thompson wrote:
>>> Bart <bc@freeuk.com> writes:
>>> [...]
>>>> In C11, "u8" is a valid user identifier, so that you can do this:
>>>>
>>>> typedef unsigned byte u8;
>>>>
>>>> But it can also be a prefix to a string literal. I believe it's called
>>>> a 'contextual keyword'.
>>> No, as Ben pointed out, it's just part of the syntax of a string
>>> literal. The L in L"abc" doesn't conflict with L as an identifier.
>>> [...]
>>>> Actually, I can't see the point of such a prefix in C. All you have to
>>>> do is to decree that string literals are UTF8 anyway.
>>> Bad idea. It would restrict C's portability to systems that support
>>> UTF-8, excluding both small embedded systems with limited character set
>>> support
>>
>> It what way would be restrict those small systems?
>
> I'm not sure. See below.
>
>> Support for UTF8 means merely that characters in a string can have
>> values outside of 0 to 127, which they could have anyway whether UTF8
>> is involved or not.
>>
>> How exactly is a u8"€" string different from "€" anyway?
>
> I haven't actually used UTF-8 string literals. As of C11/N1570, the
> standard is a bit vague about the difference.
>
> N1570 6.4.5p3:
>
> A *character string literal* is a sequence of zero or more multibyte
> characters enclosed in double-quotes, as in "xyz". A *UTF−8 string
> literal* is the same, except prefixed by u8.
>
> p6:
>
> For character string literals, the array elements have type char,
> and are initialized with the individual bytes of the multibyte
> character sequence. For UTF−8 string literals, the array elements
> have type char, and are initialized with the characters of the
> multibyte character sequence, as encoded in UTF−8.
>
> I would guess that something like u8"\xff", which specifies an invalid
> UTF-8 sequence, would be invalid, but it doesn't seem to violate any
> constraint or syntax rule, and neither gcc nor clang complains about it.
>
> I guess that for a compiler that uses EBCDIC for source code, "x" would
> be equivalent to "\xa7" (the EBCDIC code for 'x') and u8"x" would be
> equivalent to "\x78".
>
> Either the standard is insufficiently clear, or I'm missing something.
> I definitely wouldn't bet against the latter.
>

I think the key is that numerical constant specified characters are
specified in the "execution character encoding", which is implementation
defined, and doesn't need to be unicode.

Thus u8"\0xFF" might generate the byte sequence 0xC3 0xDF 0x00 to
represent the Unicode character U+00FF

On 30/07/2023 04:52, Richard Damon wrote:
> On 7/29/23 9:00 PM, Keith Thompson wrote:

>> I haven't actually used UTF-8 string literals. As of C11/N1570, the
>> standard is a bit vague about the difference.
>>
>> N1570 6.4.5p3:
>>
>> A *character string literal* is a sequence of zero or more
multibyte
>> characters enclosed in double-quotes, as in "xyz". A *UTF−8 string
>> literal* is the same, except prefixed by u8.
>>
>> p6:
>>
>> For character string literals, the array elements have type char,
>> and are initialized with the individual bytes of the multibyte
>> character sequence. For UTF−8 string literals, the array elements
>> have type char, and are initialized with the characters of the
>> multibyte character sequence, as encoded in UTF−8.
>>
>> I would guess that something like u8"\xff", which specifies an invalid
>> UTF-8 sequence, would be invalid, but it doesn't seem to violate any
>> constraint or syntax rule, and neither gcc nor clang complains about it.
>>
>> I guess that for a compiler that uses EBCDIC for source code, "x" would
>> be equivalent to "\xa7" (the EBCDIC code for 'x') and u8"x" would be
>> equivalent to "\x78".
>>
>> Either the standard is insufficiently clear, or I'm missing something.
>> I definitely wouldn't bet against the latter.
>>
>
> I think the key is that numerical constant specified characters are
> specified in the "execution character encoding", which is implementation
> defined, and doesn't need to be unicode.
>
> Thus u8"\0xFF" might generate the byte sequence 0xC3 0xDF 0x00 to
> represent the Unicode character U+00FF

(Do you mean \0xFF or \xFF ?)

I thought these were limited to 2 hex digits, but apparently you can
have as many as you like. However that looks to be ambiguous:

"\x20AC"

Is that a 4-digit hex code, or is it a 2-digit one following by normal
characters A and C?

i looked how this utf8 work, and some thoughts

I) in fact the decision to code 0-127 classic way seem to be ok
II) the decision to use abstract 32 bit numbers as a numbers of signs seem to be ok

iii) there is additional decision to encode that 128+ characters in a way each byte is then
128+ (for kinda compatibility i guess) as to this im not sure.. more saving space would
be use classical way something like
128 XX - additional 256 (or 255 if exclude 0)
129 XX - another additional 256 (or 255 if exclude 0)
200 XX XX - additional 256*256 (-256-1)
201 XX XX XX - additional

hovver maybe this rule 128+ is good also

iv) hovver if so there is a question how to do it

they chose the way all the 'tail' bytes encode witx "10" and 6 bytes (this is 128-191)
and the 'head' byte as 192+ .. this is probably right.. they could a) encode more values than 64
in tail bytes though or b) number the tail bytes to be seen which part of tail it is but
im not sure if the last is better (the tail-numbering would further eat the capacity)

as to first im not sure if the incrase tail capacityto more than 64 wouldnt be better, the maximum possible is to to have up do 127 capacily.. for 4 bytes it gave 28 bits against 24 bits, (25 bit seems standable) the problem is it couldnnt be all 128 values as some need to be marking head and head type.. there is yet a quetion if head should have capacity at all...some oculd assume that 128-191 is for tail and 192-255 for a heads but head shouldnt heve normal capacity

overally it all seem ok except mayeb the choice that heads teke this capacity.. there are 5 heads and each one has capacity from 5 to 1 bit... it may be usefull when using 2 byte encoding
gives 5+6=11 bits for 2 bytes 11 bits is 2k values wonder if they put good things there

for 4 bytes its 3+18 = 21.. maximum capacity for 6 bytes 31 bits

well overally it seems standable.. guess its in fact probably surprisingly easy to write c code which goes fo0rward by this unicode string and gives the wide char values - using masks shifts and ifs
and being proof to jumping inside wide char

overally the biggest problem was probably putting this confusion of providing all utf8 utf16 utf32
but if the practice is using mainly one on given system its ok.. buts imo probably the best to stick to that utf-8 where the asci is also used

niedziela, 30 lipca 2023 o 13:31:06 UTC+2 fir napisał(a):
> i looked how this utf8 work, and some thoughts
>
> I) in fact the decision to code 0-127 classic way seem to be ok
> II) the decision to use abstract 32 bit numbers as a numbers of signs seem to be ok
>
> iii) there is additional decision to encode that 128+ characters in a way each byte is then
> 128+ (for kinda compatibility i guess) as to this im not sure.. more saving space would
> be use classical way something like
> 128 XX - additional 256 (or 255 if exclude 0)
> 129 XX - another additional 256 (or 255 if exclude 0)
> 200 XX XX - additional 256*256 (-256-1)
> 201 XX XX XX - additional
>

this above in fact cant stand if one needs to identify tail and head but imo the alternative
for unicode coild be
0-127 ascii
0-247 tail (where this 247 is to be chosen)
248-255 heads (say eight, though also may be 4)

it then would be like
250 XX XX
251 XX XX XX
252 XX XX XX XX where capacity of XX is alomost full except that heads cant be used)

i think it is most concured pattaern to unicode, the advantage is it has more capacity
and no need to use shifts only soem need to be aware XXXXXX dont contain heads
so its not strictlu 0-0xffffff but not contains some 0xfxfxfx values but its not much a deal imo

niedziela, 30 lipca 2023 o 14:00:06 UTC+2 fir napisał(a):
> niedziela, 30 lipca 2023 o 13:31:06 UTC+2 fir napisał(a):
> > i looked how this utf8 work, and some thoughts
> >
> > I) in fact the decision to code 0-127 classic way seem to be ok
> > II) the decision to use abstract 32 bit numbers as a numbers of signs seem to be ok
> >
> > iii) there is additional decision to encode that 128+ characters in a way each byte is then
> > 128+ (for kinda compatibility i guess) as to this im not sure.. more saving space would
> > be use classical way something like
> > 128 XX - additional 256 (or 255 if exclude 0)
> > 129 XX - another additional 256 (or 255 if exclude 0)
> > 200 XX XX - additional 256*256 (-256-1)
> > 201 XX XX XX - additional
> >
> this above in fact cant stand if one needs to identify tail and head but imo the alternative
> for unicode coild be
> 0-127 ascii
> 0-247 tail (where this 247 is to be chosen)
> 248-255 heads (say eight, though also may be 4)
>
> it then would be like
> 250 XX XX
> 251 XX XX XX
> 252 XX XX XX XX where capacity of XX is alomost full except that heads cant be used)
>
> i think it is most concured pattaern to unicode, the advantage is it has more capacity
> and no need to use shifts only soem need to be aware XXXXXX dont contain heads
> so its not strictlu 0-0xffffff but not contains some 0xfxfxfx values but its not much a deal imo

for simplicity i even think 16 heads can be used to write
0xf2xxxx (where this f means head and 2 means 2 bytes of tail, and xx is from 0x00 to 0xef))
so this alternatixe unicode would look like f2 xx xx f3 xx xx xx xx xx xx f2 xx xx xx xx f3 xx xx xx xx xx
f1 rather should not be used (f0 and f1 reserved) and if someone jumps into string he need to revert back n byttes searching for fx code...
name this style maybe f-code im not sure if thsi is not better than unicode

niedziela, 30 lipca 2023 o 14:09:33 UTC+2 fir napisał(a):
> niedziela, 30 lipca 2023 o 14:00:06 UTC+2 fir napisał(a):
> > niedziela, 30 lipca 2023 o 13:31:06 UTC+2 fir napisał(a):
> > > i looked how this utf8 work, and some thoughts
> > >
> > > I) in fact the decision to code 0-127 classic way seem to be ok
> > > II) the decision to use abstract 32 bit numbers as a numbers of signs seem to be ok
> > >
> > > iii) there is additional decision to encode that 128+ characters in a way each byte is then
> > > 128+ (for kinda compatibility i guess) as to this im not sure.. more saving space would
> > > be use classical way something like
> > > 128 XX - additional 256 (or 255 if exclude 0)
> > > 129 XX - another additional 256 (or 255 if exclude 0)
> > > 200 XX XX - additional 256*256 (-256-1)
> > > 201 XX XX XX - additional
> > >
> > this above in fact cant stand if one needs to identify tail and head but imo the alternative
> > for unicode coild be
> > 0-127 ascii
> > 0-247 tail (where this 247 is to be chosen)
> > 248-255 heads (say eight, though also may be 4)
> >
> > it then would be like
> > 250 XX XX
> > 251 XX XX XX
> > 252 XX XX XX XX where capacity of XX is alomost full except that heads cant be used)
> >
> > i think it is most concured pattaern to unicode, the advantage is it has more capacity
> > and no need to use shifts only soem need to be aware XXXXXX dont contain heads
> > so its not strictlu 0-0xffffff but not contains some 0xfxfxfx values but its not much a deal imo
> for simplicity i even think 16 heads can be used to write
> 0xf2xxxx (where this f means head and 2 means 2 bytes of tail, and xx is from 0x00 to 0xef))
> so this alternatixe unicode would look like f2 xx xx f3 xx xx xx xx xx xx f2 xx xx xx xx f3 xx xx xx xx xx
> f1 rather should not be used (f0 and f1 reserved) and if someone jumps into string he need to revert back n byttes searching for fx code...
> name this style maybe f-code im not sure if thsi is not better than unicode

but utf8 is quite standable i guess ..maybe its even compatible with other bytes than 8-bit ones
which this f-code is not
but besides i think in todays time probably tehre is a need just to make architecture which makes bytes 32 bit not 8 bit, and then widechars should be stright

niedziela, 30 lipca 2023 o 14:09:33 UTC+2 fir napisał(a):
> > 252 XX XX XX XX where capacity of XX is alomost full except that heads cant be used)
> >
> > i think it is most concured pattaern to unicode, the advantage is it has more capacity
> > and no need to use shifts only soem need to be aware XXXXXX dont contain heads
> > so its not strictlu 0-0xffffff but not contains some 0xfxfxfx values but its not much a deal imo
> for simplicity i even think 16 heads can be used to write
> 0xf2xxxx (where this f means head and 2 means 2 bytes of tail, and xx is from 0x00 to 0xef))
> so this alternatixe unicode would look like f2 xx xx f3 xx xx xx xx xx xx f2 xx xx xx xx f3 xx xx xx xx xx

here above ofc the ones inside are ascii
(f2 xx xx) (f3 xx xx xx) (xx) (xx) (xx) (f2 xx xx) (xx) (xx) (f3 xx xx xx) (xx) (xx)

> f1 rather should not be used (f0 and f1 reserved) and if someone jumps into string he need to revert back n byttes searching for fx code...
> name this style maybe f-code im not sure if thsi is not better than unicode

niedziela, 30 lipca 2023 o 13:31:06 UTC+2 fir napisał(a):
> overally it all seem ok except mayeb the choice that heads teke this capacity.. there are 5 heads and each one has capacity from 5 to 1 bit... it may be usefull when using 2 byte encoding
> gives 5+6=11 bits for 2 bytes 11 bits is 2k values wonder if they put good things there
>
btw if so this means thets its maybe somewhat important to look up what is in this
chars (they probably name it code points but imo it could be named
wide chars or just chars / characters) up to 255 and up to 11 bits (2047)

as probably there is no reason to denay this wide character table, and if so this
become new ascii - so better to learn it probably

https://en.wikipedia.org/wiki/Latin-1_Supplement

sadly i dont see polisch letters ąę ćś źź ółń and big ones (18 in sum) in 256 set yet i see some real weird and not much of usefull imo, but soem usefll too

hell is the use of 0x8X or 0x9X controll codes does anytyhing really use it? (liek most of 0x0X
and 0x1X ) 60 characters liek wasted (or some would need to discover soem good use for this)

some of 0xAX 0xBX i could use

is then then real use of this 7 A ,4 E, 4 I, 6 O, 4 U ? (x2) is this real use of this if no i would more liek tosee moer additional punctuations there (but as i say no way to question it i guess)

only two blocks (0xAX , 0xBX) of punctuations, and mul and div signs ? then all letters

fortunatelly polish are among 256 - 383

but where is the rusiian?

niedziela, 30 lipca 2023 o 15:42:09 UTC+2 fir napisał(a):
> niedziela, 30 lipca 2023 o 13:31:06 UTC+2 fir napisał(a):
> > overally it all seem ok except mayeb the choice that heads teke this capacity.. there are 5 heads and each one has capacity from 5 to 1 bit... it may be usefull when using 2 byte encoding
> > gives 5+6=11 bits for 2 bytes 11 bits is 2k values wonder if they put good things there
> >
> btw if so this means thets its maybe somewhat important to look up what is in this
> chars (they probably name it code points but imo it could be named
> wide chars or just chars / characters) up to 255 and up to 11 bits (2047)
>
> as probably there is no reason to denay this wide character table, and if so this
> become new ascii - so better to learn it probably
>
> https://en.wikipedia.org/wiki/Latin-1_Supplement
>
> sadly i dont see polisch letters ąę ćś źź ółń and big ones (18 in sum) in 256 set yet i see some real weird and not much of usefull imo, but soem usefll too
>
> hell is the use of 0x8X or 0x9X controll codes does anytyhing really use it? (liek most of 0x0X
> and 0x1X ) 60 characters liek wasted (or some would need to discover soem good use for this)
>
> some of 0xAX 0xBX i could use
>
> is then then real use of this 7 A ,4 E, 4 I, 6 O, 4 U ? (x2) is this real use of this if no i would more liek tosee moer additional punctuations there (but as i say no way to question it i guess)
>
> only two blocks (0xAX , 0xBX) of punctuations, and mul and div signs ? then all letters
>
> fortunatelly polish are among 256 - 383
>
> but where is the rusiian?

overally a lot of letters and a lack of punctuationusefull things (though i seen mostly this 0-500
not seen exactly 500-2000 yet)

niedziela, 30 lipca 2023 o 15:50:44 UTC+2 fir napisał(a):
> niedziela, 30 lipca 2023 o 15:42:09 UTC+2 fir napisał(a):
> > niedziela, 30 lipca 2023 o 13:31:06 UTC+2 fir napisał(a):
> > > overally it all seem ok except mayeb the choice that heads teke this capacity.. there are 5 heads and each one has capacity from 5 to 1 bit... it may be usefull when using 2 byte encoding
> > > gives 5+6=11 bits for 2 bytes 11 bits is 2k values wonder if they put good things there
> > >
> > btw if so this means thets its maybe somewhat important to look up what is in this
> > chars (they probably name it code points but imo it could be named
> > wide chars or just chars / characters) up to 255 and up to 11 bits (2047)
> >
> > as probably there is no reason to denay this wide character table, and if so this
> > become new ascii - so better to learn it probably
> >
> > https://en.wikipedia.org/wiki/Latin-1_Supplement
> >
> > sadly i dont see polisch letters ąę ćś źź ółń and big ones (18 in sum) in 256 set yet i see some real weird and not much of usefull imo, but soem usefll too
> >
> > hell is the use of 0x8X or 0x9X controll codes does anytyhing really use it? (liek most of 0x0X
> > and 0x1X ) 60 characters liek wasted (or some would need to discover soem good use for this)
> >
> > some of 0xAX 0xBX i could use
> >
> > is then then real use of this 7 A ,4 E, 4 I, 6 O, 4 U ? (x2) is this real use of this if no i would more liek tosee moer additional punctuations there (but as i say no way to question it i guess)
> >
> > only two blocks (0xAX , 0xBX) of punctuations, and mul and div signs ? then all letters
> >
> > fortunatelly polish are among 256 - 383
> >
> > but where is the rusiian?
> overally a lot of letters and a lack of punctuationusefull things (though i seen mostly this 0-500
> not seen exactly 500-2000 yet)

ok there seem to be a bit more symbols about 700,800+ and rusiian is 1024+