muta...@gmail.com <mutazilah@gmail.com> wrote:
> On Friday, April 2, 2021 at 10:37:22 AM UTC+11, Keith Thompson wrote:
>
> > >> Have you asked any Vietnamese people what they want? I haven't,
> > >
> > > No, they probably don't realize what just happened
> > > to them anyway. I'm on the inside. I know what we
> > > did to them.
> > >
> > >> but my (admittedly limited) understanding is that UTF-8 works
> > >> fine for them.
> > >
> > > They haven't been exposed to a situation where every
> > > byte on a floppy disk counts and every cycle counts.
>
> > They haven't? No Vietnamese people have used old floppy-based computer
> > systems? You've checked?
>
> I don't have access to every Vietnamese person who
> has ever lived. The ones that I know (who I'm paying
> to work on PDOS) are way too young to have ever
> used a floppy-based computer.

I can you a bit about experience of my country. To express
national character we need 18 extra positions beyond ASCII
(9 for small letters, 9 for upper case version). In nineties
we had about 20 different coding systems. Some re-purposed
ASCII (treating some character as escape), so that they worked
even if system mangled highest bit. Most were 8-bit. Some
worked hard to preserve "graphic" PC characters, so that full
set of box drawing were available together with national
characters. Today most of them is dying, there are utilites
to recode to other formats and some old files, but normal
use is centerd around few ones, with UTF-8 having clear
lead. People did little statistic and it turned out that
national characters account for about 5% of the text, so
using UTF-8 means expansion of about 5% (because in UTF-8
they need 2 bytes) compared to single byte encoding.
This increase does not matter for most folks. Given
compression, ASCII based computer formats and binary files
increase in size is much smaller than 5%.

> > >> I suspect they value easy interopability with
> > >> non-Vietnamese text.
> > >
> > > For as long as we have the hardware available to
> > > throw at the problem, they won't notice.
>
> > Which is why they probably don't care. (I'm speculating, not speaking
> > for anyone.)
>
> Yes, but I care, even if they don't. They probably aren't
> hedging their bets against a time warp, but I am. This
> guy is hedging against something similar:
>
> https://collapseos.org/
>
> > > Only VISCII is of any use (in my opinion, obviously),
> > > but they snaffled some control characters that I'm
> > > not willing to release because it interferes with my
> > > micro-emacs keystrokes. I want to be able to ship
> > > micro-emacs and say "press ctrl-e to go to the end
> > > of the line" and I expect it to work and not generate
> > > some strange hieroglyph.
>
> > I'm pretty sure you can do that already. (I'm assuming some version of
> > micro-emacs supports UTF-8 and/or something like VSCII.)
>
> I don't know that micro-emacs can do that, but
> regardless, I'm not interested in using 2 bytes
> when only 1 byte is required.

But you are telling folks in 1965 that they should use 4 bytes,
when 3 were enough?

--
Waldek Hebisch

On Sunday, May 2, 2021 at 10:08:40 AM UTC+10, anti...@math.uni.wroc.pl wrote:

> to recode to other formats and some old files, but normal
> use is centerd around few ones, with UTF-8 having clear
> lead. People did little statistic and it turned out that
> national characters account for about 5% of the text, so
> using UTF-8 means expansion of about 5% (because in UTF-8
> they need 2 bytes) compared to single byte encoding.
> This increase does not matter for most folks. Given
> compression, ASCII based computer formats and binary files
> increase in size is much smaller than 5%.

There is a performance hit using UTF-8 like that.
No reason to take a performance hit IMO.

> > I don't know that micro-emacs can do that, but
> > regardless, I'm not interested in using 2 bytes
> > when only 1 byte is required.

> But you are telling folks in 1965 that they should use 4 bytes,
> when 3 were enough?

That's a good question.

My main concern is them putting crap into the top byte,
expecting it to be ignored forever, requiring a specific
mode to ignore it.

If instead everyone was doing:

SLR R5,R5
ICM B'0111',ADDR+1

to instead just 3 bytes into a register, with the top byte
guaranteed to be 0, then no special mode is required,
but you will still need to take care when calling code
that is stuck on 3 byte addresses.

I'll have to think more about that abstraction. I'll have
to think about what code would look like instead if
that had been done.

BFN. Paul.

On 5/4/21 11:11 AM, muta...@gmail.com wrote:
> On Sunday, May 2, 2021 at 10:08:40 AM UTC+10, anti...@math.uni.wroc.pl wrote:
>
>> to recode to other formats and some old files, but normal
>> use is centerd around few ones, with UTF-8 having clear
>> lead. People did little statistic and it turned out that
>> national characters account for about 5% of the text, so
>> using UTF-8 means expansion of about 5% (because in UTF-8
>> they need 2 bytes) compared to single byte encoding.
>> This increase does not matter for most folks. Given
>> compression, ASCII based computer formats and binary files
>> increase in size is much smaller than 5%.
>
> There is a performance hit using UTF-8 like that.
> No reason to take a performance hit IMO.
>
>>> I don't know that micro-emacs can do that, but
>>> regardless, I'm not interested in using 2 bytes
>>> when only 1 byte is required.
>
>> But you are telling folks in 1965 that they should use 4 bytes,
>> when 3 were enough?
>
> That's a good question.
>
> My main concern is them putting crap into the top byte,
> expecting it to be ignored forever, requiring a specific
> mode to ignore it.
>
> If instead everyone was doing:
>
> SLR R5,R5
> ICM B'0111',ADDR+1
>
> to instead just 3 bytes into a register, with the top byte
> guaranteed to be 0, then no special mode is required,
> but you will still need to take care when calling code
> that is stuck on 3 byte addresses.
>
> I'll have to think more about that abstraction. I'll have
> to think about what code would look like instead if
> that had been done.
>
> BFN. Paul.
>

The BIG issue with creating a 'custom' code page like this is that you
now have created files that WILL need special processing to be used by
anyone not in your close circle that have adopted the encoding.

We have lived through the code page hell before, and it isn't pretty.

Perhaps, if you only used file formats that included an explicit code
page as part of the file header it could work, but otherwise you are
just relegating your data into oblivion.

On Wednesday, May 5, 2021 at 1:34:30 AM UTC+10, Richard Damon wrote:

> The BIG issue with creating a 'custom' code page like this is that you
> now have created files that WILL need special processing to be used by
> anyone not in your close circle that have adopted the encoding.
>
> We have lived through the code page hell before, and it isn't pretty.
>
> Perhaps, if you only used file formats that included an explicit code
> page as part of the file header it could work, but otherwise you are
> just relegating your data into oblivion.

The Vietnamese code page I have in mind (which requires
negotiating giving up 6 control characters) will need to be
embedded in PDPCLIB/PDOS so that setlocale() will cease
reporting via iscntrl() that they are control characters.

PDOS is open source and available in Sourceforge, so I'm
not expecting that to be lost.

Other than the 6 control characters, the rest will be VISCII,
which is in Wikipedia.

I don't think it will take much effort to figure out which
Vietnamese character is which, out of those 6.

Also, although I haven't started to design it yet, I would
assume that PDOS would require some code to convert
keyboard scan codes into the appropriate Vietnamese
character. Actually, I have touched on this a little bit, and
I think the plan is for a shift code to be entered first, and
then the next character that is entered becomes
Vietnamese.

One of my Vietnamese programmers sent me a photo of
his keyboard so that I could confirm it was a normal US
ASCII keyboard.

And note that I am after a text-based solution, not graphics.
Although I don't mind doing a BiosWriteText() call that gets
converted into high resolution graphics. I don't consider that
to be cheating. That's basically what the Amiga needs too,
as there is no equivalent of 0xb8000. Not sure what the
technical term for that is. I also don't mind if BiosWriteText()
gets translated into a write to the serial port where there is
a VANSI (Vietnamese ANSI) terminal attached. Even if no
such terminal has ever been built to date. And I want to
support EBCDIC too, so there will be an EBCDIC-VANSI
terminal too.

BFN. Paul.

On 5/4/21 5:21 PM, muta...@gmail.com wrote:
> On Wednesday, May 5, 2021 at 1:34:30 AM UTC+10, Richard Damon wrote:
>
>> The BIG issue with creating a 'custom' code page like this is that you
>> now have created files that WILL need special processing to be used by
>> anyone not in your close circle that have adopted the encoding.
>>
>> We have lived through the code page hell before, and it isn't pretty.
>>
>> Perhaps, if you only used file formats that included an explicit code
>> page as part of the file header it could work, but otherwise you are
>> just relegating your data into oblivion.
>
> The Vietnamese code page I have in mind (which requires
> negotiating giving up 6 control characters) will need to be
> embedded in PDPCLIB/PDOS so that setlocale() will cease
> reporting via iscntrl() that they are control characters.
>
> PDOS is open source and available in Sourceforge, so I'm
> not expecting that to be lost.
>
> Other than the 6 control characters, the rest will be VISCII,
> which is in Wikipedia.
>
> I don't think it will take much effort to figure out which
> Vietnamese character is which, out of those 6.
>
> Also, although I haven't started to design it yet, I would
> assume that PDOS would require some code to convert
> keyboard scan codes into the appropriate Vietnamese
> character. Actually, I have touched on this a little bit, and
> I think the plan is for a shift code to be entered first, and
> then the next character that is entered becomes
> Vietnamese.
>
> One of my Vietnamese programmers sent me a photo of
> his keyboard so that I could confirm it was a normal US
> ASCII keyboard.
>
> And note that I am after a text-based solution, not graphics.
> Although I don't mind doing a BiosWriteText() call that gets
> converted into high resolution graphics. I don't consider that
> to be cheating. That's basically what the Amiga needs too,
> as there is no equivalent of 0xb8000. Not sure what the
> technical term for that is. I also don't mind if BiosWriteText()
> gets translated into a write to the serial port where there is
> a VANSI (Vietnamese ANSI) terminal attached. Even if no
> such terminal has ever been built to date. And I want to
> support EBCDIC too, so there will be an EBCDIC-VANSI
> terminal too.
>
> BFN. Paul.
>

The issue is that as soon as someone using this encoding tries to share
the data with anyone else, not using that system, or who might be using
a different language and thus needing a different code page, the message
will be gibberish.

If the file/data format does not have a field to declare the encoding,
you are just re-inventing the Tower of Babel that was the dark ages of
Code Pages.

On Wednesday, May 5, 2021 at 7:37:59 AM UTC+10, Richard Damon wrote:

> The issue is that as soon as someone using this encoding tries to share
> the data with anyone else, not using that system, or who might be using
> a different language and thus needing a different code page, the message
> will be gibberish.
>
> If the file/data format does not have a field to declare the encoding,
> you are just re-inventing the Tower of Babel that was the dark ages of
> Code Pages.

I expect VISCII+ to be converted into UTF-8
before sending it anywhere.

I also expect EBCDIC to be converted into ASCII
before sending it anywhere.

My plan is to make my own machine pure EBCDIC,
with a combination of both 80386 and S/390
(I need MVCLE to do moves more than 16 MB,
so can't use S/370 in the long term) instructions
being used.

Both VISCII+ and EBCDIC will run at full speed
whenever running a text-processing program
like "grep".

But both will need conversion before external
communication.

That keeps everyone honest.

BFN. Paul.