Got me thinking of how auto adjust addressing could be added to the Thor
core. There is a bit available in the scaled indexed addressing mode, so I
shoehorned in post-inc, pre-dec modes. This should work with group
register load and store too allowing auto increment for:

loop1:
LOADG g16,[r1+r2*]
STOREG g16,[r3+r2++*]
BLTU r2,1000,.loop1

I must look at adding string instructions back into the instruction set.
Previously there has been copy, set, and compare string instructions. It
is tempting to add a REP instruction modifier to the ISA. It could be a
modified branch instruction because the displacement is not needed.

RLTU r55,1000,”RR”
LOADG g16,[r1+r2*]
STOREG g16,[r3+r2++*]

On 07/05/2023 20:31, MitchAlsup wrote:
> On Sunday, May 7, 2023 at 12:49:24 PM UTC-5, David Brown wrote:
>> On 07/05/2023 18:55, John Levine wrote:
>>> It appears that Anton Ertl <an...@mips.complang.tuwien.ac.at> said:
>>>> PowerPC and ARM A32 is still there. And there's even a new
>>>> architecture with auto-increment: ARM A64.
>>>
>>> I need to take a look.
>>>
>>>>> I think the increase
>>>>> in code density wasn't worth the contortions to ensure that your data
>>>>> structures fit the few cases that the autoincrement modes handled.
>>>>
>>>> Are you thinking of the DSPs that do not have displacement addressing,
>>>> but have auto-increment, leading to a number of papers on how the
>>>> compiler should arrange the variables to make best use of that?
>>>
>>> Autoincrement only increments by the size of a single datum so it
>>> works for strings and vectors, not for arrays of structures or 2-D
>>> arrays. Compare it to the 360's BXLE loop closing instruction which
>>> put the stride in a register so it could be whatever you wanted.
>>> It also had base+index which the Vax did but the PDP-11 only sort
>>> of did if you used absolute addresses instead of a base.
>>>
>>> On the PDP-11 autoincrement allowed a two instruction string copy loop:
>>>
>>> c: movb (r1)+,(r2)+
>>> bnz c ; loop if the byte wasn't zero
>>>
>>> but how useful is that now? I don't know.
>>>
>> Similar instructions would be used for copying memory blocks, and that
>> is very useful!
> <
> Except you are moving blocks 1-byte at a time--which was fine for PDP-11 days
> and for the era of 16-bits "was sufficient" addressing.

Of course you would move the data in bigger sizes - as big as you can,
based on your (i.e., the compiler's) knowledge of alignments, sizes, etc.

Thomas Koenig <tkoenig@netcologne.de> writes:
>void bar (int a, int b, int c);
>
>void foo (int *p)
>{
> int a, b, c;
> a = *p++;
> b = *p++;
> c = *p++;
> bar (a, b, c);
>}
>
>results in
>
> lw a2,8(a0)
> lw a1,4(a0)
> lw a0,0(a0)
> tail bar
>
>on RISC-V, for example (aarch64 plays games with load double,
>so it's a bit harder to read).
>
>But I believe Mitch was referring to the assembler equivalent, where
>p be held in a register.
>
>Autodecrement and increment is done on 386ff. How do they avoid
>the register dependency of the stack register? Special handling?

Yes. My understanding is that they do something similar in the
decoding hardware to what the compiler does for the code above (and of
course the hardware probably does not eliminate the update of the
stack pointer as dead code).

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>

On Monday, May 8, 2023 at 3:36:24 AM UTC+1, robf...@gmail.com wrote:

> loop1:
> LOADG g16,[r1+r2*]
> STOREG g16,[r3+r2++*]
> BLTU r2,1000,.loop1
>
> I must look at adding string instructions back into the instruction set.

yeah can i suggest really don't do that. what happens if you want
to support UCS-2 (strncpyW)? then UCS-4? more than that: the
concepts needed to efficiently support strings, well you have to
add them anyway so why not make them first-order concepts
at the ISA level?

(i am assuming a Horizontal-First Vector ISA here: this does
not apply to Mitch's 66000 which is Vertical-First)

first thing: Fault-First is needed. explained here:
https://alastairreid.github.io/papers/sve-ieee-micro-2017.pdf

this basically is a contractual declaration, "i want you to
load *UP TO* a set maximum number of elements, and
to TELL me how many were actually loaded"

second: extend that same concept onto data: "i want you
to perform some operation *UP TO* a set maximum
number of elements, but if as part of that *ELEMENT*
there is a test that fails, STOP and tell me where you
stopped".

the first concept allows you to safely issue LOADs
knowing full well that no page-fault or other exception
will occur, because the hardware is ORDERED to avoid
them.

the second concept allows you to detect e.g. a null-chr
within a sequential block, but still expressed as a Vector
operation.

the combination of these two allows you to speculatively
load massive parallel blocks of sequential data, that are
then tested in parallel for zero, after which it is plain
sailing to perform the copy.

at all times the Vector Length remains within required
bounds, having been first truncated to take care of potential
exceptions and then having been truncated up to (and
including) the null-chr.

note at lines 52 and 55 that they are both "post-increment".
this is a Vector Load where hardware is permitted to notice
that where the fundamental element operation is a *Scalar*
Load-with-Update, a repeated run of Updates can
be optimised out to only hit the register file with the very
last of those Updates.

of course all of this is completely irrelevant for a Vertical-First
ISA (or an ISA with Vertical-First Vectorisation Mode),
because everything looks to a Vertical-First ISA (such as
Mitch's 66000) like Scalar Looping.

Horizontal-First on the other hand you know that a
large batch of Element-operations are going to hit the
back-end and consequently may micro-code a much more
efficient suite of operations that take up far less resources
than if the individual element operations were naively
thrown into Execute. (a good example is the big-integer
3-in 2-out multiply instruction we are proposing to Power ISA,
which uses one of the Read-regs and one of the Write-regs as
a 64-bit carry. when chained: 1st operation: 3-in 1-out middle-ops
2-in 1-out last-op 2-in 2-out).

https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/decoder/isa/test_caller_svp64_ldst.py;hb=HEAD#l36

44 "mtspr 9, 3", # move r3 to CTR
45 "addi 0,0,0", # initialise r0 to zero
46 # chr-copy loop starts here:
47 # for (i = 0; i < n && src[i] != '\0'; i++)
48 # dest[i] = src[i];
49 # VL (and r1) = MIN(CTR,MAXVL=4)
50 "setvl 1,0,%d,0,1,1" % maxvl,
51 # load VL bytes (update r10 addr)
52 "sv.lbzu/pi *16, 1(10)", # should be /lf here as well
53 "sv.cmpi/ff=eq/vli *0,1,*16,0", # cmp against zero, truncate VL
54 # store VL bytes (update r12 addr)
55 "sv.stbu/pi *16, 1(12)",
56 "sv.bc/all 0, *2, -0x1c", # test CTR, stop if cmpi failed
57 # zeroing loop starts here:
58 # for ( ; i < n; i++)
59 # dest[i] = '\0';
60 # VL (and r1) = MIN(CTR,MAXVL=4)
61 "setvl 1,0,%d,0,1,1" % maxvl,
62 # store VL zeros (update r12 addr)
63 "sv.stbu/pi 0, 1(12)",
64 "sv.bc 16, *0, -0xc", # dec CTR by VL, stop at zero

On Sunday, May 7, 2023 at 5:00:09 PM UTC+1, Anton Ertl wrote:
> John Levine <jo...@taugh.com> writes:
> >Here it is 50 years later and they're all gone.
> PowerPC and ARM A32 is still there.

yyep.

> >also made it harder to parallelize and pipeline stuff since address
> >modes had side effects that had to be scheduled around or potentially
> >unwound in a page fault.

see https://groups.google.com/g/comp.arch/c/_-dp_ZU6TN0/m/G1lzn4M3BgAJ
for reference to Load/Store Fault-First. only useful in Horizontal-First
ISAs (Vertical-First avoids the problem entirely).

> Pipelining was apparently no problem, as evidenced by several early
> RISCs (ARM (A32), HPPA, PowerPC) having auto-increment.

note that Power ISA Architects debated 20+ years ago whether
to add both pre- and post- Update (not quite the same as
auto-increment but you can consider RB or an Immediate to
be "the amount to auto-increment by" which is real handy).

due to space considerations (it's a hell of a lot of instructions
to add) they went with pre-update, on the basis that post-update
may be synthesised by (ha ha) performing a subtract *outside*
of the loop prior to entering the loop.

sigh :) it works...

l.

On Sunday, May 7, 2023 at 3:00:37 AM UTC+1, MitchAlsup wrote:
> Consider a string of *p++
> a = *p++;
> b = *p++;
> c = *p++;
> <
> Here we see the failure of the ++ or -- notation.
> The LD of b is dependent on the ++ of a
> The LD of c is dependent on the ++ of b
> Whereas if the above was written::
> <
> a = p[0];
> b = p[1];
> c = p[2];
> p +=3;

in my mind this is the sort of thing that a compiler pass
should recognise, and perform a miniature AST-rewrite.

at which point *another* pass could spot that if it allocates
a b and c in consecutive registers it may also perform
a 3-long Vector LD. but at that point we are straying into
the bottomless-money-pit of Auto-Vectorisation...

l.

On Saturday, May 6, 2023 at 4:38:20 PM UTC+1, John Levine wrote:

> Here it is 50 years later and they're all gone. I think the increase
> in code density wasn't worth the contortions to ensure that your data
> structures fit the few cases that the autoincrement modes handled.

i thought that too ("few modes") until i realised that you can use
LD-with-Update in a Vector Loop with zero-checking to perform
linked-list-pointer-chasing in a single instruction.

> It
> also made it harder to parallelize and pipeline stuff since address
> modes had side effects that had to be scheduled around or potentially
> unwound in a page fault.

i mentioned in another post about ARM SVE Load-Fault-First
which helps there. i suspect that even Vertical-First ISAs
would have the same issues, once amortisation has been
carried out at the back-end (multiple loops merged into
back-end SIMD).

see ARM SVE paper about pointer-chasing (figure 6)
https://alastairreid.github.io/papers/sve-ieee-micro-2017.pdf

i realised that a repeated-application-of-LD-ST-Update
can chase down the linked-list whilst also dropping
the list structure pointers into consecutive registers.
by also then adding Data-Dependent Fail-First (check
if the data loaded is NULL) you can get the Vector
Operation to stop at or after the NULL, and truncate
such that subsequent Vector operations do not attempt
to go beyond the NULL.

that's a *big* application of auto-update.

also you can use the same instruction to chase double-linked
lists *simultaneously* by making the offset of the updated
register be 2 away from the read-address instead of 1:

sv.ldu/ff=NULL *x+2, *x

what that is doing is, it is reading the address from
sequential registers starting at x, but it is *storing*
the address loaded at registers starting at x+2.

consequently it can be either chasing a single double-linked
list *or* chasing two single-linked-lists, terminating at
the first NULL. at which point to be honest things get
slightly messy as you have to work out which list is
valid, sigh (as you can tell this is a WIP).

l.

"luke.l...@gmail.com" <luke.leighton@gmail.com> writes:
>On Monday, May 8, 2023 at 3:36:24=E2=80=AFAM UTC+1, robf...@gmail.com wrote=
>:
>
>> loop1:=20
>> LOADG g16,[r1+r2*]=20
>> STOREG g16,[r3+r2++*]=20
>> BLTU r2,1000,.loop1=20
>>=20
>> I must look at adding string instructions back into the instruction set.=
>=20
>
>yeah can i suggest really don't do that. what happens if you want
>to support UCS-2 (strncpyW)? then UCS-4? more than that: the
>concepts needed to efficiently support strings, well you have to
>add them anyway so why not make them first-order concepts
>at the ISA level?

UTF8 should be good enough for everything; best to deprecate
USC-2 et al.

It appears that John Levine <johnl@taugh.com> said:
>It appears that Anton Ertl <anton@mips.complang.tuwien.ac.at> said:
>>PowerPC and ARM A32 is still there. And there's even a new
>>architecture with auto-increment: ARM A64.
>
>I need to take a look.

I took a look at ARM and what they did is quite clever. The increment
or decrement amount is a field in the instruction, so up to the field
size (8 bits plus sign as I recall) you can have whatever stride you
want. It also has only one address per instruction so you don't have
the issues you did on the PDP-11 and Vax.

For block memory copies, there's three instructions, roughly prolog,
bocy, epilog, to do it. They don't seem to use autoincrement.

--
Regards,
John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

John Levine <johnl@taugh.com> writes:
>It appears that John Levine <johnl@taugh.com> said:
>>It appears that Anton Ertl <anton@mips.complang.tuwien.ac.at> said:
>>>PowerPC and ARM A32 is still there. And there's even a new
>>>architecture with auto-increment: ARM A64.
>>
>>I need to take a look.
>
>I took a look at ARM and what they did is quite clever. The increment
>or decrement amount is a field in the instruction, so up to the field
>size (8 bits plus sign as I recall) you can have whatever stride you
>want. It also has only one address per instruction so you don't have
>the issues you did on the PDP-11 and Vax.
>
>For block memory copies, there's three instructions, roughly prolog,
>bocy, epilog, to do it. They don't seem to use autoincrement.

Those instructions (FEAT_MOP) are very new - I'm not aware of any shipping
ARMv8 processors that support them yet.

Like the VAX MOVC3/5 instructions, FEAT_MOP updates registers and allows
synchronous (e.g. page fault) and asynchronous (interrupts) during operation;
updating the registers appropriately. There is a special exception that
may be caused if the thread is moved to a different CPU during a copy.

In article <Yia6M.2700564$iU59.525916@fx14.iad>, scott@slp53.sl.home
(Scott Lurndal) wrote:

> Those instructions (FEAT_MOP) are very new - I'm not aware of any
> shipping ARMv8 processors that support them yet.

They appear to be an ARMv9 feature.
<https://developer.arm.com/documentation/ddi0602/2021-12/Base-Instructions
/CPYFPTN--CPYFMTN--CPYFETN--Memory-Copy-Forward-only--reads-and-writes-unp
rivileged-and-non-temporal->

Those are available in Qualcomm Snapdragon 7 Gen 1 onwards and Snapdragon
8 Gen 1 onwards, and MediaTek Dimensionity 9000 chips. So there are
several models of Android 'phone that have them, but not much else. I
have some Snapdragon 8 Gen 1 development kit devices that use them at
work; the chip was announced in November '21 and 'phones appeared in
summer '22.

This is a situation where manufacturers that use ARM core designs can get
ahead of fully custom designs: the Apple M-series chips aren't ARMv9 yet.

The ARM Neoverse V2, N2 and E2 cores support ARMv9, but they were
announced last September and nothing with them has shipped yet.

John

On 5/8/2023 11:22 AM, Scott Lurndal wrote:
> "luke.l...@gmail.com" <luke.leighton@gmail.com> writes:
>> On Monday, May 8, 2023 at 3:36:24=E2=80=AFAM UTC+1, robf...@gmail.com wrote=
>> :
>>
>>> loop1:=20
>>> LOADG g16,[r1+r2*]=20
>>> STOREG g16,[r3+r2++*]=20
>>> BLTU r2,1000,.loop1=20
>>> =20
>>> I must look at adding string instructions back into the instruction set.=
>> =20
>>
>> yeah can i suggest really don't do that. what happens if you want
>> to support UCS-2 (strncpyW)? then UCS-4? more than that: the
>> concepts needed to efficiently support strings, well you have to
>> add them anyway so why not make them first-order concepts
>> at the ISA level?
>
> UTF8 should be good enough for everything; best to deprecate
> USC-2 et al.
>

For many use-cases (transmission and storage), UTF-8 is a sane default,
but there are cases where UTF-8 is not ideal, such as inside console
displays or text editors.

Still makes sense to keep support UTF-16 around for the cases where it
is useful.

Though, for an "advanced" text interface, it usually makes sense to have
additional bits per character cell, say:
(31:28): Background Color
(27:24): Foreground Color
(23:20): Attribute Flags
(19: 0): Codepoint

Or 64-bits if one wants more color-depth and/or things like font size
(or additional attribute modifiers, such as skin-tone modifier for
emojis, etc).

This mostly allowing the text rendering to work in a typical "stream of
character cells" sense.

Though, this sort of approach is generally unable to represent things
like "Zalgo text" (formed by using an excessive number of diacritics and
similar over each letter), and I am not entirely sure how "standard"
text-rendering deals with this sort of thing.

Say, a "straightforward" implementation with 64-bit character cells only
allowing for 1 or 2 diacritics per character.

Then again, it doesn't seem to work in the other text editors I use
anyways, so the inability to represent it is likely a non-issue in most
use cases. (Say, the text editor will strip off most of the diacritics
leaving only the base text).

Well, and similarly approaches like representing each character cell as
a small pixel bitmap (say, 16 colors from a per-cell palette), also
wouldn't be able to represent "Zalgo text" (say, if each cell bitmap
only allows the character to extend 50% out each side of its nominal
bounds).

This is with a 32x32 bitmap per character cell (assuming nominal 16x16
text rendering), but this would need ~ 1K per rendered character cell
(horridly impractical).

Then again, it is possible that only a fixed number of such characters
could exist at any moment, and then be treated as "transient virtual
characters".

Seems almost like many of the text layout renders are operating directly
on a raster image though, without using intermediate character cells.

....

I don't really bother with any of this for TestKern, which (at present)
doesn't even support the full BMP, and what little is supported is
limited to what can be represented directly in 8x8x1 pixel character cells.

It appears that David Brown <david.brown@hesbynett.no> said:
>> Except you are moving blocks 1-byte at a time--which was fine for PDP-11 days
>> and for the era of 16-bits "was sufficient" addressing.
>
>Of course you would move the data in bigger sizes - as big as you can,
>based on your (i.e., the compiler's) knowledge of alignments, sizes, etc.

Which, as discussed in a lot of other messages, you do with groups of
loads and stores where autoincrement isn't very useful.

As I said a few messages ago, I can see how the ARM version with the
stride in the instruction could be useful for stepping through arrays,
but I wouldn't want to get cleverer than that.
--
Regards,
John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

According to BGB <cr88192@gmail.com>:
>> UTF8 should be good enough for everything; best to deprecate
>> USC-2 et al.
>
>For many use-cases (transmission and storage), UTF-8 is a sane default,
>but there are cases where UTF-8 is not ideal, such as inside console
>displays or text editors.
>
>Still makes sense to keep support UTF-16 around for the cases where it
>is useful.

I can see UCS-2 if you're willing to permanently limit yourself to the
subset of Unicode is supports. UTF-16 with surrogate pairs is about as
pessimal an encoding as I can imagine. It's not fixed length, it
doesn't sort consistently like UTF-8 does, and it's not even very
compacy.

In your text editor if you have room I'd say use UTF-32 everywhere, if
not, store stuff in UTF-8 and expand it to UTF-32 when you're working
with it.

--
Regards,
John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

jgd@cix.co.uk (John Dallman) writes:
>In article <Yia6M.2700564$iU59.525916@fx14.iad>, scott@slp53.sl.home
>(Scott Lurndal) wrote:

>
>The ARM Neoverse V2, N2 and E2 cores support ARMv9, but they were
>announced last September and nothing with them has shipped yet.

V9 has a number of "optional" features as similar to ARMv8
which defines multiple "versions" that require certain features
(e.g. v8.1 through v8.8).

N2 is V9.0 which doesn't include FEAT_MOP. See the TRM for N2.

https://developer.arm.com/documentation/102099/0000/The-Neoverse-N2--core

If the feature is supported, the TRM will indicate the appropriate
value in the MOPS field of ID_AA64ISA2_EL1, which the current
N2 Cores do not support.

As for the snapdragon 7 (Cortex 710) processors, they do not support
FEAT_MOPS (which are part of a version after V9.0). MOPS is also
allowed in ARMv8.8 implementations (I am not aware of any extant v8.8 chips).

https://developer.arm.com/documentation/101800/latest

BGB <cr88192@gmail.com> writes:
>On 5/8/2023 11:22 AM, Scott Lurndal wrote:
>> "luke.l...@gmail.com" <luke.leighton@gmail.com> writes:
>>> On Monday, May 8, 2023 at 3:36:24=E2=80=AFAM UTC+1, robf...@gmail.com wrote=
>>> :
>>>
>>>> loop1:=20
>>>> LOADG g16,[r1+r2*]=20
>>>> STOREG g16,[r3+r2++*]=20
>>>> BLTU r2,1000,.loop1=20
>>>> =20
>>>> I must look at adding string instructions back into the instruction set.=
>>> =20
>>>
>>> yeah can i suggest really don't do that. what happens if you want
>>> to support UCS-2 (strncpyW)? then UCS-4? more than that: the
>>> concepts needed to efficiently support strings, well you have to
>>> add them anyway so why not make them first-order concepts
>>> at the ISA level?
>>
>> UTF8 should be good enough for everything; best to deprecate
>> USC-2 et al.
>>
>
>For many use-cases (transmission and storage), UTF-8 is a sane default,
>but there are cases where UTF-8 is not ideal, such as inside console
>displays or text editors.

I disagree with that. Linux-based systems, for example, have no problem using UTF-8
exclusively for editors, x-terms and any other i18n'd application.

>
>Still makes sense to keep support UTF-16 around for the cases where it
>is useful.

It's a painful and non-universal mechanism. Certainly not worth adding
support in the processor for it.

In article <cqb6M.534496$Olad.225271@fx35.iad>, scott@slp53.sl.home
(Scott Lurndal) wrote:

> V9 has a number of "optional" features as similar to ARMv8
> which defines multiple "versions" that require certain features
> (e.g. v8.1 through v8.8).
>
> N2 is V9.0 which doesn't include FEAT_MOP. See the TRM for N2.

Oh, rats. I was under the impression that ARMv9 was uniform, but clearly
this is wrong.

John

On Monday, May 8, 2023 at 7:56:23 PM UTC+1, John Dallman wrote:

> Oh, rats. I was under the impression that ARMv9 was uniform, but clearly
> this is wrong.

not only is it non-uniform there is silicon errata making different hardware
completely binary-incompatible. of course ARM does not care because they
sell to "Silicon Partners" not end-users, and nobody has noticed because
all those disparate systems run Android which is Java bytecode. therefore
as long as workarounds for the errors are compiled into the *java interpreter*
nobody even notices.

step out of that android apps box and start compiling native binaries you are into a
world of pain.

l.

"luke.l...@gmail.com" <luke.leighton@gmail.com> writes:
>On Monday, May 8, 2023 at 7:56:23=E2=80=AFPM UTC+1, John Dallman wrote:
>
>> Oh, rats. I was under the impression that ARMv9 was uniform, but clearly=
>=20
>> this is wrong.=20
>
>not only is it non-uniform there is silicon errata making different hardwar=
>e
>completely binary-incompatible. of course ARM does not care because they
>sell to "Silicon Partners" not end-users, and nobody has noticed because
>all those disparate systems run Android which is Java bytecode. therefore
>as long as workarounds for the errors are compiled into the *java interpret=
>er*
>nobody even notices.

Actually, the android phones run the linux operating system and
assorted native utilities; The application
level android runtime (dalvik) executes bytecode.

Similar to the CPUID instruction on intel, ARM provides registers that
describe which features are implemented on each chip. Software is
expected to not use unimplemented features by testing to see if they're
available. Now, those registers aren't available to user-mode code, but
at least in linux, the OS provides an interface that applications can
use to determine which features are implemented.

Fundamentally, it's no different than the many generations of Intel
and AMD processors each of which implement different sets of SSE/MMX/SGX
et alia features.

>
>step out of that android apps box and start compiling native binaries you a=
>re into a
>world of pain.

Some examples would be useful. All of our chips are ARMv8 (and now ARMv9)
and we've had no "incompatabilities" between generations other than newer
chips have newer features (the ID registers allow the OS to determine
which features are supported).

On 5/8/2023 1:47 PM, Scott Lurndal wrote:
> BGB <cr88192@gmail.com> writes:
>> On 5/8/2023 11:22 AM, Scott Lurndal wrote:
>>> "luke.l...@gmail.com" <luke.leighton@gmail.com> writes:
>>>> On Monday, May 8, 2023 at 3:36:24=E2=80=AFAM UTC+1, robf...@gmail.com wrote=
>>>> :
>>>>
>>>>> loop1:=20
>>>>> LOADG g16,[r1+r2*]=20
>>>>> STOREG g16,[r3+r2++*]=20
>>>>> BLTU r2,1000,.loop1=20
>>>>> =20
>>>>> I must look at adding string instructions back into the instruction set.=
>>>> =20
>>>>
>>>> yeah can i suggest really don't do that. what happens if you want
>>>> to support UCS-2 (strncpyW)? then UCS-4? more than that: the
>>>> concepts needed to efficiently support strings, well you have to
>>>> add them anyway so why not make them first-order concepts
>>>> at the ISA level?
>>>
>>> UTF8 should be good enough for everything; best to deprecate
>>> USC-2 et al.
>>>
>>
>> For many use-cases (transmission and storage), UTF-8 is a sane default,
>> but there are cases where UTF-8 is not ideal, such as inside console
>> displays or text editors.
>
> I disagree with that. Linux-based systems, for example, have no problem using UTF-8
> exclusively for editors, x-terms and any other i18n'd application.
>

As noted, UTF-8 makes sense for "transmission", say, sending text to or
from the console; or in the files loaded or saved from a text editor, etc...

Trying to process, edit, and redraw text *directly* in UTF-8 form
internally would be a massive PITA, and would be computationally
expensive, hence why a "character cell" approach is useful. But, as
noted, 32 or 64 bit cells usually make more sense here as, for things
like "syntax highlighting" etc, it makes sense to mark out the
text-colors in the editor buffers (rather than during the redraw process).

Things like variable-size text rendering add some complexity, but these
are mostly keeping track of the width of each character cell, and the
maximum height for the cells in the row.

Then when one saves out the text, or copies it to the OS clipboard, etc,
it is converted back to UTF-8 (or UTF-16).

As for fonts, there are various strategies:
8x8x1, 8x16x1, or 16x16x1 bitmap
Works, but fairly limited, does not deal with resizable text.
Small pixel bitmap (say, 16x16 or 32x32, 2..8 bpp)
Can deal with things like emojis, but not really resizable.
Signed Distance Fields
Resizable, but less ideal for full-color images (1).
Small vector images for each glyph
Traditional form of True-Type Fonts
Needlessly expensive to draw glyphs this way.

Scaling bitmap fonts with either nearest neighbor or bilinear
insterpolation does not give good looking text (nearest neighbor giving
inconsistent jagged edges, bilinear giving blurry text).

So, "Signed Distance Fields" are a good workaround, but mostly make the
most sense for representing monochrome images.

Effectively, "good" 8-color results require a 6 component image, with 2
components per color bit. For a monochrome image and SDF would need a 2
component image.
A 16-color image would need 8 components to represent effectively with
an SDF.

An SDF can be done using 1 component per channel, but the edge quality
isn't as good (one component forms encoding a combined XY distance from
an edge, and 2 component separately encoding the X and Y distances).

Usual algorithm is to interpolate the texels using bilinear
interopolation or similar, and then threshold the results per color bit
(then one can feed this through a small color palette). Traditionally,
this process being done in a fragment shader or similar.

I guess traditionally, one using a 256x256 texture for every 256 glyphs,
with 16x16 texels per glyph.

Here, the full Unicode BMP would need 256 textures, or roughly 8MB if
each SDF is encoded using DXT1. Though, one trick is to store the glyphs
as a 16x16x1 bitmap font, and then dynamically converting blocks of
glyphs into SDF form (this is how some of my past 3D engines had worked
IIRC).

Though, currently, I haven't really gotten to this stage yet with
TestKern, still just sorta using 8x8x1 pixel bitmap fonts for now.

And, at the moment, I am experimenting with 640x400 and 800x600
256-color modes, and have started working on adding mouse support
(somewhat needed if I add any sort of GUI to this).

In this case, 640x400 8-bpp mode having the advantage that it needs less
memory bandwidth, so the screen is slightly less of a broken jittery
mess (and also, the 800x600 mode currently uses a non-standard 36Hz
refresh).

I guess one possibility could be to give the display hardware an
interface to talk directly with DDR controller (and effectively bypass
the L2 cache). Mostly as the properties the L2 cache adds are "not
particularly optimal" for the access patterns of screen-refresh.

An "L2 bypass path" could potentially be able to sustain high enough
bandwidth to avoid the screen looking like a broken mess when trying to
operate at "slightly higher" resolutions.

It is pros/cons between 256-color and color-cell:
Color cell gives better color fidelity, but more graphical artifacts;
256-color has fewer obvious artifacts, but the color fidelity kinda
sucks (going the RGB555 -> Indexed route; with a "generic" palette);
Drawing the screen image using ordered dither sorta helps, but also
doesn't look particularly good either.

Apparently Half-Life had used this approach (rendering internally using
RGB555 but then reducing the final image back down to 256 color in the
software renderer), but IIRC it looked a lot better than what I am
currently getting.

These images sort of showing the issues I am dealing with:
https://twitter.com/cr88192/status/1654288824669708290

One showing the issue that plagues the 640x400 hi-color mode (and also
800x600 modes), and the other showing the "kinda meh" color rendition
with a fixed 256-color "OS palette" (of the options tested, this being
the palette layout that got the lowest RMSE in my collection of test
images).

Well, along with the 256-color image showing a bug that I have fixed (it
was a bug when doing a partial update of copying the internal
framebuffer to VRAM).

Note that the screen framebuffer is still internally drawn in RGB555,
and then converted to 256-color when being copied into VRAM (well, as
opposed to feeding it through a color-cell encoder).

So, internally this is a 512K screen framebuffer in 640x400 mode, or 1MB
for 800x600. The window also having its own backing buffer (which Doom
draws into, triggering the window stack to be redrawn into the screen
buffer, and then uploaded to VRAM).

>>
>> Still makes sense to keep support UTF-16 around for the cases where it
>> is useful.
>
> It's a painful and non-universal mechanism. Certainly not worth adding
> support in the processor for it.
>

CPU shouldn't really need to know or care.

For all it needs to know about it, it is dealing with 16 or 32 bit WORD
or DWORD values, or packed 16 or 32 bit integer vectors.

The C compiler maybe needs to know/care, and some parts of the C library
which cross paths with this.

On 5/8/2023 1:35 PM, John Levine wrote:
> According to BGB <cr88192@gmail.com>:
>>> UTF8 should be good enough for everything; best to deprecate
>>> USC-2 et al.
>>
>> For many use-cases (transmission and storage), UTF-8 is a sane default,
>> but there are cases where UTF-8 is not ideal, such as inside console
>> displays or text editors.
>>
>> Still makes sense to keep support UTF-16 around for the cases where it
>> is useful.
>
> I can see UCS-2 if you're willing to permanently limit yourself to the
> subset of Unicode is supports. UTF-16 with surrogate pairs is about as
> pessimal an encoding as I can imagine. It's not fixed length, it
> doesn't sort consistently like UTF-8 does, and it's not even very
> compacy.
>

For many contexts, it is sufficient.
People were also apparently happy enough with codepages back in the 1980s.

Granted, UTF-16 is a bit niche...

For text files and API interfaces, I am mostly using UTF-8 for pretty
much everything.

Though, for example, the FAT32 filesystem backend is limited internally
to 1252 (8.3 names) and UTF-16 (LFNs). So, in this case, the filesystem
driver needs to do any character conversion.

In the current driver, the filenames are internally normalized into
UTF-8 form, where it will walk the directory, reading and converting
each name to a normalized form, and then checking if this matches the
filename it is looking for (previously, it would figure if the name fit
an 8.3 or LFN name pattern, and then treat these cases as two different
search strategies).

Also TestKern treats FAT names as case-sensitive, unlike Windows which
is traditionally case-insensitive (though, no idea how Windows would
deal with encountering cases where there are two files with names that
differ only in case, ...).

Though, for text files I took the Unix style "raw blob of bytes"
approach (things like code-page conversion or CR/LF inside "stdio" being
"stuff I don't want to poke with a stick").

Decided to mostly leave out going too much into filesystem related
annoyances.

> In your text editor if you have room I'd say use UTF-32 everywhere, if
> not, store stuff in UTF-8 and expand it to UTF-32 when you're working
> with it.
>

As noted, I have often used 32 or 64 bit character cells in
text-editors. Though, mostly this is for holding things like text color
and attribute metadata.

Usually, each line of text would be given its own array of cells as well
(as opposed to representing newlines and similar directly in the editor).

Line length and similar is a possible issue, but "in practical use" it
probably doesn't matter if the editor has a 256 character line limit,
with a sort of "implicit mandatory word-wrap" past this point (if people
don't like the word wrap, it is their fault for using unreasonably long
lines...); with conventional word-wrap merely enforcing the more
traditional 76..80 character line limit.

Admittedly, I am still not entirely sure how "Zalgo text" is handled,
but testing it in a few (Notepad-style) text editors (not written by
me), it seems to typically be stripped down to around 1 or 2 diacritics
per character.

So, the "crawling mess of stacking diacritics" effect seems to be far
from a universal behavior.

Also not sure if this can be considered as "intentional and well
behaved", or merely "an abuse of the Unicode encoding scheme".

Otherwise, not really sure whether my BJX2 core has the computational
power to effectively manage signed-distance-field text rendering. I
guess I will need to cross that bridge when I get to it.

Most likely option would be going from 16x16x1 to SDF, and then using
the SDF to synthesize bitmap fonts in whatever size is needed (then
mostly rendering using bitmap cells). Well, as opposed to re-running all
the SDF math every time a character is drawn.

I guess this approach could also conceivably be used with True-Type
Fonts as well, and/or render a true-type glyph in a larger size (say,
64x64) and then turning it into a lower-resolution SDF image.

Not looked too much into how True-Type Font handling traditionally works
though.

....

On Monday, May 8, 2023 at 8:28:33 PM UTC+1, Scott Lurndal wrote:

> Actually, the android phones run the linux operating system and
> assorted native utilities;

i know. the binaries - all of them - come compiled with the
Board Support Package supplied by ARM specifically for that
Silicon Partner (Samsung, Allwinner, TI).

and of course, ARM fudges things by providing a *matching*
version of gcc (etc.) that by default contains all the workarounds
for the faulty silicon HDL that they provided that Silicon Partner
with...

> The application
> level android runtime (dalvik) executes bytecode.

dalvik will be compiled with the compiler workarounds. therefore as
far as *users* are concerned (downloaders of android "apps")
"everything just works" because you never, ever see actual assembler
in a *java bytecode* program.

> Fundamentally, it's no different than the many generations of Intel
> and AMD processors each of which implement different sets of SSE/MMX/SGX
> et alia features.

except ARM doesn't give a stuff. why would they? are you paying them
royalties for an ARM License?

> Some examples would be useful. All of our chips are ARMv8 (and now ARMv9)
> and we've had no "incompatabilities" between generations other than newer
> chips have newer features (the ID registers allow the OS to determine
> which features are supported).

you wouldn't - because of the disconnect due to the prevalence of
Android hiding the problem.

it's only when you try to actually take one of these smartphones,
or any product such as those from Hardkernel, and *remove* the
Android OS and *replace* it with e.g. debian then start compiling
*linux* binaries for yourself that you run into the incompatibility
issues.

this is third-hand knowledge over a voice conference call from a
developer who ran into this extremely weird problem, so i cannot
provide further specifics right now.

l.

"luke.l...@gmail.com" <luke.leighton@gmail.com> writes:
>On Monday, May 8, 2023 at 8:28:33=E2=80=AFPM UTC+1, Scott Lurndal wrote:
>
>> Actually, the android phones run the linux operating system and=20
>> assorted native utilities;
>
>i know. the binaries - all of them - come compiled with the
>Board Support Package supplied by ARM specifically for that
>Silicon Partner (Samsung, Allwinner, TI).
>
>and of course, ARM fudges things by providing a *matching*
>version of gcc (etc.) that by default contains all the workarounds
>for the faulty silicon HDL that they provided that Silicon Partner
>with...
>
>> The application=20
>> level android runtime (dalvik) executes bytecode.=20
>
>dalvik will be compiled with the compiler workarounds. therefore as
>far as *users* are concerned (downloaders of android "apps")
>"everything just works" because you never, ever see actual assembler
>in a *java bytecode* program.
>
> > Fundamentally, it's no different than the many generations of Intel=20
>> and AMD processors each of which implement different sets of SSE/MMX/SGX=
>=20
>> et alia features.=20
>
>except ARM doesn't give a stuff. why would they? are you paying them
>royalties for an ARM License?

Yes.

>
>> Some examples would be useful. All of our chips are ARMv8 (and now ARMv9)=
>=20
>> and we've had no "incompatabilities" between generations other than newer=
>=20
>> chips have newer features (the ID registers allow the OS to determine=20
>> which features are supported).
>
>you wouldn't - because of the disconnect due to the prevalence of
>Android hiding the problem.

We don't run android. We run server grade linux workloads.

>
>it's only when you try to actually take one of these smartphones,
>or any product such as those from Hardkernel, and *remove* the
>Android OS and *replace* it with e.g. debian then start compiling
>*linux* binaries for yourself that you run into the incompatibility
>issues.

You still haven't enumerated any of the soi disant "incompatibility issues".

>
>this is third-hand knowledge over a voice conference call from a
>developer who ran into this extremely weird problem, so i cannot
>provide further specifics right now.

So, one swallow doesn't make a summer.

https://en.wikipedia.org/wiki/AWS_Graviton
https://www.datacenterdynamics.com/en/news/googles-in-house-arm-data-center-cpus-pass-key-milestone-ahead-of-reported-2025-cloud-launch/

BGB <cr88192@gmail.com> writes:
>On 5/8/2023 1:35 PM, John Levine wrote:
>> According to BGB <cr88192@gmail.com>:
>>>> UTF8 should be good enough for everything; best to deprecate
>>>> USC-2 et al.
>>>
>>> For many use-cases (transmission and storage), UTF-8 is a sane default,
>>> but there are cases where UTF-8 is not ideal, such as inside console
>>> displays or text editors.
>>>
>>> Still makes sense to keep support UTF-16 around for the cases where it
>>> is useful.
>>
>> I can see UCS-2 if you're willing to permanently limit yourself to the
>> subset of Unicode is supports. UTF-16 with surrogate pairs is about as
>> pessimal an encoding as I can imagine. It's not fixed length, it
>> doesn't sort consistently like UTF-8 does, and it's not even very
>> compacy.
>>
>
>For many contexts, it is sufficient.
>People were also apparently happy enough with codepages back in the 1980s.

That wasn't my experience. And you're limiting yourself to the
microsoft world when you discuss codepages. There was a lot more
to computing back then and IBM, Univac, Burroughs all had international
customers and all had proprietary mechanisms to support them (I had
to design the i18n/l10n support for the Burroughs MCP/VS circa '85)
and there was no common codeset support in EBCDIC those days (leaving
aside de-facto IBM encodings, e.g CP37).

>Though, for example, the FAT32 filesystem backend is limited internally
>to 1252 (8.3 names) and UTF-16 (LFNs). So, in this case, the filesystem
>driver needs to do any character conversion.

It also limits the characters from 1252 that are allowable. IIRC,
the colon character isn't allowed in a FAT32 filename, even when
using LFNs.

The only character not allowed in Unix/Linux UTF-8 filenames is
the forward slash character, and due to the OS API, the nul-byte.

On 5/8/2023 5:36 PM, Scott Lurndal wrote:
> BGB <cr88192@gmail.com> writes:
>> On 5/8/2023 1:35 PM, John Levine wrote:
>>> According to BGB <cr88192@gmail.com>:
>>>>> UTF8 should be good enough for everything; best to deprecate
>>>>> USC-2 et al.
>>>>
>>>> For many use-cases (transmission and storage), UTF-8 is a sane default,
>>>> but there are cases where UTF-8 is not ideal, such as inside console
>>>> displays or text editors.
>>>>
>>>> Still makes sense to keep support UTF-16 around for the cases where it
>>>> is useful.
>>>
>>> I can see UCS-2 if you're willing to permanently limit yourself to the
>>> subset of Unicode is supports. UTF-16 with surrogate pairs is about as
>>> pessimal an encoding as I can imagine. It's not fixed length, it
>>> doesn't sort consistently like UTF-8 does, and it's not even very
>>> compacy.
>>>
>>
>> For many contexts, it is sufficient.
>> People were also apparently happy enough with codepages back in the 1980s.
>
> That wasn't my experience. And you're limiting yourself to the
> microsoft world when you discuss codepages. There was a lot more
> to computing back then and IBM, Univac, Burroughs all had international
> customers and all had proprietary mechanisms to support them (I had
> to design the i18n/l10n support for the Burroughs MCP/VS circa '85)
> and there was no common codeset support in EBCDIC those days (leaving
> aside de-facto IBM encodings, e.g CP37).
>

OK, I had thought the codepage system was "borderline universal", well
except for the computers that did things differently (for example,
Commodore 64, Apple II, ATARI ST, etc, apparently lacking things like
lower-case letters and various other ASCII characters; ...).

Apparently it was also common on the NES to only have a limited subset
of ASCII, since any character cells used for ASCII by extension could
not be used for tile or sprite graphics.

This differs slightly from the text-mode in the BJX2 display hardware,
which currently allows for 1024 unique character cells to be active at
the same time.

Though, at the moment this is mostly used for the 1252 characters, and a
limited range of other characters.

Eg:
0000..007F: ASCII
0080..00FF: 8859-1 / 1252
0100..017F: Graphics (PETSCII+Misc)
0180..01FF: CP437 Glyphs
0200..03FF: Dynamically assigned.

The higher mappings partly exclude some redundant characters, such as
the PETSCII block excludes ASCII characters, and instead adds more
graphics characters, and the CP437 block omits those that are redundant
with 1252.

There are a few blocks beyond this (just non-fixed), including things
like the Greek and Cyrillic alphabets and similar.

The font-space doesn't currently map 1:1 with the BMP; with a remapping
table used to map from Unicode codepoints to the various glyphs
currently existing in the font. I ended up drawing a lot of them myself,
as the existing standardized Unicode fonts (like Unifont) weren't really
designed for 8x8 pixel character cells.

>> Though, for example, the FAT32 filesystem backend is limited internally
>> to 1252 (8.3 names) and UTF-16 (LFNs). So, in this case, the filesystem
>> driver needs to do any character conversion.
>
> It also limits the characters from 1252 that are allowable. IIRC,
> the colon character isn't allowed in a FAT32 filename, even when
> using LFNs.
>
> The only character not allowed in Unix/Linux UTF-8 filenames is
> the forward slash character, and due to the OS API, the nul-byte.
>

Hmm...

I had also assumed Windows-like character limitations here:
: ; / \ ...
Not being allowed.

Others, like space, being in the "sorta allowed but preferably avoided"
category (in my own file naming conventions, I usually consider space as
a "not allowed" character).

Though, in this case, thus far TestKern had used a "vaguely Unix-like
organization":
/ : Virtual, created at boot time.
/boot : SDcard's FAT32 image mounted here
/usr : An "OS image" can be mounted here.
/bin : Symlink to /usr/bin
/etc : Symlink to /usr/etc
...

Thus far, most of the ported software ends up in "/boot", since this is
the part Windows can access.

Not a whole lot in '/bin' yet, as most of the core commands thus far are
built directly into the shell (including a hex viewer and text editor
and similar).

Names like "foo:/" or "foo://" are special, and can be handled with
special handlers (partly intended for URIs in general). Conceptually,
something like Windows drive-letters could be treated as URI's, but this
isn't currently done.

Similarly, given in cases where it usually comes up in a browser on
Windows, filesystem references are usually something like:
file://k:/Foo/bar.txt

This seems like a strike against treating 'k:/' and similar as URI-like
spaces.

And, in a practical sense, 'k:/' would save relatively little over
'/mnt/k' or similar. And, similarly, MS seems to have spent the past 20+
years trying to push the drive-letter system by the wayside anyways (at
least as far as the GUI is concerned).

>