I posted this article on my blog:

https://www.bitsnbites.eu/three-fundamental-flaws-of-simd

....basically stating the obvious (IMO), without putting too much
emphasis on how some alternatives solve some of the issues etc.

There were quite a few replies on reddit [1] and hackernesws [2],
and judging by the comments it seems that many software developers are
actually very fond of packed SIMD. :-/

/Marcus

[1]
https://www.reddit.com/r/programming/comments/p0yn45/three_fundamental_flaws_of_simd

[2] https://news.ycombinator.com/item?id=28114934

Marcus <m.delete@this.bitsnbites.eu> schrieb:
> I posted this article on my blog:
>
> https://www.bitsnbites.eu/three-fundamental-flaws-of-simd
>
> ...basically stating the obvious (IMO), without putting too much
> emphasis on how some alternatives solve some of the issues etc.
>
> There were quite a few replies on reddit [1] and hackernesws [2],
> and judging by the comments it seems that many software developers are
> actually very fond of packed SIMD. :-/

People are familiar with SIMD, and they apparently are not willing
to entertain the thought that there could be something better.

Plus, somebody who has some skill at wringing something useful
out of SIMD, which may be a source of pride and even monetary
compensation, will shy away from thinking too hard about anything
that may be better.

(When I learned about SIMD, coming from a vector computer background,
I certainly was disappointed by its limitations.)

Marcus [2021-08-12 19:08:37] wrote:
> I posted this article on my blog:
> https://www.bitsnbites.eu/three-fundamental-flaws-of-simd
> ...basically stating the obvious (IMO), without putting too much
> emphasis on how some alternatives solve some of the issues etc.
> There were quite a few replies on reddit [1] and hackernesws [2],
> and judging by the comments it seems that many software developers are
> actually very fond of packed SIMD. :-/

Most software developers are basically alien to computer architecture
as an engineering discipline.

The ISA is imposed on them from outside because it is decided by the
machines they can use/buy. From that point of view, they can only
understand your article as "don't use the SIMD primitives provided by
your hardware" ;-)

Stefan

On 8/12/2021 1:36 PM, Stefan Monnier wrote:
> Marcus [2021-08-12 19:08:37] wrote:
>> I posted this article on my blog:
>> https://www.bitsnbites.eu/three-fundamental-flaws-of-simd
>> ...basically stating the obvious (IMO), without putting too much
>> emphasis on how some alternatives solve some of the issues etc.
>> There were quite a few replies on reddit [1] and hackernesws [2],
>> and judging by the comments it seems that many software developers are
>> actually very fond of packed SIMD. :-/
>
> Most software developers are basically alien to computer architecture
> as an engineering discipline.
>
> The ISA is imposed on them from outside because it is decided by the
> machines they can use/buy. From that point of view, they can only
> understand your article as "don't use the SIMD primitives provided by
> your hardware" ;-)
>

And, from the hardware design front, SIMD is comparably cheap and easy
to implement, more so if it can be done in a way which mostly reuses
parts of the CPU core that one already has laying around for Non-SIMD
operations.

Doing vectors by using more advanced forms of pipe-lining is not exactly
ideal in some cases, particularly if it would require increasing the
effective number of pipeline stages or the number of register file ports.

Though, I will admit that SIMD isn't necessarily the best option in an
ISA design elegance sense...

On Thu, 12 Aug 2021 17:27:35 -0000 (UTC), Thomas Koenig
<tkoenig@netcologne.de> wrote:

>(When I learned about SIMD, coming from a vector computer background,
>I certainly was disappointed by its limitations.)

Coming from Connection Machine I was even /more/ disappointed.
POD vectors are nice, but vectors of pointers are a lot nicer.

On Thursday, August 12, 2021 at 12:27:37 PM UTC-5, Thomas Koenig wrote:
> Marcus <m.de...@this.bitsnbites.eu> schrieb:
> > I posted this article on my blog:
> >
> > https://www.bitsnbites.eu/three-fundamental-flaws-of-simd
> >
> > ...basically stating the obvious (IMO), without putting too much
> > emphasis on how some alternatives solve some of the issues etc.
> >
> > There were quite a few replies on reddit [1] and hackernesws [2],
> > and judging by the comments it seems that many software developers are
> > actually very fond of packed SIMD. :-/
<
> People are familiar with SIMD, and they apparently are not willing
> to entertain the thought that there could be something better.
<
My, personal, guess is that people simply write code in HLLs and
let the compiler do its thing--not really caring about the SIMDness
of the code.
<
Compiler writers are aware and conscious of SIMDness caught between
the ever growing SIMD complexity and the screwy things they have to
do to get good code spit out the other end.
<
Operating System people are pretty ignorant, except for the size of the
stuff they have to save/restore around context switches, and when they
get inside various system library functions that the compiler spit out in
SIMD form.
>
> Plus, somebody who has some skill at wringing something useful
> out of SIMD, which may be a source of pride and even monetary
> compensation, will shy away from thinking too hard about anything
> that may be better.
>
> (When I learned about SIMD, coming from a vector computer background,
> I certainly was disappointed by its limitations.)

On Thursday, August 12, 2021 at 8:08:41 PM UTC+3, Marcus wrote:
> I posted this article on my blog:
>
> https://www.bitsnbites.eu/three-fundamental-flaws-of-simd
>
> ...basically stating the obvious (IMO), without putting too much
> emphasis on how some alternatives solve some of the issues etc.
>
> There were quite a few replies on reddit [1] and hackernesws [2],
> and judging by the comments it seems that many software developers are
> actually very fond of packed SIMD. :-/
>
> /Marcus
>
>
> [1]
> https://www.reddit.com/r/programming/comments/p0yn45/three_fundamental_flaws_of_simd
>
> [2] https://news.ycombinator.com/item?id=28114934

You could spend your time better than writing it.
I could spend my time better than reading it.

On Thursday, August 12, 2021 at 12:08:41 PM UTC-5, Marcus wrote:
> I posted this article on my blog:
>
> https://www.bitsnbites.eu/three-fundamental-flaws-of-simd

The fundamental flaw in SIMD is

a) one should be able to encode a vectorizable loop once and have it
run at the maximum performance of each future generation machine.
One should never have to spit out different instructions just because
the width of the SIMD registers was widened.

b) one should be able to encode a SIMD instruction such that it performs
as wide as the implementation has resources (or as narrow) using the
same OpCodes.

On Thursday, August 12, 2021 at 11:08:41 AM UTC-6, Marcus wrote:

> and judging by the comments it seems that many software developers are
> actually very fond of packed SIMD. :-/

Well, if it's all you can get, it's better than nothing.

On my web page, there's a discussion of an imaginary computer architecture
which began as a simple example of how computers work, but which then
grew to include every feature that existed on some historical computer
somewhere.

So it included both packed SIMD, like MMX and its successors, and
unpacked SIMD like the Cray-I, and illustrated how the two differed;
for example, there are two different ways to provide hardware assist
for FFT, and for some reason one was more suitable to packed SIMD
and the other to unpacked SIMD, or at least so I thought when I
prepared the page.

http://www.quadibloc.com/arch/ar0102.htm

et seq. ...

John Savard

On 8/12/2021 8:20 PM, Quadibloc wrote:
> On Thursday, August 12, 2021 at 11:08:41 AM UTC-6, Marcus wrote:
>
>> and judging by the comments it seems that many software developers are
>> actually very fond of packed SIMD. :-/
>
> Well, if it's all you can get, it's better than nothing.
>
> On my web page, there's a discussion of an imaginary computer architecture
> which began as a simple example of how computers work, but which then
> grew to include every feature that existed on some historical computer
> somewhere.
>
> So it included both packed SIMD, like MMX and its successors, and
> unpacked SIMD like the Cray-I, and illustrated how the two differed;
> for example, there are two different ways to provide hardware assist
> for FFT, and for some reason one was more suitable to packed SIMD
> and the other to unpacked SIMD, or at least so I thought when I
> prepared the page.
>
> http://www.quadibloc.com/arch/ar0102.htm
>
> et seq. ...
>
> John Savard
>

Fairly described as "ISA Stockholm Syndrome"?

On 2021-08-12 20:36, Stefan Monnier wrote:
> Marcus [2021-08-12 19:08:37] wrote:
>> I posted this article on my blog:
>> https://www.bitsnbites.eu/three-fundamental-flaws-of-simd
>> ...basically stating the obvious (IMO), without putting too much
>> emphasis on how some alternatives solve some of the issues etc.
>> There were quite a few replies on reddit [1] and hackernesws [2],
>> and judging by the comments it seems that many software developers are
>> actually very fond of packed SIMD. :-/
>
> Most software developers are basically alien to computer architecture
> as an engineering discipline.
>
> The ISA is imposed on them from outside because it is decided by the
> machines they can use/buy. From that point of view, they can only
> understand your article as "don't use the SIMD primitives provided by
> your hardware" ;-)
>
>
> Stefan

Funny thing is that I come from a software developer background, and I
have hated those aspects mentioned in the article for as long as I can
remember.

When I first learned about how Cray-1 worked (some four years ago I
think) I was like "Wow! It must be a joy to program that thing".

Then I created my own vector ISA and wrote some demos, and I was like
"Wow! It's really a joy to program this thing".

I naively though that others would have similar feelings.

/Marcus

On 2021-08-13 00:59, Michael S wrote:
> On Thursday, August 12, 2021 at 8:08:41 PM UTC+3, Marcus wrote:
>> I posted this article on my blog:
>>
>> https://www.bitsnbites.eu/three-fundamental-flaws-of-simd
>>
>> ...basically stating the obvious (IMO), without putting too much
>> emphasis on how some alternatives solve some of the issues etc.
>>
>> There were quite a few replies on reddit [1] and hackernesws [2],
>> and judging by the comments it seems that many software developers are
>> actually very fond of packed SIMD. :-/
>>
>> /Marcus
>>
>>
>> [1]
>> https://www.reddit.com/r/programming/comments/p0yn45/three_fundamental_flaws_of_simd
>>
>> [2] https://news.ycombinator.com/item?id=28114934
>
> You could spend your time better than writing it.
> I could spend my time better than reading it.
>

Feel free to skip reading it :-)

I think that some people need to read it, though, since apparently it's
far from obvious to most people. I wrote it since I found myself
explaining the same things over and over in different forums, and it's
easier to have a text to refer to.

/Marcus

On 2021-08-13 02:05, MitchAlsup wrote:
> On Thursday, August 12, 2021 at 12:08:41 PM UTC-5, Marcus wrote:
>> I posted this article on my blog:
>>
>> https://www.bitsnbites.eu/three-fundamental-flaws-of-simd
>
> The fundamental flaw in SIMD is
>
> a) one should be able to encode a vectorizable loop once and have it
> run at the maximum performance of each future generation machine.
> One should never have to spit out different instructions just because
> the width of the SIMD registers was widened.
>
> b) one should be able to encode a SIMD instruction such that it performs
> as wide as the implementation has resources (or as narrow) using the
> same OpCodes.
>

Yep!

I also added:

c) Packed SIMD is just as sensitive to data hazards as scalar code is,
so you either need proper OoO HW, or you need to unroll all SIMD loops
in SW. And since /some/ implementations are in-order (at least for the
SIMD part), the compiler /always/ unrolls loops -> code size bloat ->
I$ penalty.

d) Tail handling for data sets that are not multiples of the SIMD
width...

/Marcus

MitchAlsup wrote:
> On Thursday, August 12, 2021 at 12:08:41 PM UTC-5, Marcus wrote:
>> I posted this article on my blog:
>>
>> https://www.bitsnbites.eu/three-fundamental-flaws-of-simd
>
> The fundamental flaw in SIMD is
>
> a) one should be able to encode a vectorizable loop once and have it
> run at the maximum performance of each future generation machine.
> One should never have to spit out different instructions just because
> the width of the SIMD registers was widened.
>
> b) one should be able to encode a SIMD instruction such that it performs
> as wide as the implementation has resources (or as narrow) using the
> same OpCodes.

This has been done well in C#, they have vector operations that will
turn into optimal SIMD instructions during the final JIT/AOT stage, this
way they can optimize for the local CPU, and update the compiler for
future platforms.

Terje

--
- <Terje.Mathisen at tmsw.no>
"almost all programming can be viewed as an exercise in caching"

Marcus <m.delete@this.bitsnbites.eu> writes:
>I posted this article on my blog:
>
>https://www.bitsnbites.eu/three-fundamental-flaws-of-simd

Flaw 1 is addressed with SVE. Interestingly, SVE has a flaw that
becomes apparent in the recently announced CPUs: processes cannot
migrate between cores with different register width, so they choose
128 bit wide registers for both small and large cores. How could SVE
have been modified to avoid this flaw? Does RVV have it?

Flaw 2 starts out with a false claim. There is no requirement that
execution units are as wide as the registers, and AMD has delivered a
lot of CPUs where the execution units were half as wide as the
registers: Palomino and K8 implemented SSE (and, for K8, SSE2) with
64-bit wide functional units, Jaguar and later cats, Bulldozer and its
descendents, and Zen 1 implemented AVX-256 with 128-bit-wide
functional units. Could be a solution for ARMs SVE flaw.

In-order CPUs don't play a role for the AMD64 architecture, and
compilers certainly don't unroll AVX code for that (there is no
AVX-capable in-order CPU). Compilers unroll loops in order to reduce
loop overhead, and for additional optimization options.

Flaw 3: In the comments section, you reveal that tail handling is a
problem for auto-vectorization. My take is that auto-vectorization is
a flawed concept, mainly because it is unreliable: you rub the
compiler the wrong way, and it will stop auto-vectorizing the loop
without giving any warning. Manual vectorization is a better
approach, and when done right, tail handling is no problem. See
Sections 2.2-2.4 of

http://www.complang.tuwien.ac.at/papers/ertl18manlang.pdf

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>

On Friday, August 13, 2021 at 12:05:43 PM UTC+3, Anton Ertl wrote:
> Marcus <m.de...@this.bitsnbites.eu> writes:
> >I posted this article on my blog:
> >
> >https://www.bitsnbites.eu/three-fundamental-flaws-of-simd
> Flaw 1 is addressed with SVE. Interestingly, SVE has a flaw that
> becomes apparent in the recently announced CPUs: processes cannot
> migrate between cores with different register width, so they choose
> 128 bit wide registers for both small and large cores. How could SVE
> have been modified to avoid this flaw? Does RVV have it?
>
> Flaw 2 starts out with a false claim. There is no requirement that
> execution units are as wide as the registers, and AMD has delivered a
> lot of CPUs where the execution units were half as wide as the
> registers: Palomino and K8 implemented SSE (and, for K8, SSE2) with
> 64-bit wide functional units, Jaguar and later cats, Bulldozer and its
> descendents, and Zen 1 implemented AVX-256 with 128-bit-wide
> functional units. Could be a solution for ARMs SVE flaw.
>

Intel had 64-bit FP EUs for SSE/SSE2/3 in their main CPU line since introduction of SSE and up until Merom,
i.e. ~8 years later.
On Atom side single precision EUs were 128-bit, but double-precision was half-width for many generations.
May be, still is, I didn't check.

I am a fan of >1 ratio between register width and EUs width myself.
But the reasonable ratios nowadays, i.e.
in presence of caches,
wide superscalar cores,
several cores sharing at least part of on-chip cache,
power consumption as main bottleneck,
no less than 16 software-visible VRs,
other things, I forgot,
are 1, 2 or 4.
CRAY-like ratio today is a horrible idea.
IIRC, Alpha Tarantula proposal argued for 8, but their cache subsystem was very different from how it's done today
and power consumption was low on the list of their priorities.

> In-order CPUs don't play a role for the AMD64 architecture, and
> compilers certainly don't unroll AVX code for that (there is no
> AVX-capable in-order CPU). Compilers unroll loops in order to reduce
> loop overhead, and for additional optimization options.
>

One reason to unroll, even on OoO, is a true (RaW) dependency through accumulator,
which is very typical in inner-product loops.

> Flaw 3: In the comments section, you reveal that tail handling is a
> problem for auto-vectorization. My take is that auto-vectorization is
> a flawed concept, mainly because it is unreliable: you rub the
> compiler the wrong way, and it will stop auto-vectorizing the loop
> without giving any warning. Manual vectorization is a better
> approach, and when done right, tail handling is no problem.

Very true.

> See
> Sections 2.2-2.4 of
>
> http://www.complang.tuwien.ac.at/papers/ertl18manlang.pdf
>
> - anton
> --
> 'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
> Mitch Alsup, <c17fcd89-f024-40e7...@googlegroups.com>

On 2021-08-13 10:54, Terje Mathisen wrote:
> MitchAlsup wrote:
>> On Thursday, August 12, 2021 at 12:08:41 PM UTC-5, Marcus wrote:
>>> I posted this article on my blog:
>>>
>>> https://www.bitsnbites.eu/three-fundamental-flaws-of-simd
>>
>> The fundamental flaw in SIMD is
>>
>> a) one should be able to encode a vectorizable loop once and have it
>> run at the maximum performance of each future generation machine.
>> One should never have to spit out different instructions just because
>> the width of the SIMD registers was widened.
>>
>> b) one should be able to encode a SIMD instruction such that it performs
>> as wide as the implementation has resources (or as narrow) using the
>> same OpCodes.
>
> This has been done well in C#, they have vector operations that will
> turn into optimal SIMD instructions during the final JIT/AOT stage, this
> way they can optimize for the local CPU, and update the compiler for
> future platforms.

....which probably means that they will be able to map well to other
kinds of vector architectures too.

Even if a compiler / language can hide the details of the underlying
vector architecture, packed SIMD still suffers from code bloat though:
Essentially you need to describe things like pipelining and tail
handling in code rather than having the HW take care of it, which hurts
code density /and/ increases front end load.

>
> Terje
>

On 2021-08-13 05:24, Ivan Godard wrote:

[snip]

> Fairly described as "ISA Stockholm Syndrome"?

I'll have to remember that! :-D

/Marcus

On 8/13/2021 3:11 AM, Anton Ertl wrote:
> Marcus <m.delete@this.bitsnbites.eu> writes:
>> I posted this article on my blog:
>>
>> https://www.bitsnbites.eu/three-fundamental-flaws-of-simd
>
> Flaw 1 is addressed with SVE. Interestingly, SVE has a flaw that
> becomes apparent in the recently announced CPUs: processes cannot
> migrate between cores with different register width, so they choose
> 128 bit wide registers for both small and large cores. How could SVE
> have been modified to avoid this flaw? Does RVV have it?
>

IME, moving to vectors larger than 128 bits doesn't tend to gain much
over 128 bits. As the register size gets larger, there is less that can
utilize it effectively.

It is like integer size:
16 bits: can be used heavily, may or may not be sufficient;
32 bits: used heavily, usually sufficient;
64 bits: used occasionally, often useful;
128 bits: used rarely, sometimes useful;
256 bits: novelty size...

For SIMD vectors, there is roughly a factor of 4 relation:
64 bits: can be used heavily, frequently insufficient;
128 bits: used heavily, usually sufficient;
256 bits: used occasionally, sometimes useful;
512 bits: rarely useful.

Once the SIMD vector exceeds the length of the data one would likely
express using vectors, its utility drops off sharply.

The march to endlessly bigger SIMD vectors would make sense if each step
up gave a linear improvement, but makes less sense if it is in
diminishing returns territory.

So, this is part of why my current policy for BJX2 is doing 64 and 128
bit "vectors" and calling it good enough.

Not going to go 256 bits, at least not on a 3-wide core.

Similar tricks to what I used for 128-bit vectors could be used to allow
256-bit vectors on a 6-wide / SMT capable core, since most of the
mechanisms and plumbing would already be in place. Such a vector would
effectively use groups of 4 registers in parallel (and ganging the
memory ports from both SMT threads to allow 256-bit load/store).

So, implicitly, if I had a GSVY, it would implicitly depend on the
existence of WEX-6W...

But, then one might ask: OK, so why not then just issue a 128-bit SIMD
operation on Lane1A and Lane1B?...

And, I could respond: What exactly do you think it is that these 256-bit
operations would be doing?...

For related reasons, one can't have 128-bit operations on a core that
doesn't already support WEX-3W.

> Flaw 2 starts out with a false claim. There is no requirement that
> execution units are as wide as the registers, and AMD has delivered a
> lot of CPUs where the execution units were half as wide as the
> registers: Palomino and K8 implemented SSE (and, for K8, SSE2) with
> 64-bit wide functional units, Jaguar and later cats, Bulldozer and its
> descendents, and Zen 1 implemented AVX-256 with 128-bit-wide
> functional units. Could be a solution for ARMs SVE flaw.
>

FWIW: Despite having 128-bit SIMD, my BJX2 core doesn't actually
(currently) have any units which work natively with 128-bit data.

The register file still mostly operates as 64 bits, and many 128 bit
operations are effectively performed by running two 64-bit operations in
parallel. In many cases, it is simply expanding a single SIMD
instruction over multiple lanes (effectively, a bundle of virtual
operations).

In the ALUX extensions, in a few cases, the ALUs effectively combine
"Voltron style" by feeding bits between each other.

The 128-bit shift operations are effectively two 64-bit funnel shift
operations "in disguise", ...

Well, also having 64-bit units which can occasionally combine for
128-bit operations is a lot less wasteful than having a 128-bit unit
where 99% of the time its capabilities go unused.

And, if one's data doesn't fit nicely into a 128-bit vector... they can
often use two different 64-bit vector ops in parallel.

Packed Integer SIMD isn't done by adding more ALUs, but rather by
splitting up the sub-units within a carry-select adder:
In you select the results where each 16-bit element had a carry-in of
zero, a packed-word ADD magically appears.

For FPU SIMD, there is only one FPU, so SIMD operations internally just
do 4 operations in a row (by sequentially feeding the values into the
FADD or FMUL units and capturing the result out the other end).

I did it the way I did not necessarily because it was "best", but
because I could do it cheaply...

There are ways it could be made better/faster/..., but they would add
resource cost.

Similarly, "going narrower" and presenting vector operations in terms of
scalar operations on individual elements would either:
Slow everything down by essentially turning it back into scalar code;
Require adding a large number of execute lanes and register ports.

Neither of these options is desirable.

> In-order CPUs don't play a role for the AMD64 architecture, and
> compilers certainly don't unroll AVX code for that (there is no
> AVX-capable in-order CPU). Compilers unroll loops in order to reduce
> loop overhead, and for additional optimization options.
>

Yep.

> Flaw 3: In the comments section, you reveal that tail handling is a
> problem for auto-vectorization. My take is that auto-vectorization is
> a flawed concept, mainly because it is unreliable: you rub the
> compiler the wrong way, and it will stop auto-vectorizing the loop
> without giving any warning. Manual vectorization is a better
> approach, and when done right, tail handling is no problem. See
> Sections 2.2-2.4 of
>
> http://www.complang.tuwien.ac.at/papers/ertl18manlang.pdf
>

Generally agreed.
I am actively against autovectorization on BJX2.
To what extent vectors are usable, they are in the form of explicit
language extensions.

I have done things in a way which which I feel doesn't suck nearly as
bad as the "xmmintrin.h" system, and also allows a subset of the GCC
vector extensions.

My approach also tries to be much less of an "ugly wart" on the C
language, so does things in ways that I feel are more consistent with
traditional C semantics (can use native operators, cast conversions, ...).

Though, this does mean some amount of expansion to the typesystem, but
as I feel it, expanding out the C typesystem and numeric tower is much
less evil than something like auto-vectorization, which implies
interpreting C code as if it were something very unlike what was
actually written (and for the compiler to essentially perform magic
tricks with the semantics), or "xmmintrin.h" which is more like thinly
veiled x86 ASM shoehorned into C's existing syntax.

Granted, it does also mean that, unless written to do so, existing
portable C code will not make any use of these sorts of vector extensions.

....

On Friday, August 13, 2021 at 1:53:04 AM UTC-5, Marcus wrote:
> On 2021-08-12 20:36, Stefan Monnier wrote:
> > Marcus [2021-08-12 19:08:37] wrote:
> >> I posted this article on my blog:
> >> https://www.bitsnbites.eu/three-fundamental-flaws-of-simd
> >> ...basically stating the obvious (IMO), without putting too much
> >> emphasis on how some alternatives solve some of the issues etc.
> >> There were quite a few replies on reddit [1] and hackernesws [2],
> >> and judging by the comments it seems that many software developers are
> >> actually very fond of packed SIMD. :-/
> >
> > Most software developers are basically alien to computer architecture
> > as an engineering discipline.
> >
> > The ISA is imposed on them from outside because it is decided by the
> > machines they can use/buy. From that point of view, they can only
> > understand your article as "don't use the SIMD primitives provided by
> > your hardware" ;-)
> >
> >
> > Stefan
> Funny thing is that I come from a software developer background, and I
> have hated those aspects mentioned in the article for as long as I can
> remember.
>
> When I first learned about how Cray-1 worked (some four years ago I
> think) I was like "Wow! It must be a joy to program that thing".
>
> Then I created my own vector ISA and wrote some demos, and I was like
> "Wow! It's really a joy to program this thing".
<
You would enjoy programming in My 66000 ISA.
>
> I naively though that others would have similar feelings.
>
> /Marcus

On Friday, August 13, 2021 at 10:59:42 AM UTC-5, BGB wrote:
> On 8/13/2021 3:11 AM, Anton Ertl wrote:
> > Marcus <m.de...@this.bitsnbites.eu> writes:
> >> I posted this article on my blog:
> >>
> >> https://www.bitsnbites.eu/three-fundamental-flaws-of-simd
> >
> > Flaw 1 is addressed with SVE. Interestingly, SVE has a flaw that
> > becomes apparent in the recently announced CPUs: processes cannot
> > migrate between cores with different register width, so they choose
> > 128 bit wide registers for both small and large cores. How could SVE
> > have been modified to avoid this flaw? Does RVV have it?
> >
> IME, moving to vectors larger than 128 bits doesn't tend to gain much
> over 128 bits. As the register size gets larger, there is less that can
> utilize it effectively.
>
> It is like integer size:
> 16 bits: can be used heavily, may or may not be sufficient;
> 32 bits: used heavily, usually sufficient;
> 64 bits: used occasionally, often useful;
> 128 bits: used rarely, sometimes useful;
> 256 bits: novelty size...
>
> For SIMD vectors, there is roughly a factor of 4 relation:
> 64 bits: can be used heavily, frequently insufficient;
> 128 bits: used heavily, usually sufficient;
> 256 bits: used occasionally, sometimes useful;
> 512 bits: rarely useful.
>
> Once the SIMD vector exceeds the length of the data one would likely
> express using vectors, its utility drops off sharply.
>
>
> The march to endlessly bigger SIMD vectors would make sense if each step
> up gave a linear improvement, but makes less sense if it is in
> diminishing returns territory.
<
I am afraid I have to disagree with you here:: The march to endlessly bigger
SIMD vectors is because the root of the necessity was never addressed
correctly.
<
For example, all of the SIMD instructions in x86-64 are "performed"
by the addition of exactly 2 My 66000 instructions, and may more
the x86-64 are possible to express. ISA explosions shold be proactively
prevented not embraced.
>
>
> So, this is part of why my current policy for BJX2 is doing 64 and 128
> bit "vectors" and calling it good enough.
>
> Not going to go 256 bits, at least not on a 3-wide core.
<
Not going to 256-bits wide EVER.
>
>
> Similar tricks to what I used for 128-bit vectors could be used to allow
> 256-bit vectors on a 6-wide / SMT capable core, since most of the
> mechanisms and plumbing would already be in place. Such a vector would
> effectively use groups of 4 registers in parallel (and ganging the
> memory ports from both SMT threads to allow 256-bit load/store).
>
> So, implicitly, if I had a GSVY, it would implicitly depend on the
> existence of WEX-6W...
>
>
> But, then one might ask: OK, so why not then just issue a 128-bit SIMD
> operation on Lane1A and Lane1B?...
>
> And, I could respond: What exactly do you think it is that these 256-bit
> operations would be doing?...
>
> For related reasons, one can't have 128-bit operations on a core that
> doesn't already support WEX-3W.
> > Flaw 2 starts out with a false claim. There is no requirement that
> > execution units are as wide as the registers, and AMD has delivered a
> > lot of CPUs where the execution units were half as wide as the
> > registers: Palomino and K8 implemented SSE (and, for K8, SSE2) with
> > 64-bit wide functional units, Jaguar and later cats, Bulldozer and its
> > descendents, and Zen 1 implemented AVX-256 with 128-bit-wide
> > functional units. Could be a solution for ARMs SVE flaw.
> >
> FWIW: Despite having 128-bit SIMD, my BJX2 core doesn't actually
> (currently) have any units which work natively with 128-bit data.
>
> The register file still mostly operates as 64 bits, and many 128 bit
> operations are effectively performed by running two 64-bit operations in
> parallel. In many cases, it is simply expanding a single SIMD
> instruction over multiple lanes (effectively, a bundle of virtual
> operations).
>
> In the ALUX extensions, in a few cases, the ALUs effectively combine
> "Voltron style" by feeding bits between each other.
>
> The 128-bit shift operations are effectively two 64-bit funnel shift
> operations "in disguise", ...
>
>
> Well, also having 64-bit units which can occasionally combine for
> 128-bit operations is a lot less wasteful than having a 128-bit unit
> where 99% of the time its capabilities go unused.
>
> And, if one's data doesn't fit nicely into a 128-bit vector... they can
> often use two different 64-bit vector ops in parallel.
<
All of these wide things are performed using CARRY instruction.
>
>
>
> Packed Integer SIMD isn't done by adding more ALUs, but rather by
> splitting up the sub-units within a carry-select adder:
> In you select the results where each 16-bit element had a carry-in of
> zero, a packed-word ADD magically appears.
>
>
> For FPU SIMD, there is only one FPU, so SIMD operations internally just
> do 4 operations in a row (by sequentially feeding the values into the
> FADD or FMUL units and capturing the result out the other end).
>
>
> I did it the way I did not necessarily because it was "best", but
> because I could do it cheaply...
>
> There are ways it could be made better/faster/..., but they would add
> resource cost.
>
>
> Similarly, "going narrower" and presenting vector operations in terms of
> scalar operations on individual elements would either:
> Slow everything down by essentially turning it back into scalar code;
> Require adding a large number of execute lanes and register ports.
>
> Neither of these options is desirable.
> > In-order CPUs don't play a role for the AMD64 architecture, and
> > compilers certainly don't unroll AVX code for that (there is no
> > AVX-capable in-order CPU). Compilers unroll loops in order to reduce
> > loop overhead, and for additional optimization options.
> >
> Yep.
> > Flaw 3: In the comments section, you reveal that tail handling is a
> > problem for auto-vectorization. My take is that auto-vectorization is
> > a flawed concept, mainly because it is unreliable: you rub the
> > compiler the wrong way, and it will stop auto-vectorizing the loop
> > without giving any warning. Manual vectorization is a better
> > approach, and when done right, tail handling is no problem. See
> > Sections 2.2-2.4 of
> >
> > http://www.complang.tuwien.ac.at/papers/ertl18manlang.pdf
> >
> Generally agreed.
> I am actively against autovectorization on BJX2.
> To what extent vectors are usable, they are in the form of explicit
> language extensions.
>
> I have done things in a way which which I feel doesn't suck nearly as
> bad as the "xmmintrin.h" system, and also allows a subset of the GCC
> vector extensions.
>
> My approach also tries to be much less of an "ugly wart" on the C
> language, so does things in ways that I feel are more consistent with
> traditional C semantics (can use native operators, cast conversions, ...).
>
>
> Though, this does mean some amount of expansion to the typesystem, but
> as I feel it, expanding out the C typesystem and numeric tower is much
> less evil than something like auto-vectorization, which implies
> interpreting C code as if it were something very unlike what was
> actually written (and for the compiler to essentially perform magic
> tricks with the semantics), or "xmmintrin.h" which is more like thinly
> veiled x86 ASM shoehorned into C's existing syntax.
>
> Granted, it does also mean that, unless written to do so, existing
> portable C code will not make any use of these sorts of vector extensions.
>
> ...

BGB <cr88192@gmail.com> schrieb:
> On 8/13/2021 3:11 AM, Anton Ertl wrote:
>> Marcus <m.delete@this.bitsnbites.eu> writes:
>>> I posted this article on my blog:
>>>
>>> https://www.bitsnbites.eu/three-fundamental-flaws-of-simd
>>
>> Flaw 1 is addressed with SVE. Interestingly, SVE has a flaw that
>> becomes apparent in the recently announced CPUs: processes cannot
>> migrate between cores with different register width, so they choose
>> 128 bit wide registers for both small and large cores. How could SVE
>> have been modified to avoid this flaw? Does RVV have it?
>>
>
> IME, moving to vectors larger than 128 bits doesn't tend to gain much
> over 128 bits. As the register size gets larger, there is less that can
> utilize it effectively.

I grant you that there is a decreasing return, which currently
levels off at 256 bits - AVX2 is being used a lot in video codecs.

AVX512 is a fiasco, but mainly because Intel overspent its heat
budget and has to clock down the CPU a _lot_ to use it, destroying
most if any advantage in using it.

> Generally agreed.
> I am actively against autovectorization on BJX2.
> To what extent vectors are usable, they are in the form of explicit
> language extensions.

Count me out for using your architecture, then.

Explicit language extensions

- lock in the user to a specific architecture and compiler
- expose architecture details which should not be visible
- make code hard to write, read and thus maintain

[...]

> My approach also tries to be much less of an "ugly wart" on the C
> language, so does things in ways that I feel are more consistent with
> traditional C semantics (can use native operators, cast conversions, ...).

There are other programming languages than C. What would you
propose for Fortran, for example?

MitchAlsup <MitchAlsup@aol.com> schrieb:

> You would enjoy programming in My 66000 ISA.

So would I.

Any chance of this happening in the forseeable future?

On 8/13/2021 11:44 AM, Thomas Koenig wrote:
> BGB <cr88192@gmail.com> schrieb:
>> On 8/13/2021 3:11 AM, Anton Ertl wrote:
>>> Marcus <m.delete@this.bitsnbites.eu> writes:
>>>> I posted this article on my blog:
>>>>
>>>> https://www.bitsnbites.eu/three-fundamental-flaws-of-simd
>>>
>>> Flaw 1 is addressed with SVE. Interestingly, SVE has a flaw that
>>> becomes apparent in the recently announced CPUs: processes cannot
>>> migrate between cores with different register width, so they choose
>>> 128 bit wide registers for both small and large cores. How could SVE
>>> have been modified to avoid this flaw? Does RVV have it?
>>>
>>
>> IME, moving to vectors larger than 128 bits doesn't tend to gain much
>> over 128 bits. As the register size gets larger, there is less that can
>> utilize it effectively.
>
> I grant you that there is a decreasing return, which currently
> levels off at 256 bits - AVX2 is being used a lot in video codecs.
>
> AVX512 is a fiasco, but mainly because Intel overspent its heat
> budget and has to clock down the CPU a _lot_ to use it, destroying
> most if any advantage in using it.
>
>
>> Generally agreed.
>> I am actively against autovectorization on BJX2.
>> To what extent vectors are usable, they are in the form of explicit
>> language extensions.
>
> Count me out for using your architecture, then.
>
> Explicit language extensions
>
> - lock in the user to a specific architecture and compiler
> - expose architecture details which should not be visible
> - make code hard to write, read and thus maintain
>

The traditional solution to this is to use ifdefs.

One already has to do this if they want the same code to use SSE and
NEON in a semi-effective manner.

Or use the common subset that exists with GCC's
"__attribute__((vector_size(N)))" system.

Otherwise, it is like saying that no one can use inline ASM, or that one
will refuse to use a compiler which supports using inline ASM.

One can just write traditional scalar code, and have it perform as such.
Its performance may suck in comparison, but it isn't precluded.

IME, auto vectorization only sometimes helps on traditional targets, and
frequently (if the optimizer is overzealous) can turn into a wrecking
ball on the performance front (performing worse, sometimes
significantly, than its non vectorized counterpart).

If it were up to me though, there would be explicit attributes to tell
the compiler what it should or should not vectorize.

Or, maybe, if some standardized way were defined to specify vector types
in C (preferably more concise than GCC's notation).

Say, for example, if the compiler allowed:
float[[vector(4)]] vec; //define a 4-element floating point vector.
Or:
[[vector(4)]] float vec; //basically equivalent.

> [...]
>
>> My approach also tries to be much less of an "ugly wart" on the C
>> language, so does things in ways that I feel are more consistent with
>> traditional C semantics (can use native operators, cast conversions, ...).
>
> There are other programming languages than C. What would you
> propose for Fortran, for example?
>

Dunno, not enough overlap between use-cases.

BJX2 is not intended for supercomputers or scientific computing, rather
I was more intending it for robot and machine control tasks. Originally
this was partway to address some annoyances I had with using ARM based
controllers for these tasks.

But, this part is an uphill battle, though despite its much slower
clock-speeds, it fares relatively well at some of the tasks I intended
it for, though sadly some tasks are seriously hindered by the available
memory bandwidth.

On 8/13/2021 11:29 AM, MitchAlsup wrote:
> On Friday, August 13, 2021 at 10:59:42 AM UTC-5, BGB wrote:
>> On 8/13/2021 3:11 AM, Anton Ertl wrote:
>>> Marcus <m.de...@this.bitsnbites.eu> writes:
>>>> I posted this article on my blog:
>>>>
>>>> https://www.bitsnbites.eu/three-fundamental-flaws-of-simd
>>>
>>> Flaw 1 is addressed with SVE. Interestingly, SVE has a flaw that
>>> becomes apparent in the recently announced CPUs: processes cannot
>>> migrate between cores with different register width, so they choose
>>> 128 bit wide registers for both small and large cores. How could SVE
>>> have been modified to avoid this flaw? Does RVV have it?
>>>
>> IME, moving to vectors larger than 128 bits doesn't tend to gain much
>> over 128 bits. As the register size gets larger, there is less that can
>> utilize it effectively.
>>
>> It is like integer size:
>> 16 bits: can be used heavily, may or may not be sufficient;
>> 32 bits: used heavily, usually sufficient;
>> 64 bits: used occasionally, often useful;
>> 128 bits: used rarely, sometimes useful;
>> 256 bits: novelty size...
>>
>> For SIMD vectors, there is roughly a factor of 4 relation:
>> 64 bits: can be used heavily, frequently insufficient;
>> 128 bits: used heavily, usually sufficient;
>> 256 bits: used occasionally, sometimes useful;
>> 512 bits: rarely useful.
>>
>> Once the SIMD vector exceeds the length of the data one would likely
>> express using vectors, its utility drops off sharply.
>>
>>
>> The march to endlessly bigger SIMD vectors would make sense if each step
>> up gave a linear improvement, but makes less sense if it is in
>> diminishing returns territory.
> <
> I am afraid I have to disagree with you here:: The march to endlessly bigger
> SIMD vectors is because the root of the necessity was never addressed
> correctly.
> <
> For example, all of the SIMD instructions in x86-64 are "performed"
> by the addition of exactly 2 My 66000 instructions, and may more
> the x86-64 are possible to express. ISA explosions shold be proactively
> prevented not embraced.

The ISA explosion can be contained.

x86 and ARM just sorta did a particularly bad job at it, as they sort of
awkwardly hacked it onto an ISA design where it didn't really fit, so
pretty much the entire ISA needs to be duplicated several times over.

Well, and also NEON integrating format conversions within its operations
being kinda absurd.

>>
>>
>> So, this is part of why my current policy for BJX2 is doing 64 and 128
>> bit "vectors" and calling it good enough.
>>
>> Not going to go 256 bits, at least not on a 3-wide core.
> <
> Not going to 256-bits wide EVER.

That is also an option.

On a core which could support it, something like:
PADDYF R24, R36, R44
Would be functionally equivalent to:
PADDXF R26, R38, R46 | PADDXF R24, R36, R44

And, one could just define the latter form as the one to use, or add
256-bit vectors as a kind of syntactic sugar in the assembler.

>>
>>
>> Similar tricks to what I used for 128-bit vectors could be used to allow
>> 256-bit vectors on a 6-wide / SMT capable core, since most of the
>> mechanisms and plumbing would already be in place. Such a vector would
>> effectively use groups of 4 registers in parallel (and ganging the
>> memory ports from both SMT threads to allow 256-bit load/store).
>>
>> So, implicitly, if I had a GSVY, it would implicitly depend on the
>> existence of WEX-6W...
>>
>>
>> But, then one might ask: OK, so why not then just issue a 128-bit SIMD
>> operation on Lane1A and Lane1B?...
>>
>> And, I could respond: What exactly do you think it is that these 256-bit
>> operations would be doing?...
>>
>> For related reasons, one can't have 128-bit operations on a core that
>> doesn't already support WEX-3W.
>>> Flaw 2 starts out with a false claim. There is no requirement that
>>> execution units are as wide as the registers, and AMD has delivered a
>>> lot of CPUs where the execution units were half as wide as the
>>> registers: Palomino and K8 implemented SSE (and, for K8, SSE2) with
>>> 64-bit wide functional units, Jaguar and later cats, Bulldozer and its
>>> descendents, and Zen 1 implemented AVX-256 with 128-bit-wide
>>> functional units. Could be a solution for ARMs SVE flaw.
>>>
>> FWIW: Despite having 128-bit SIMD, my BJX2 core doesn't actually
>> (currently) have any units which work natively with 128-bit data.
>>
>> The register file still mostly operates as 64 bits, and many 128 bit
>> operations are effectively performed by running two 64-bit operations in
>> parallel. In many cases, it is simply expanding a single SIMD
>> instruction over multiple lanes (effectively, a bundle of virtual
>> operations).
>>
>> In the ALUX extensions, in a few cases, the ALUs effectively combine
>> "Voltron style" by feeding bits between each other.
>>
>> The 128-bit shift operations are effectively two 64-bit funnel shift
>> operations "in disguise", ...
>>
>>
>> Well, also having 64-bit units which can occasionally combine for
>> 128-bit operations is a lot less wasteful than having a 128-bit unit
>> where 99% of the time its capabilities go unused.
>>
>> And, if one's data doesn't fit nicely into a 128-bit vector... they can
>> often use two different 64-bit vector ops in parallel.
> <
> All of these wide things are performed using CARRY instruction.

Something like CARRY doesn't really map onto how things are done in
BJX2, where I prefer to avoid contextual encodings.

Granted, prefixes like Jumbo or Op64 could be considered as turning the
following instruction into a contextual encoding, but this can be
sidestepped in the Jumbo or Op64 prefix is seen as composing a new
widened encoding.

Similarly, given they are generally required to be adjacent sort of
avoids "spooky action at a distance" behaviors.

Sadly, this sorta breaks down with the WEX-6W "interleave" semantics,
but trying to define a 6-wide core in terms of two overlapping 3-wide
pipelines introduces a bit of hair. Off-hand, there wasn't a good
alternative that was less awful, aside from limiting the use of Jumbo
prefixes to match their use in the 3-wide profile.

This is how I defined the 5-wide profile to behave, which loses some
potential performance, with the trade-off of "less hair" as it allows
ignoring the use of pipeline interleaving.

Whether or not a 6W core is viable is still "yet to be seen".

Another option could be to support SMT without supporting 6W operation.
In this case, the two pipelines would be independent, and the second
pipeline simply go unused if not running in SMT. However, this could
sidestep some of the issues with trying to support a unified 12R/6W
register file (by essentially just running two 6R/3W register files in
parallel with a little bit of additional plumbing trickery).

>>
>>
>>
>> Packed Integer SIMD isn't done by adding more ALUs, but rather by
>> splitting up the sub-units within a carry-select adder:
>> In you select the results where each 16-bit element had a carry-in of
>> zero, a packed-word ADD magically appears.
>>
>>
>> For FPU SIMD, there is only one FPU, so SIMD operations internally just
>> do 4 operations in a row (by sequentially feeding the values into the
>> FADD or FMUL units and capturing the result out the other end).
>>
>>
>> I did it the way I did not necessarily because it was "best", but
>> because I could do it cheaply...
>>
>> There are ways it could be made better/faster/..., but they would add
>> resource cost.
>>
>>
>> Similarly, "going narrower" and presenting vector operations in terms of
>> scalar operations on individual elements would either:
>> Slow everything down by essentially turning it back into scalar code;
>> Require adding a large number of execute lanes and register ports.
>>
>> Neither of these options is desirable.
>>> In-order CPUs don't play a role for the AMD64 architecture, and
>>> compilers certainly don't unroll AVX code for that (there is no
>>> AVX-capable in-order CPU). Compilers unroll loops in order to reduce
>>> loop overhead, and for additional optimization options.
>>>
>> Yep.
>>> Flaw 3: In the comments section, you reveal that tail handling is a
>>> problem for auto-vectorization. My take is that auto-vectorization is
>>> a flawed concept, mainly because it is unreliable: you rub the
>>> compiler the wrong way, and it will stop auto-vectorizing the loop
>>> without giving any warning. Manual vectorization is a better
>>> approach, and when done right, tail handling is no problem. See
>>> Sections 2.2-2.4 of
>>>
>>> http://www.complang.tuwien.ac.at/papers/ertl18manlang.pdf
>>>
>> Generally agreed.
>> I am actively against autovectorization on BJX2.
>> To what extent vectors are usable, they are in the form of explicit
>> language extensions.
>>
>> I have done things in a way which which I feel doesn't suck nearly as
>> bad as the "xmmintrin.h" system, and also allows a subset of the GCC
>> vector extensions.
>>
>> My approach also tries to be much less of an "ugly wart" on the C
>> language, so does things in ways that I feel are more consistent with
>> traditional C semantics (can use native operators, cast conversions, ...).
>>
>>
>> Though, this does mean some amount of expansion to the typesystem, but
>> as I feel it, expanding out the C typesystem and numeric tower is much
>> less evil than something like auto-vectorization, which implies
>> interpreting C code as if it were something very unlike what was
>> actually written (and for the compiler to essentially perform magic
>> tricks with the semantics), or "xmmintrin.h" which is more like thinly
>> veiled x86 ASM shoehorned into C's existing syntax.
>>
>> Granted, it does also mean that, unless written to do so, existing
>> portable C code will not make any use of these sorts of vector extensions.
>>
>> ...