John Savard <quadibloc@servername.invalid> writes:
>And if memory bandwidth issues make Cray-style vector machines
>impractical, then wouldn't it be even worse for GPUs?

The claim by Mitch Alsup is that latency makes the Crays impractical,
because of chaining issues. Do GPUs have chaining? My understanding
is that GPUs deal with latency in the barrel processor way: use
another data-parallel thread while waiting for memory. Tera also
pursued this idea, but the GPUs succeeded with it.

>If
>most problems anyone would want to use a vector CPU for today do
>involve a large amount of memory, used in a random fashion, so as to
>fit poorly in cache

When the working set is larger than the cache, it does not fit even
when accesses regularly. Prefetchers can reduce the latency, but they
will not increase the bandwidth.

So if you have a problem that walks through a lot of memory and
performs only a few operations per data item, that's where CPUs will
wait for memory a lot, due to limited bandwidth (and you won't benefit
from SIMD/vector instructions on these kinds of problems). For that
kind of stuff you better use GPUs, which have memory systems with more
bandwidth.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>

Lawrence D'Oliveiro <ldo@nz.invalid> writes:
>On Tue, 23 Apr 2024 19:25:22 -0600, John Savard wrote:
>Looking at an old Cray-1 manual, it mentions, among other things, sixty
>four 64-bit intermediate scalar “T” registers, and eight 64-element vector
>“V” registers of 64 bits per element. That’s a lot of registers.
>
>RISC-V has nothing like this, as far as I can tell. Right at the top of
>the spec I linked earlier, it says:
>
> The vector extension adds 32 architectural vector registers,
> v0-v31 to the base scalar RISC-V ISA.
>
> Each vector register has a fixed VLEN bits of state.
>
>So, no “big vector registers” that I can see? It says that VLEN must be a
>power of two no bigger than 2**16, which does sound like a lot, but then
>the example they give only has VLEN = 128.

It's an example. If you think you can make and profitably sell a CPU
with VLEN=4096 (the number of bits in one of Cray-1's vector
registers), that would be compliant with the spec, and would run
programs written for RISC-V with the vector extension. Or you can
make one with VLEN=65536 and claim that you have the longest one:-).

This leaves you free to decide VLEN based on the costs and benefits in
the context of the other design decisions you have made and on the
programs you expect to run.

Note that the Fujitsu A64FX (which implements the similar ARM Scalable
Vector Extension and was designed for supercomputing) chooses a
512-bit vector implementation.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>

On Wed, 24 Apr 2024 06:16:58 GMT, Anton Ertl wrote:

> For that kind of stuff you better use GPUs, which have memory systems
> with more bandwidth.

But with more limited memory, which is typically not upgradeable.

On Wed, 24 Apr 2024 02:00:10 +0000, mitchalsup@aol.com (MitchAlsup1)
wrote:

>Everyone has to have hope on something.

But false hopes are a waste of time.

The reason for my interest in long vectors is primarily because I
imagine that, if the Cray I was an improvement on the IBM System/360
Model 195, then, apparently, today a chip like the Cray I would be
the next logical step after the Pentium II (OoO plus cache, just like
a Model 195).

And that's a very naïve way of looking at the issue, so of course it
can be wrong.

I can, however, believe that latency, not bandwith as such, is the
killer. That's true for regular CPU compute, and so of course it would
be a limiting factor for vector machines.

What do vector machines do?

Well, apparently they do things like multiply 2048 by 2048 matrices.
Which is why they need stride. And since modern DRAMs like to give you
16 consecutive values at a time... oh, well, you can multiply 16 rows
of the matrix at once. Each matrix would take 32 megabytes of storage,
so that does fit in cache, at least.

But they've managed to get GPUs to multiply matrices - and they're
quite good at it, which is why we're having all this amazing progress
in AI recently. So it's quite possible that long vector machines have
too narrow a niche, between plain CPUs (more flexible) and GPUs (less
flexible).

John Savard

On Wed, 24 Apr 2024 00:57:07 -0600, John Savard wrote:

> But they've managed to get GPUs to multiply matrices - and they're quite
> good at it, which is why we're having all this amazing progress in AI
> recently.

Worth noting that this AI stuff requires very low-precision floats: 16-
bit, even 8-bit. And they sacrifice mantissa bits in favour of exponents--
down to something like maybe only a couple of mantissa bits in the 8-bit
format.

On 4/24/2024 2:08 AM, Lawrence D'Oliveiro wrote:
> On Wed, 24 Apr 2024 00:57:07 -0600, John Savard wrote:
>
>> But they've managed to get GPUs to multiply matrices - and they're quite
>> good at it, which is why we're having all this amazing progress in AI
>> recently.
>
> Worth noting that this AI stuff requires very low-precision floats: 16-
> bit, even 8-bit. And they sacrifice mantissa bits in favour of exponents--
> down to something like maybe only a couple of mantissa bits in the 8-bit
> format.

Yeah... Something like S.E4.F3 or similar is almost useless, but not
entirely useless.

In my case I have a few things to allow for 8-bit formats as storage,
but ended up going with Binary16 as the smallest floating-point format
used for actual computation.

In some of my own small experiments with NN's (mostly perceptron style),
FP8 was an OK format for storing weights and biases, but generally had
an adverse effect if used for the accumulators (which need at least
slightly higher precision to "actually accumulate stuff" effectively).

Something like FP12 (S.E5.F6) would probably be OK though, and in most
other areas one can "throw precision to the wind" with this stuff.

But, this was my own limited experience.

Probably, if doing something like, say, multiplying two FP8 values and
then accumulating as a 16-bit fixed-point value or similar, converting
the result back to FP8 as part of the activation function, this could
probably also work for specialized hardware.

Or, maybe even go smaller for the impulses, say, 3-bit:
000: 0.0 001: 0.5 010: 1.0 011: 1.5+
100: -0.0 101: -0.5 110: -1.0 111: -1.5+

Where, say, weights could be FP8 but impulses would be mostly 3-bit
(since, following the activation function, the actual values are no
longer particularly relevant).

This could be used to further reduce the storage cost, and the cost of
multipliers (which are basically a 6-bit lookup and a bit-shift in this
case). Though, FP8*FP8 isn't that much more, and would possibly be
cheaper if the output is Binary16 (though, Binary16 would be more
expensive to accumulate than, say, a 16-bit fixed-point value).

....

John Savard <quadibloc@servername.invalid> writes:
>On Wed, 24 Apr 2024 02:00:10 +0000, mitchalsup@aol.com (MitchAlsup1)
>wrote:
>
>>Everyone has to have hope on something.
>
>But false hopes are a waste of time.
>
>The reason for my interest in long vectors is primarily because I
>imagine that, if the Cray I was an improvement on the IBM System/360
>Model 195, then, apparently, today a chip like the Cray I would be
>the next logical step after the Pentium II (OoO plus cache, just like
>a Model 195).

But the Cray-1 is not an improvement on the Model 195. It has no
cache. Neither the Cray-1 nor the Model 195 have OoO as the term is
commonly understood today: OoO execution, in-order completion,
allowing register renaming, speculative execution, and precise
exceptions. One may consider the Model 91/195 a predecessor of
today's OoO, because it supports register renaming, and you "just"
need to add a reorder buffer to get in-order completion and
speculative execution.

>Well, apparently they do things like multiply 2048 by 2048 matrices.
>Which is why they need stride.

You can multiply dense matrices of any size efficiently with stride 1.
And caches help a lot for matrix multiply; in HPC circles, (dense)
matrix multiply is known as cache-friendly problem.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>

Lawrence D'Oliveiro <ldo@nz.invalid> writes:
>On Wed, 24 Apr 2024 06:16:58 GMT, Anton Ertl wrote:
>
>> For that kind of stuff you better use GPUs, which have memory systems
>> with more bandwidth.
>
>But with more limited memory, which is typically not upgradeable.

And yet, supercomputers these days often have lots of GPUs. The
software crisis still is not yet there in supercomputing, so they
manage to do with explicit moving of data between the high-bandwidth
GPU memory and the lower-bandwidth bigger memory, just like in the
days of the Cray-1 (or was it the CDC-6600?), which has a fast memory
and a bigger slow memory.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>

On 4/24/24 1:57 AM, John Savard wrote:
> What do vector machines do?

They keep a pipeline full.

So you can do something in 64+7 clock cycles instead of 64*7.

If the pipeline gets shorter the benefit decreases of course. And if you
have some other way to keep that pipeline full, you don't need vectors.

--
http://davesrocketworks.com
David Schultz

Anton Ertl wrote:

> John Savard <quadibloc@servername.invalid> writes:
>>And if memory bandwidth issues make Cray-style vector machines
>>impractical, then wouldn't it be even worse for GPUs?

> The claim by Mitch Alsup is that latency makes the Crays impractical,
> because of chaining issues. Do GPUs have chaining? My understanding
> is that GPUs deal with latency in the barrel processor way: use
> another data-parallel thread while waiting for memory. Tera also
> pursued this idea, but the GPUs succeeded with it.

> - anton

Consider:: an 8 deep CRAY-like vector calculation with 8 cycle latency
memory and 6 cycle latency FMAC::

|LD|LD|LD|LD|LD|LD|LD|LD|
|FM|FM|FM|FM|FM|FM|FM|FM|
|ST|ST|ST|ST|ST|ST|ST|ST|

Not much parallelism. Now consider the same machine above with longer
vectors::

|LD|LD|LD|LD|LD|LD|LD|LD|LD|LD|LD|LD|LD|LD|LD|LD|
|FM|FM|FM|FM|FM|FM|FM|FM|FM|FM|FM|FM|FM|FM|FM|FM|
|ST|ST|ST|ST|ST|ST|ST|ST|ST|ST|ST|ST|ST|ST|ST|ST|

Now we have considerable parallelism with no change in latencies.

Later consider the top execution profile augmented with a bit of OoO
and a second memory port::

|LD|LD|LD|LD|LD|LD|LD|LD|
|FM|FM|FM|FM|FM|FM|FM|FM|
|SA|SA|SA|SA|SA|SA|SA|SA| |Sd|Sd|Sd|Sd|Sd|Sd|Sd|Sd|

Finally consider a GBOoO implementation::

|LD|LD|LD|LD|LD|LD|LD|LD|LD|LD|LD|LD|LD|LD|LD|LD|...
|Fq|Fq|Fq|Fq|Fq|Fq|Fq|Fq|Fq|Fq|Fq|Fq|Fq|Fq|Fq|Fq|...
|FM|FM|FM|FM|FM|FM|FM|FM|...
|SA|SA|SA|SA|SA|SA|SA|SA|SA|SA|SA|SA|SA|SA|SA|SA|...
|Sd|Sd|Sd|Sd|Sd|Sd|Sd|Sd|...

Here it takes an execution window 18 deep to reach pipeline saturation,
but once you do, the core runs at 3 instructions and arguably 4 units
of work per cycle {without including loop overheads}. In order to
achieve such performance one needs to issue the whole loop in 1 cycle.

You have to have the requisite bandwidths {AGEN, bank access, address
routing bandwidth, result return bandwidth, FMAC bandwidth} but you
also have to have the requisite latencies (and excecution window width)
or it falls apart that enable the vector chaining to work.

MitchAlsup1 wrote:
> Lawrence D'Oliveiro wrote:
>
>> On Tue, 23 Apr 2024 18:36:49 -0500, BGB wrote:
>
>>> DIV:
>>> Didn't bother with this.
>>> Typically faked using multiply-by-reciprocal and taking the high result.
>
>> Another Cray-ism! ;)
>
> Not IEEE 754 legal.

Well, it _is_ legal if you carry enough bits in your reciprocal...but at
that point you would instead use a better algorithm to get the correct
result both faster and using less power.

Terje

--
- <Terje.Mathisen at tmsw.no>
"almost all programming can be viewed as an exercise in caching"

On Wed, 24 Apr 2024 09:18:56 GMT, Anton Ertl wrote:

> But the Cray-1 is not an improvement on the Model 195.

The Cray-1 was widely regarded as the fastest computer in the world, when
it came out. Cruising speed of something over 80 megaflops, hitting bursts
of about 120.

IBM did try to compete in the “supercomputer” field for a while longer,
but I think by about ten years later, it had given up.

On Wed, 24 Apr 2024 09:28:06 GMT, Anton Ertl wrote:

> Lawrence D'Oliveiro <ldo@nz.invalid> writes:
>
>>On Wed, 24 Apr 2024 06:16:58 GMT, Anton Ertl wrote:
>>
>>> For that kind of stuff you better use GPUs, which have memory systems
>>> with more bandwidth.
>>
>>But with more limited memory, which is typically not upgradeable.
>
> And yet, supercomputers these days often have lots of GPUs.

Some do, some don’t. I’m not sure that GPUs are accepted as de rigueur in
supercomputer design yet. I think this is just another instance of Ivan
Sutherland’s “wheel of reincarnation”
<http://www.cap-lore.com/Hardware/Wheel.html>.

Terje Mathisen wrote:

> MitchAlsup1 wrote:
>> Lawrence D'Oliveiro wrote:
>>
>>> On Tue, 23 Apr 2024 18:36:49 -0500, BGB wrote:
>>
>>>> DIV:
>>>> Didn't bother with this.
>>>> Typically faked using multiply-by-reciprocal and taking the high result.
>>
>>> Another Cray-ism! ;)
>>
>> Not IEEE 754 legal.

> Well, it _is_ legal if you carry enough bits in your reciprocal...

Maybe--at best. There are certain pair of numerator::denominator that require
over 120-reciprocal bits* in order to deliver a properly rounded result using
an intermediate reciprocation.

(*) the reciprocal fraction bits--wider than long double.

> but at
> that point you would instead use a better algorithm to get the correct
> result both faster and using less power.

> Terje

On Wed, 24 Apr 2024 22:33:17 -0000 (UTC), Lawrence D'Oliveiro
<ldo@nz.invalid> wrote:
>On Wed, 24 Apr 2024 09:28:06 GMT, Anton Ertl wrote:

>> And yet, supercomputers these days often have lots of GPUs.

>Some do, some don’t. I’m not sure that GPUs are accepted as de rigueur in
>supercomputer design yet. I think this is just another instance of Ivan
>Sutherland’s “wheel of reincarnation”
><http://www.cap-lore.com/Hardware/Wheel.html>.

What do GPUs do, when they're included in supercomputers?

One of the things that those supercomputers that _do_ include GPUs are
praised for is being energy-efficient.

The problem with GPUs is that since their computational capabilities
are built on what the shader part does, their flexibility is limited.
This is what has made me think there could be a place for Cray-style
vectors. So some supercomputers don't have GPU accelerators, because
they're intended to work on problems for which GPU accelerators
wouldn't provide much help.

Since when GPUs _can_ be used, they save lots of electricity, I doubt
strongly that they're just a passing fad.

John Savard

On Wed, 24 Apr 2024 23:10:47 -0600, John Savard wrote:

> One of the things that those supercomputers that _do_ include GPUs are
> praised for is being energy-efficient.

That I never heard before. I heard it in relation to ARM CPUs, yes, GPUs,
no.

On Wed, 24 Apr 2024 22:29:36 -0000 (UTC)
Lawrence D'Oliveiro <ldo@nz.invalid> wrote:

> On Wed, 24 Apr 2024 09:18:56 GMT, Anton Ertl wrote:
>
> > But the Cray-1 is not an improvement on the Model 195.
>
> The Cray-1 was widely regarded as the fastest computer in the world,
> when it came out. Cruising speed of something over 80 megaflops,
> hitting bursts of about 120.
>
> IBM did try to compete in the “supercomputer” field for a while
> longer, but I think by about ten years later, it had given up.

In late 80s IBM joined forces with "attack of killing micros".
Their first POWER CPU was released in 1990 and did 82 MFLOPS (peak).

A single processor of contemporary Cray Y-MP was 4 times faster.
A single processor of older Cray-2 was almost 6 times faster, but by
1990 it was discontinued.
Wikipedia says that power consumption of Cray-2 was 150–200 kW,
probably for 4 processors with 2 GB of memory and peripherals.
I can't find data about power consumption of IBM Power processor. My
guess would be ~40 W for CPU and 1000-1500 W for a whole RS/6000
Model 550 with 1 GB of memory.

BTW, in the latest Top500 list you ca see IBM at #7 spot.
Things that carry name of Cray listed at #2 and #5. They are,
respectively, Intel Inside and AMD Inside.

According to Lawrence D'Oliveiro <ldo@nz.invalid>:
>On Wed, 24 Apr 2024 09:18:56 GMT, Anton Ertl wrote:
>
>> But the Cray-1 is not an improvement on the Model 195.
>
>The Cray-1 was widely regarded as the fastest computer in the world, when
>it came out. Cruising speed of something over 80 megaflops, hitting bursts
>of about 120.

Its main practical improvement was that you could get two Crays for the price
of one 360/195. (Not exactly, but close enough.)

>IBM did try to compete in the “supercomputer” field for a while longer,
>but I think by about ten years later, it had given up.

IBM had tried to make computers very fast by making them very
complicated. STRETCH was fantastically complex for something built out
of individual transistors. The /91 and /195 had instrucion queues and
reservation stations and loop mode. Cray went in the opposite
direction, making a much simpler computer where each individual bit
down to the chips and the wires, were as fast as possible.

In many ways it was a preview of RISC.

--
Regards,
John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

On Thu, 25 Apr 2024 05:39:55 -0000 (UTC)
Lawrence D'Oliveiro <ldo@nz.invalid> wrote:

> On Wed, 24 Apr 2024 23:10:47 -0600, John Savard wrote:
>
> > One of the things that those supercomputers that _do_ include GPUs
> > are praised for is being energy-efficient.
>
> That I never heard before. I heard it in relation to ARM CPUs, yes,
> GPUs, no.

If you never heard about *that*, I can only imagine what else you
didn't hear about supercomputers.

Back when Fugaku was new, it was highly praised for being GPU-less
design that matches and slightly exceeds an efficiency of
GPU-based (and other vector accelerator based) supercomputers. But that
was possible only because NVidea had unusually long pause between
successive generations of Tesla and at the same moment AMD and
Intel GPGPUs were not yet considered fit for serious supercomputing.

That was in November 2019. Never before or since.
Right now the best GPU-less entry on Green500 list is #48 (still the
same A64FX CPU as Fugaku, but smaller configuration) and it delivers 4x
less sustained FLOP/Watt than the top spot, based on NVIDIA H100.

According to Lawrence D'Oliveiro <ldo@nz.invalid>:
>On Wed, 24 Apr 2024 23:10:47 -0600, John Savard wrote:
>
>> One of the things that those supercomputers that _do_ include GPUs are
>> praised for is being energy-efficient.
>
>That I never heard before. I heard it in relation to ARM CPUs, yes, GPUs,
>no.

NVIDIA says their new Blackwell GPU takes 2000 watts, and is between
7x and 25x more power efficient than the current H100, but that's
still a heck of a lot of power. Data centers have had to come up with
higher capacity power and cooling when each rack can use 40 to 50KW.

I mean, my entire house is wired for 24KW and usually runs at more
like 4KW including a heat pump that heats the house.

--
Regards,
John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

On Thu, 25 Apr 2024 05:39:55 -0000 (UTC), Lawrence D'Oliveiro
<ldo@nz.invalid> wrote:

>On Wed, 24 Apr 2024 23:10:47 -0600, John Savard wrote:
>
>> One of the things that those supercomputers that _do_ include GPUs are
>> praised for is being energy-efficient.
>
>That I never heard before. I heard it in relation to ARM CPUs, yes, GPUs,
>no.

Here's one example of an item about this:

https://www.infoworld.com/article/2627720/gpus-boost-energy-efficiency-in-supercomputers.html

John Savard

John Levine wrote:

> According to Lawrence D'Oliveiro <ldo@nz.invalid>:
>>On Wed, 24 Apr 2024 09:18:56 GMT, Anton Ertl wrote:
>>
>>> But the Cray-1 is not an improvement on the Model 195.
>>
>>The Cray-1 was widely regarded as the fastest computer in the world, when
>>it came out. Cruising speed of something over 80 megaflops, hitting bursts
>>of about 120.

> Its main practical improvement was that you could get two Crays for the price
> of one 360/195. (Not exactly, but close enough.)

>>IBM did try to compete in the “supercomputer” field for a while longer,
>>but I think by about ten years later, it had given up.

> IBM had tried to make computers very fast by making them very
> complicated. STRETCH was fantastically complex for something built out
> of individual transistors. The /91 and /195 had instrucion queues and
> reservation stations and loop mode. Cray went in the opposite
> direction, making a much simpler computer where each individual bit
> down to the chips and the wires, were as fast as possible.

> In many ways it was a preview of RISC.

Seymore only did fast and simple, starting before the CDC 6600.....

On Thu, 25 Apr 2024 15:52:36 +0000
mitchalsup@aol.com (MitchAlsup1) wrote:
>
> Seymore only did fast and simple, starting before the CDC 6600.....

Do you attribute not exactly simple 6600 Scoreboard to Thornton?

Michael S wrote:

> On Thu, 25 Apr 2024 15:52:36 +0000
> mitchalsup@aol.com (MitchAlsup1) wrote:
>>
>> Seymore only did fast and simple, starting before the CDC 6600.....

> Do you attribute not exactly simple 6600 Scoreboard to Thornton?

If you measure simplicity by gate count--the scoreboard was considerably
simpler than the reservation station design of Tomasulo.

On Thu, 25 Apr 2024 17:34:32 +0000
mitchalsup@aol.com (MitchAlsup1) wrote:

> Michael S wrote:
>
> > On Thu, 25 Apr 2024 15:52:36 +0000
> > mitchalsup@aol.com (MitchAlsup1) wrote:
> >>
> >> Seymore only did fast and simple, starting before the CDC
> >> 6600.....
>
> > Do you attribute not exactly simple 6600 Scoreboard to Thornton?
>
> If you measure simplicity by gate count--the scoreboard was
> considerably simpler than the reservation station design of Tomasulo.

Both were far from simple by day's standards.

BTW, was not low gate count of Scoreboard mostly due to creative usage
of what was later named wired logic connections, i.e. something that
stopped working in high-speed VLSI around 1985-1990 ?