Terje Mathisen <terje.mathisen@tmsw.no> schrieb:

> For a lot of crypto work, the data arrays are actually quite small,
> making the overhead of a coprocessor much more significant. I'd rather
> dedicate one of my 16 cores to do the AES-256 crypto, along with the key
> handling, authentication etc.

POWER has both - instructions for an AES round (one round per cycle
throughput) and a coprocessor.

I suppose it might be possible to push through more than one AES
round per cylce, but that would of course cost more silicon area.

On Tuesday, June 14, 2022 at 1:09:17 AM UTC+3, Thomas Koenig wrote:
> Terje Mathisen <terje.m...@tmsw.no> schrieb:
> > For a lot of crypto work, the data arrays are actually quite small,
> > making the overhead of a coprocessor much more significant. I'd rather
> > dedicate one of my 16 cores to do the AES-256 crypto, along with the key
> > handling, authentication etc.
> POWER has both - instructions for an AES round (one round per cycle
> throughput) and a coprocessor.
>
> I suppose it might be possible to push through more than one AES
> round per cylce, but that would of course cost more silicon area.

One round per cycle - throughput or latency?
1/cycle throughput - easy, but there are modes (e.g. CBC encrypt) where
it's not too useful. On the other hand, in other modes it's useful,
but only with non-trivial effort on SW part.
1 cycle latency - very very hard. At frequencies typical for POWER processors -
likely impossible.

On Monday, June 13, 2022 at 4:44:17 PM UTC-5, Ivan Godard wrote:
> On 6/13/2022 1:02 PM, MitchAlsup wrote:
> > On Monday, June 13, 2022 at 12:48:49 PM UTC-5, EricP wrote:
> >> Stephen Fuld wrote:
> >>> On 6/11/2022 3:12 PM, MitchAlsup wrote:

> >> I think you want a HW FORK instruction in here someplace
> >> which passes certain registers read only to the AP.
> > <
> > Send Message and Relinquish
> > and
> > Send Message and Continue
> > <
> > Both of these send and ABI full of arguments to a "method"
> > {Method is a means of identifying a thread to run where you may not
> > need to know its name of executing environment--Thread running
> > under a different Guest OS, for example.}
> >>
> >> A later HW JOIN would wait for and sync with the AP,
> >> and transfer back results to registers indicated with a bitmask.
> > <
> > Return Message
<
> That's a co-routine model, and it works so long as the information flow
> is visit-like and the duration is long enough to pay the overhead. If
> all you have is a visit, then everything looks like a visit. I cheer for
> your efforts toward a fast visit; it will help, but it's not a general
> solution. I expect trouble in the dispatch selector, and nightmares in
> debugging.
<
Send Message and Relinquish takes on-the-order of 20-ish cycles
from sender to control transfer arrives at service provider. {And
multiple threads can target the same servicer at essentially the
same time without bothering to check. Services are performed in
arrival order.}
<
Return Message takes another 20-ish cycles from returner to
originating sender.
<
Both of these 20-ish numbers are AFTER the core has wandered
through its cache hierarchy and gotten to the interconnect fabric.
I have seen designs where the adder would be as low as 5 and
seen designs where this was greater than 20.

On Monday, June 13, 2022 at 5:01:44 PM UTC-5, EricP wrote:
> Ivan Godard wrote:
> > On 6/13/2022 9:12 AM, EricP wrote:
> >> Terje Mathisen wrote:
> >> SIMD might have trouble dealing with a vector of structs
> >> with fields of different types and sizes.
> >> I suspect VVM would handle this naturally.
> >
> > Narrowing (or widening) of intermediate results seems difficult because
> > the number of lanes in the bypass changes on the fly. A widen needs to
> > buffer while the following stages of the pipe are double-pumped, while a
> > narrow needs to stall while earlier stages are. Of course, the whole
> > pipe could run at the lane count of the widest data and just ignore the
> > wastage, just as we ignore idle FUs that a code doesn't use.
<
> I didn't pay much attention to previous discussions about lanes
> as I was mostly interested in the VVM alias mapper and its
> forwarding mechanism, so apologies if this was already discussed.
>
> But it seems to me that this "packing problem" is straight
> forwardly handled by separating the operands into Int and FP streams,
> then just packing the operands together into packets
> based on the next higher naturally aligned field.
> When packet is full or the loop ends, dispatch the operand packet
> to a VVM Reservation station.
<
I do not foresee any problem performing widening or narrowing in
VVM implementations.
>
> The question is how to do inter-packet or scalar operand forwarding.
> And scheduling, specifically knowing when all the operands in a packet
> can launch together, and when they need to break apart because of
> FU resource limitations, or to avoid dependency deadlock.

Michael S <already5chosen@yahoo.com> schrieb:
> On Tuesday, June 14, 2022 at 1:09:17 AM UTC+3, Thomas Koenig wrote:
>> Terje Mathisen <terje.m...@tmsw.no> schrieb:
>> > For a lot of crypto work, the data arrays are actually quite small,
>> > making the overhead of a coprocessor much more significant. I'd rather
>> > dedicate one of my 16 cores to do the AES-256 crypto, along with the key
>> > handling, authentication etc.
>> POWER has both - instructions for an AES round (one round per cycle
>> throughput) and a coprocessor.
>>
>> I suppose it might be possible to push through more than one AES
>> round per cylce, but that would of course cost more silicon area.
>
> One round per cycle - throughput or latency?

Throughput. Of course, a complete AES has 14 rounds, so throughput
for AES-256 on POWER via vcipher is limited to one 128-bit word
every 14 cycles.

Each round xors five values together, with values from a lookup table.

One thin that might be limiting is register encoding - having a
256-bit key, and 128 bit of encrypted and decrypted data would
need to address eight 64-bit registers at once. Does fit lightly with
POWER's instructions :-)

> 1/cycle throughput - easy, but there are modes (e.g. CBC encrypt) where
> it's not too useful.

I meant per round, not per complete encryption.

>On the other hand, in other modes it's useful,
> but only with non-trivial effort on SW part.
> 1 cycle latency - very very hard. At frequencies typical for POWER processors -
> likely impossible.

Agreed.

On Tuesday, June 14, 2022 at 9:23:12 AM UTC+3, Thomas Koenig wrote:
> Michael S <already...@yahoo.com> schrieb:
> > On Tuesday, June 14, 2022 at 1:09:17 AM UTC+3, Thomas Koenig wrote:
> >> Terje Mathisen <terje.m...@tmsw.no> schrieb:
> >> > For a lot of crypto work, the data arrays are actually quite small,
> >> > making the overhead of a coprocessor much more significant. I'd rather
> >> > dedicate one of my 16 cores to do the AES-256 crypto, along with the key
> >> > handling, authentication etc.
> >> POWER has both - instructions for an AES round (one round per cycle
> >> throughput) and a coprocessor.
> >>
> >> I suppose it might be possible to push through more than one AES
> >> round per cylce, but that would of course cost more silicon area.
> >
> > One round per cycle - throughput or latency?
> Throughput. Of course, a complete AES has 14 rounds, so throughput
> for AES-256 on POWER via vcipher is limited to one 128-bit word
> every 14 cycles.
>
> Each round xors five values together, with values from a lookup table.

xor, shuffles octets, shuffles (permutes) bits in octets, something else...
I used to know this stuff (had implemented it in FPGA twice), but by now
only remember in principles, not details.

>
> One thin that might be limiting is register encoding - having a
> 256-bit key, and 128 bit of encrypted and decrypted data would
> need to address eight 64-bit registers at once. Does fit lightly with
> POWER's instructions :-)

I am not sure that I understand your concerns about encoding.
Each individual round has only 2 inputs - result of the previous round
and "round key" which is a pre-calculated product of Key and Index of
the round. If your architecture has enough of 128-bit registers then you
hold all round keys (14 in case of AES256) in registers all the time.
If it does not, then you hold as many as fit and load the rest dynamically
from memory (cache). The later case is more complicated, but on superscalar
HW not necessarily slower.

> > 1/cycle throughput - easy, but there are modes (e.g. CBC encrypt) where
> > it's not too useful.
> I meant per round, not per complete encryption.

Sure. I looked in the manual. On POWER9 latency=6 cycles.
The problem with CBC mode encrypt is that not only each round depend on the
previous, but each 128-bit block of the message depends on result of encryption
of the previous block. So, you can't encrypt blocks in parallel, making the
whole process effectively latency-bound.
Of course, in theory, you can achieve good throughput by encrypting several
messages in parallel, but practically it's extremely inconvenient.

On Power, I'd guess, the recommended solution is - fill gaps with multiple HW
threads. But that works only on servers or in benchmarks, not in real-world
client use cases.
Another solution, as you mentioned, is AES accelerator, which appears to be
specifically crafted for CBC encrypt mode. But use of accelerator is also less
convenient than use of SW library and at least on POWER9 accelerator appears to
be attached through DMA, so setup overhead is likely in hundreds or thousands
of clock cycles. And the throughput of accelerator (on POWER9) is not
particularly great - 6.4 Gbits/sec for AES256.
For comparison, nice operations, e.g. CBC decrypt, done in SW on core running at
3.5GHz could approach 30 Gbits/sec.

> >On the other hand, in other modes it's useful,
> > but only with non-trivial effort on SW part.
> > 1 cycle latency - very very hard. At frequencies typical for POWER processors -
> > likely impossible.
> Agreed.

Still, 6 is not the best possible latency at [relatively modest] POWER9 frequencies.
May be, when they designed the unit they were hoping that POWER9 will be clocked
higher, similarly to POWER8. Or, may be, it's was not considered important.

On Mon, 13 Jun 2022 14:33:27 -0700, Ivan Godard
<ivan@millcomputing.com> wrote:

> :
>there's an intellectual tractability problem: imperative time lines seem
>to be difficult enough for the next programmer off a bus, have we any
>hope to get the world to write in Lisp?

In many ways, Python appears to be a poor implementation of Lisp.

YMMV,
George

On Monday, June 13, 2022 at 4:33:30 PM UTC-5, Ivan Godard wrote:

> there's an intellectual tractability problem: imperative time lines seem
> to be difficult enough for the next programmer off a bus, have we any
> hope to get the world to write in Lisp?
<
If you run your C program through the preprocessor, you do get "lots of
infuriating small parentheses".

George Neuner <gneuner2@comcast.net> writes:
> In many ways, Python appears to be a poor implementation of Lisp.

My memory is until, hmmm, Python 2.7 I think, this:

l = [...]
sum = 0
for x in xrange(len(l)):
sum += l[x]

Was an O(n^2) operation (lists were basically car/cdr constructs).
2.7 switched them to a more obvious data structure. One saw very
aggressive use of tuples and generators to avoid subscripting
what--from the outside--looked like arrays.

I don't know about a "poor implementation", but it seems clear
that awareness of Lisp was present during Python's development.

(I wonder about generators and if/how they cribbed from Icon.)

Andy Valencia
Home page: https://www.vsta.org/andy/
To contact me: https://www.vsta.org/contact/andy.html

On 15/06/2022 04:03, Andy Valencia wrote:
> George Neuner <gneuner2@comcast.net> writes:
>> In many ways, Python appears to be a poor implementation of Lisp.
>
> My memory is until, hmmm, Python 2.7 I think, this:
>
> l = [...]
> sum = 0
> for x in xrange(len(l)):
> sum += l[x]
>
> Was an O(n^2) operation (lists were basically car/cdr constructs).
> 2.7 switched them to a more obvious data structure. One saw very
> aggressive use of tuples and generators to avoid subscripting
> what--from the outside--looked like arrays.
>
> I don't know about a "poor implementation", but it seems clear
> that awareness of Lisp was present during Python's development.
>
> (I wonder about generators and if/how they cribbed from Icon.)
>

How would that compare to the way you would write this in Python, rather
than what looks like a translation of some other language (C, perhaps?)
into Python?

In Python, you'd write :

l = [...]
s = sum(l)

If it is more complex than simple addition handled by the built-in "sum"
function, so that you need a loop, you'd write :

l = [...]
s = 0
for x in l :
sum += l

or you'd use a functional programming style :

s = reduce(lambda a, b : a + b, l)

Efficiency in Python is primarily about writing expressions that can be
handled by the C runtime and library - thus for big numerical
calculations, you use numpy (fast C libraries wrapped in Python
convenience) rather than raw Python code.

You can certainly criticise the speed of Python in many ways.
(Apparently the upcoming Python 3.11 can be up to twice as fast as 3.10,
which also means that even 3.10 is at least twice as slow as it needed
to be.) But it only makes sense to look at the speed of code written in
the appropriate way for any given language.

On 6/13/2022 2:33 PM, Ivan Godard wrote:
> On 6/13/2022 12:32 PM, Stephen Fuld wrote:
>> On 6/13/2022 12:01 PM, Ivan Godard wrote:

Big snip - In doing the snip, I may have messed up the attributions. If
so, I apologize.

>>> As  with all async units. What happens if the AP is used by an
>>> interrupt handler, inadvertently or not? By a second thread? We know
>>> how to deal with that for unshared i/o devices - at too great an
>>> overhead for a crypto  unit. Or we can save the state, like we do
>>> regs - and mess up call and task switch. Or stall until the unit
>>> completes, and have livelock and response time problems.
>>>
>>> The only solution found so far has been to push it all off to software.
>>
>> That is one of the things that, I hope, the hardware queuing mechanism
>> that Mitch alluded to some time ago in connection with I/O starts will
>> take care of, at least most of the time.
>>
>> My idea of an overview of how it would work is essentially, a "Start
>> Attached Processor" instruction would activate the AP if it is
>> available, else put an entry on the APs HW queue and then return to
>> executing instructions.  The SAP instruction acts similarly to other
>> long running instructions (Mitch's example was a load miss that had to
>> go to DRAM).  Execution can continue until the results of the
>> instruction are needed.  Eventually, execution will stall when no more
>> instructions that don't need the results are available.  This all fits
>> into the hardware, but, as I wrote in another post, I see some
>> problems with software handling, especially with long running
>> operations like encrypting a long string.  But essentially, most of
>> your questions are answered "Just like any other long running
>> instruction."
>>
>> Clearly, there are lots of things to work out, but I think this
>> approach is worth exploring.
>>
>>
>
> The question is "how long is long?"  We can, with complications,
> tolerate times in the hundreds of cycle by using an instruction model.
> We can,  with complications, tolerate times in the billions of cycles by
> using an i/o device model. What then for times in the thousands and
> millions of cycles?
>
> Uncanny valley.

I think that is pretty much correct. If Mitch can indeed deliver a
message passing mechanism (including messages to/from hardware
components) with the about 20 cycles latency he is aiming for, I think
that will fix the problem certainly for the millions of cycles, and even
for most of the thousands. We will see.

--
- Stephen Fuld
(e-mail address disguised to prevent spam)

MitchAlsup wrote:
[snip]
> In todays fabrication technology, I am more in the mood for a 512-bit
> "bus" to memory (cache line in a single beat).

This would seem to be impractical for an interface to conventional
DRAM. (Presumably this is another annoyance with the price-per-bit
tyranny. GDDR seems to allow wider chip interfaces. HBM seems to
use narrower channel interfaces between the interface chip and the
host while being very wide between DRAM chips and the interface
chip — it is interesting that the substantial area loss from
through silicon vias is acceptable for the bandwidth gains but
similar area costs are not acceptable for latency reduction.) DDR5
DIMMs even split the 64-bit data interface into two channels!

> Heck, once you get 16-32
> cores on a die you are going to need this kind of BW.

Do wider channels really help bandwidth that much compared to more
narrower channels given high thread count (and out-of-order [and
perhaps prefetching] providing some memory-level parallelism)?
Caches would seem to reduce miss rates such that multiple threads
would be generating many misses at the same time (i.e., misses
seem like to be phased and the phases for different threads are
unlikely to overlap — good for localized bandwidth demand, bad for
using threads to increase MLP).

I may have mentioned some time the concept of a wide interface
with multiple phases in a intra-block non-uniform latency cache.
This would be applying width pipelining to cache access; the first
chunk (possibly predicted critical chunk placed closest to
expected user) arrives faster and later accesses can use those
interface wires for their first chunk in the next cycle. Higher
latency chunks might be the same size or be larger (depending on
physical design tradeoffs — e.g., the first chunks might be in
lower density, lower latency SRAM arrays and favor only 64-bit
width), but each beat a full cache block could be transferred
(high bandwidth).

I am skeptical that such intra-block NUCA makes sense. The concept
is sufficiently obvious that if it was worthwhile it would already
have been implemented. (I assume that such an implementation would
have been detected even if never documented.) Even block-based
NUCA does not seem to be generally implemented.

I mention this mainly to point out that a cache block wide
interface does not have to transfer the entire cache block in a
single beat — width pipelining is possible. Yet I would also love
for my mentioning such to inspire an actual implementation.☺

"Paul A. Clayton" <paaronclayton@gmail.com> writes:
>MitchAlsup wrote:
>[snip]
>> In todays fabrication technology, I am more in the mood for a 512-bit
>> "bus" to memory (cache line in a single beat).
>
>This would seem to be impractical for an interface to conventional
>DRAM.

I don't know if Mitch Alsup is thinking about the interface to DRAM,
or to outer cache levels.

For DRAM, it would certainly impractical, because the difference
between DRAM cycle time and transfer speeds has led to burst lengths
of at least 16 beats for DDR5 (8 for DDR4 and DDR3, 4 for DDR2, 2 for
DDR). In order to serve 64-byte cache lines in 16 beats, DDR5 has
narrowed each channel to 32 bits. You don't get any real speedup with
a 512-bit channel with burst lengths of 1 (i.e., SDR RAM), because you
now have to wait for the DRAM cycle to complete before you can perform
the next access.

But I am sure that Mitch Alsup knows this and he was thinking about
interfaces to outer cache levels.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>

"Paul A. Clayton" <paaronclayton@gmail.com> writes:
>MitchAlsup wrote:
>[snip]
>> In todays fabrication technology, I am more in the mood for a 512-bit
>> "bus" to memory (cache line in a single beat).
>
>This would seem to be impractical for an interface to conventional
>DRAM. (Presumably this is another annoyance with the price-per-bit
>tyranny. GDDR seems to allow wider chip interfaces. HBM seems to
>use narrower channel interfaces between the interface chip and the
>host while being very wide between DRAM chips and the interface
>chip — it is interesting that the substantial area loss from
>through silicon vias is acceptable for the bandwidth gains but
>similar area costs are not acceptable for latency reduction.) DDR5
>DIMMs even split the 64-bit data interface into two channels!

Most modern multicore CPUs use a mesh structure in place of a bus.
The ARM CMN-600, for example, has two 256-bit data channels, one
for each direction, and supports up to 16 memory controllers.

https://developer.arm.com/documentation/100180/0302/?lang=en

Striping the physical address space across multiple memory controllers
allows greater bandwidth from memory to LLC/SLC; which is distributed across
points in the mesh.

Other processor vendor interconnect implementations may have wider
data channels (e.g. 512 bits) (Intel Mesh or AMD InfinityFabric,
although it's not clear that Intel has completely abandoned the
ring structured bus or crossbar switch).

https://ieeexplore.ieee.org/document/6275442?reload=true

Typical internal busses to PCIe or on-board I/O controllers are already 256,
512 (or more) bits wide in modern SoCs (consider the bandwidth requirements
for a DPU managing multiple 100/200/400 Gb/sec ethernet ports at line rate, for example).

On Saturday, November 26, 2022 at 10:22:39 AM UTC-6, Anton Ertl wrote:
> "Paul A. Clayton" <paaron...@gmail.com> writes:
> >MitchAlsup wrote:
> >[snip]
> >> In todays fabrication technology, I am more in the mood for a 512-bit
> >> "bus" to memory (cache line in a single beat).
> >
> >This would seem to be impractical for an interface to conventional
> >DRAM.
> I don't know if Mitch Alsup is thinking about the interface to DRAM,
> or to outer cache levels.
>
> For DRAM, it would certainly impractical, because the difference
> between DRAM cycle time and transfer speeds has led to burst lengths
> of at least 16 beats for DDR5 (8 for DDR4 and DDR3, 4 for DDR2, 2 for
> DDR). In order to serve 64-byte cache lines in 16 beats, DDR5 has
> narrowed each channel to 32 bits. You don't get any real speedup with
> a 512-bit channel with burst lengths of 1 (i.e., SDR RAM), because you
> now have to wait for the DRAM cycle to complete before you can perform
> the next access.
<
Direct copy and paste from Wikipedia::
<
HBM memory bus is very wide in comparison to other DRAM memories
such as DDR4 or GDDR5. An HBM stack of four DRAM dies (4‑Hi) has two
128‑bit channels per die for a total of 8 channels and a width of 1024 bits
in total. A graphics card/GPU with four 4‑Hi HBM stacks would therefore
have a memory bus with a width of 4096 bits. In comparison, the bus width
of GDDR memories is 32 bits, with 16 channels for a graphics card with a
512‑bit memory interface.[12] HBM supports up to 4 GB per package.
>
> But I am sure that Mitch Alsup knows this and he was thinking about
> interfaces to outer cache levels.
>
> - anton
> --
> 'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
> Mitch Alsup, <c17fcd89-f024-40e7...@googlegroups.com>

Anton Ertl wrote:
> "Paul A. Clayton" <paaronclayton@gmail.com> writes:
>> MitchAlsup wrote:
>> [snip]
>>> In todays fabrication technology, I am more in the mood for a 512-bit
>>> "bus" to memory (cache line in a single beat).
>>
>> This would seem to be impractical for an interface to conventional
>> DRAM.
>
> I don't know if Mitch Alsup is thinking about the interface to DRAM,
> or to outer cache levels >
> For DRAM, it would certainly impractical, because the difference
> between DRAM cycle time and transfer speeds has led to burst lengths
> of at least 16 beats for DDR5 (8 for DDR4 and DDR3, 4 for DDR2, 2 for
> DDR). In order to serve 64-byte cache lines in 16 beats, DDR5 has
> narrowed each channel to 32 bits. You don't get any real speedup with
> a 512-bit channel with burst lengths of 1 (i.e., SDR RAM), because you
> now have to wait for the DRAM cycle to complete before you can perform
> the next access.
>
> But I am sure that Mitch Alsup knows this and he was thinking about
> interfaces to outer cache levels.

I am also sure he knows that, but I am not certain if he meant
that he would prefer a wide interface to memory even if he knew it
would not happen, similarly to how he would prefer a greater
emphasis on latency for DRAM.

I do not think the cycle time is physically required to be as long
as it is (other than for cost-per-bit optimization). Obviously
designing each chip as if it was two chips would allow halving the
cycle time when addressing different "chips" and I rather suspect
that with sufficient power accessing different banks within a chip
would not be problematic. Even the single internal array cycle
time is presumably a cost-driven choice.

Mitch Alsup has mentioned the latency achieved for an embedded
DRAM design he worked on. IBM has used embedded DRAM for outer
levels of cache in POWER and zArch. These designs included not
merely latency improvements but "cycle time" improvements (though
I seem to recall IBM's design had cycle time limits much greater
than SRAM).

(I think it is kind of sad that one cannot get, e.g., 1 GiB of low
latency DRAM and use it effectively. Effective use would probably
require early L3 cache miss detection/prediction, which would be
somewhat challenging I suspect. With more integrated memory — on
board or even in-package — there should be opportunities not only
for increasing bandwidth but also for reducing latency.)

On Saturday, November 26, 2022 at 1:46:06 PM UTC-6, Paul A. Clayton wrote:
> Anton Ertl wrote:
> > "Paul A. Clayton" <paaron...@gmail.com> writes:
> >> MitchAlsup wrote:

> > But I am sure that Mitch Alsup knows this and he was thinking about
> > interfaces to outer cache levels.
<
> I am also sure he knows that, but I am not certain if he meant
> that he would prefer a wide interface to memory even if he knew it
> would not happen, similarly to how he would prefer a greater
> emphasis on latency for DRAM.
<
It depends on what your technology allows you to do. at one end
of the scale is 1 wire operating at optical frequencies, at the other
end is the number of wires equal to a cache line. You can't go lower
than 1 wire, and you don't need more than 1 cache line per cycle
per step in the interconnect.
>
> I do not think the cycle time is physically required to be as long
> as it is (other than for cost-per-bit optimization). Obviously
> designing each chip as if it was two chips would allow halving the
<
The prominent delay is not logic, but wires, therefore you get the
delay of 71% if you split the chip in half. SQRT( ½ )
<
> cycle time when addressing different "chips" and I rather suspect
> that with sufficient power accessing different banks within a chip
> would not be problematic. Even the single internal array cycle
> time is presumably a cost-driven choice.
>
> Mitch Alsup has mentioned the latency achieved for an embedded
> DRAM design he worked on. IBM has used embedded DRAM for outer
> levels of cache in POWER and zArch. These designs included not
> merely latency improvements but "cycle time" improvements (though
> I seem to recall IBM's design had cycle time limits much greater
> than SRAM).
<
I built a DRAM macro--including layout and SPICE modeling. Using
the same sense amplifiers as I used for my SRAM design, the access
time for DRAM was only a few picoseconds longer than SRAM and
the DRAM was at least 6× more dense. Cycle time was 3× but access
time was not usefully different.
<
What most people don't realize is that most of the delay in accessing
DRAM has to do with intra-chip wire delay, not cell access. Another large
component is waiting for Power and Ground to stabilize after an ACTivate
or PREcharge; so one can (again) use those sensitive analog sense amps.
>
> (I think it is kind of sad that one cannot get, e.g., 1 GiB of low
> latency DRAM and use it effectively. Effective use would probably
> require early L3 cache miss detection/prediction, which would be
> somewhat challenging I suspect. With more integrated memory — on
> board or even in-package — there should be opportunities not only
> for increasing bandwidth but also for reducing latency.)
<
DRAM manufactures spend Billions (20B+) to put up FABs to manufacture
DRAMS. DRAM suppliers put up $30M factories to build packages to put
those DRAM chips in; the cost of the chip and the cost of the package are
rather equal in true volume manufacture.
<
Would you be willing to pay 50% more for a DRAM with 2× as many data
pins ? {Yeah, me neither}.

"Paul A. Clayton" <paaronclayton@gmail.com> writes:
>I do not think the cycle time is physically required to be as long
>as it is

I think it is. The cycle time is determined by the time for reading
the contents of the DRAM cells, amplifying the results, then writing
the reconstructed content of the bank back.

>(other than for cost-per-bit optimization). Obviously
>designing each chip as if it was two chips would allow halving the
>cycle time when addressing different "chips"

Yes, you can perform n reads per DRAM cycle with n channels; people
are doing that with independent channel contents, but yes, if you are
prepared to get only half the usable memory for the same money, you
can write the same stuff to two channels at the same time, and read
from both channels at the same. Not sure what's this is intended to
achieve, though.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>

On Sunday, November 27, 2022 at 1:47:58 AM UTC+2, Anton Ertl wrote:
> "Paul A. Clayton" <paaron...@gmail.com> writes:
> >I do not think the cycle time is physically required to be as long
> >as it is
> I think it is. The cycle time is determined by the time for reading
> the contents of the DRAM cells, amplifying the results, then writing
> the reconstructed content of the bank back.

That's true for Row Miss and even more so for Row Conflict.
But not for Row Hit, which is hopefully the most common case.

> >(other than for cost-per-bit optimization). Obviously
> >designing each chip as if it was two chips would allow halving the
> >cycle time when addressing different "chips"
> Yes, you can perform n reads per DRAM cycle with n channels; people
> are doing that with independent channel contents, but yes, if you are
> prepared to get only half the usable memory for the same money, you
> can write the same stuff to two channels at the same time, and read
> from both channels at the same. Not sure what's this is intended to
> achieve, though.
> - anton
> --
> 'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
> Mitch Alsup, <c17fcd89-f024-40e7...@googlegroups.com>

On Sunday, November 27, 2022 at 2:20:54 AM UTC+2, Michael S wrote:
> On Sunday, November 27, 2022 at 1:47:58 AM UTC+2, Anton Ertl wrote:
> > "Paul A. Clayton" <paaron...@gmail.com> writes:
> > >I do not think the cycle time is physically required to be as long
> > >as it is
> > I think it is. The cycle time is determined by the time for reading
> > the contents of the DRAM cells, amplifying the results, then writing
> > the reconstructed content of the bank back.
> That's true for Row Miss and even more so for Row Conflict.
> But not for Row Hit, which is hopefully the most common case.

Oh, I answered before thinking.
No, it's not true even for Row Miss/Conflict.
Row Miss today is ~13 ns, a cycle time of control machine within
DRAM chip is typically 2.5ns. So, even today Row Miss takes
multiple cycles which clearly demonstrates that the two are
in principle not related to each other.
But I am not sure that Paul is correct, that 2.5ns is due to optimization
for cost per bit. More likely it's due to optimization for several things
but primarily for power and for simplicity of interface of DRAM chip.
Faster control machine will likely require one more supply voltage.

> > >(other than for cost-per-bit optimization). Obviously
> > >designing each chip as if it was two chips would allow halving the
> > >cycle time when addressing different "chips"
> > Yes, you can perform n reads per DRAM cycle with n channels; people
> > are doing that with independent channel contents, but yes, if you are
> > prepared to get only half the usable memory for the same money, you
> > can write the same stuff to two channels at the same time, and read
> > from both channels at the same. Not sure what's this is intended to
> > achieve, though.
> > - anton
> > --
> > 'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
> > Mitch Alsup, <c17fcd89-f024-40e7...@googlegroups.com>

MitchAlsup wrote:
> On Saturday, November 26, 2022 at 1:46:06 PM UTC-6, Paul A. Clayton wrote:
[snip]
>> I do not think the cycle time is physically required to be as long
>> as it is (other than for cost-per-bit optimization). Obviously
>> designing each chip as if it was two chips would allow halving the
> <
> The prominent delay is not logic, but wires, therefore you get the
> delay of 71% if you split the chip in half. SQRT( ½ )

My statement was a result of misunderstanding. I had thought the
cycle time referenced was for the chip and not the specific DRAM
array(s) accessed. I only meant that clearly two different DRAM
chips could be accessed with minimal delay between accesses since
they are completely independent. Accesses to the same array but a
different row — what was intended by Anton Ertl — would have a
longer cycle time.

I was not thinking about latency of internal array/sense amplifier
row to external pads, which seems to be what your statement concerns.

(This does bring up the possibility I may have mentioned sometime
of width pipelined access from multiple arrays with varying
distances from the output pads and possibly variation in row size
(wire delay and sense amplifier sensitivity) or even capacitor
sizing. This would be exploiting the narrower transmission to hide
latency from more distant or otherwise higher latency arrays.
Accessing more arrays would obviously cost more energy and would
amplify Row Hammer effects.)

[snip]
> I built a DRAM macro--including layout and SPICE modeling. Using
> the same sense amplifiers as I used for my SRAM design, the access
> time for DRAM was only a few picoseconds longer than SRAM and
> the DRAM was at least 6× more dense. Cycle time was 3× but access
> time was not usefully different.

Obviously higher density would reduce wire delay at a larger scale
*and* facilitate more on-chip capacity for which higher bandwidth
would be less expensive.

> What most people don't realize is that most of the delay in accessing
> DRAM has to do with intra-chip wire delay, not cell access. Another large
> component is waiting for Power and Ground to stabilize after an ACTivate
> or PREcharge; so one can (again) use those sensitive analog sense amps.

There is also latency on the other end: checking three levels of
cache, possibly multiple on-chip-network hopes to the memory
controller, memory controller delay, and probably other factors.
Some of these could perhaps be reduced (cache miss prediction or
hit filtering, on-chip NUMA [possibly even sending on-chip data to
a processing node closer to the memory interface, overlapping that
latency with the memory access latency???]). Chip-to-chip latency
is also non-zero.

>> (I think it is kind of sad that one cannot get, e.g., 1 GiB of low
>> latency DRAM and use it effectively. Effective use would probably
>> require early L3 cache miss detection/prediction, which would be
>> somewhat challenging I suspect. With more integrated memory — on
>> board or even in-package — there should be opportunities not only
>> for increasing bandwidth but also for reducing latency.)
>
> DRAM manufactures spend Billions (20B+) to put up FABs to manufacture
> DRAMS. DRAM suppliers put up $30M factories to build packages to put
> those DRAM chips in; the cost of the chip and the cost of the package are
> rather equal in true volume manufacture.
>
> Would you be willing to pay 50% more for a DRAM with 2× as many data
> pins ? {Yeah, me neither}.

Given that High Bandwidth Memory has a significant market, at
least some users are willing to pay for higher *bandwidth*.
Perhaps a market might be build for lower latency.

A 50% per bit price premium might not be horrifying for a
half-latency memory with 10% the capacity of main memory. If
memory was half the system cost, this would increase system cost
by less than 10%.

(This also brings the question of whether a "small low latency"
memory could be sufficiently useful. The effectiveness would
depend on the workload, software optimization, *and* hardware
optimization. Using such as an L4 cache has the danger of
increasing average access time if miss detection adds much
latency. Using such as a fast memory presents utilization issues
from page-sized allocations and other factors.)

I would probably not be willing to pay that premium because my
current computer is fast enough for writing, content display (even
most PDFs — 2000-page ISA manuals can be slow — and heavily
javascript inflicted web pages), and light gaming. I also tend to
be a little stingy on personal spending — partially financial
constraints, partially ethics, and partially self-conceit issues.
I also tend to fall prey to the false emphasis on low
unit/acquisition cost versus value and total cost, to middle
option bias, and to other fallacies as well as decision phobia.

Some organizations are able to make somewhat more rational and
information-based decisions.

Since such would presumably be packaged with the CPU, some of the
packaging cost relative to pad count considerations might be
reduced. (The packaging cost would be higher, but Apple is already
integrating DRAM into the processor package [I think].)