On 7/7/23 12:14, Kalevi Kolttonen wrote:
....
> He did describe his lab setup by saying that there are
> "hundreds of high-end multicore servers performing RTL
> simulations 24x7 using NFS".
>
> I have no idea what RTL is, ...
Scott can tell us, of course, but it might be one of these:
<https://en.wikipedia.org/wiki/RTL#Electronics>

kalevi@kolttonen.fi (Kalevi Kolttonen) writes:
>Bonita Montero <Bonita.Montero@gmail.com> wrote:
>> Am 07.07.2023 um 17:53 schrieb Kalevi Kolttonen:
>>
>>> Do you happen to have any good reasons why he would lie
>>> about their lab?
>>
>> 100GbE is a backbone-technology, maybe for linking switches
>> with further lower speed links. For which application would
>> you need a fileserver which can supply 12,5GB/s ?
>
>He did describe his lab setup by saying that there are
>"hundreds of high-end multicore servers performing RTL
>simulations 24x7 using NFS".

RTL (Register Transfer Language) aka HDL (Hardware Description Language)
aka Verilog. A hardware description language used
to design advanced (3nm process in our case) processor chips. With
lots of cores (36 or more) and the aforementioned hardware accelerator
blocks. The simulations actually simulate every gate clock by clock under various
directed and randomized test cases to ensure that the very expensive 3nm
process masks create a working chip. Each block is verified (simulated)
individual, groups of blocks are simulated together and various
full-chip simulations (the entire design) are also run. Then there are
the back-end tasks - floorplanning and pnr (Place and Route) to locate each
RTL block totaling many billion transistors. Then there is timing closure[*],
often with margins in the 10s of picosecond range.

[*] Ensuring that signal propogation in any particular combinatorial
path doesn't exceed the clock interval.

I'd also point out that the major customers for server grade processors
are AWS, Azure, Google and Facebook; all of which use 25/40Gbe(past) or 100Gbe(present)
rack to rack and rack to egress/ingress pipes and at least 10g rack to each server. As
David pointed out, with 192 core processors now available, 192 guest VMs can
easly saturate a 100Gbe link on a server.

Meanwhile, 400Gbe optical PHY modules became available a couple years ago
signaling that 200Gbe and 400Gbe are soon to follow.

Am 07.07.2023 um 19:00 schrieb Scott Lurndal:

> RTL (Register Transfer Language) aka HDL (Hardware Description Language)
> aka Verilog. A hardware description language used
> to design advanced (3nm process in our case) processor chips. With
> lots of cores (36 or more) and the aforementioned hardware accelerator
> blocks. The simulations actually simulate every gate clock by clock under various
> directed and randomized test cases to ensure that the very expensive 3nm
> process masks create a working chip. Each block is verified (simulated)
> individual, groups of blocks are simulated together and various
> full-chip simulations (the entire design) are also run. Then there are
> the back-end tasks - floorplanning and pnr (Place and Route) to locate each
> RTL block totaling many billion transistors. Then there is timing closure[*],
> often with margins in the 10s of picosecond range.

And why should that need such large and fast transfers ?

Am 07.07.2023 um 18:02 schrieb David Brown:
> On 07/07/2023 17:30, Bonita Montero wrote:
>> Am 07.07.2023 um 16:05 schrieb Scott Lurndal:
>>
>>> Our lab fileservers (using 25gb, 40gb and 100gb network adapters)
>>> serve hundreds of high-end multicore servers performing RTL simulations
>>> 24x7 using NFS.
>>
>> I don't believe you at all, that's pure phantasy.
>
> Do you think Scott is deliberately lying here? ...

Yes.

>
> It would appear you know practically nothing about servers or
> networking, especially for more demanding uses.  That's fine, of course
> - no one knows about everything, and most people have little interest in
> such things unless they actually need to know about them.  However, it
> is absurd to suggest that just because /you/ can't imagine what how such
> systems might be used, they don't exist.  And it is arrogant and
> obnoxious to accuse those who /do/ know, and /do/ use such systems, of
> lying about it.
>
>
>
>> But fileservers don't have high CPU-load anyway.
>> So manual segementation wouldn't hurt, even more
>> because in a LAN-segment you have jumbo frames.
>>
>>
>

Am 07.07.2023 um 18:14 schrieb Kalevi Kolttonen:

> He did describe his lab setup by saying that there are
> "hundreds of high-end multicore servers performing RTL
> simulations 24x7 using NFS".

Simulations needing constant I/Os with such a speed ???

> I have no idea what RTL is, it could perhaps be something
> to do with CPUs, I don't know. But it is obvious that what
> they are doing is no joke.
>
> br,
> KK

David Brown <david.brown@hesbynett.no> writes:
>On 07/07/2023 17:30, Bonita Montero wrote:
>> Am 07.07.2023 um 16:05 schrieb Scott Lurndal:
>>
>>> Our lab fileservers (using 25gb, 40gb and 100gb network adapters)
>>> serve hundreds of high-end multicore servers performing RTL simulations
>>> 24x7 using NFS.
>>
>> I don't believe you at all, that's pure phantasy.
>
>Do you think Scott is deliberately lying here? That's quite the accusation.

Indeed.

>
>It would appear you know practically nothing about servers or
>networking, especially for more demanding uses. That's fine, of course
>- no one knows about everything, and most people have little interest in
>such things unless they actually need to know about them. However, it
>is absurd to suggest that just because /you/ can't imagine what how such
>systems might be used, they don't exist. And it is arrogant and
>obnoxious to accuse those who /do/ know, and /do/ use such systems, of
>lying about it.
>
>
>
>> But fileservers don't have high CPU-load anyway.

I would argue that this isn't actually the case. Consider,
for example, file servers that support deduplication.

https://www.netapp.com/data-management/what-is-data-deduplication/

Scott Lurndal <scott@slp53.sl.home> wrote:
> RTL (Register Transfer Language)

When reading GCC documentation many years ago, I came
across this RTL:

https://gcc.gnu.org/onlinedocs/gcc-7.1.0/gccint/RTL.html

Does it have anything to do with the RTL you guys use?

br,
KK

On 07/07/2023 19:01, Bonita Montero wrote:
> Am 07.07.2023 um 19:00 schrieb Scott Lurndal:
>
>> RTL (Register Transfer Language) aka HDL (Hardware Description Language)
>> aka Verilog.  A hardware description language used
>> to design advanced (3nm process in our case) processor chips.  With
>> lots of cores (36 or more) and the aforementioned hardware accelerator
>> blocks.    The simulations actually simulate every gate clock by clock
>> under various
>> directed and randomized test cases to ensure that the very expensive 3nm
>> process masks create a working chip.    Each block is verified
>> (simulated)
>> individual,  groups of blocks are simulated together and various
>> full-chip simulations (the entire design) are also run.  Then there are
>> the back-end tasks - floorplanning and pnr (Place and Route) to locate
>> each
>> RTL block totaling many billion transistors.  Then there is timing
>> closure[*],
>> often with margins in the 10s of picosecond range.
>
> And why should that need such large and fast transfers ?
>

Since you haven't a clue what RTL simulation involves, while Scott
clearly does, why do you feel qualified to doubt him?

RTL simulation for these kinds of chips are /massive/ tasks. (Note that
the "36 or more cores" refers to the chips designed at Scott's company -
not the servers used for simulation, which will have as many cores as
economically feasible.) These chips will have billions of gates, which
need to be simulated. My guess is that they will be using many
thousands of processor cores spread across hundreds of machines for
doing the work, each machine having perhaps half a terabyte of ram.
Since you are simulating a single chip, each server can only simulate
part of it (in time and space), and you need a great deal of
communication between the machines to accurately model the interactions
between parts. So there will be very fast networking between the servers.

Then there are the files and databases involved, and output files. The
design files will be measured in gigabytes, as will the output files,
and there are hundreds of simulation servers accessing these files from
central repositories. Of course they will need massive bandwidths on
the file servers. And you will want the lowest practical latencies for
the file serving - even though file serving takes relatively little
processing power, you want to spread the different clients around the
cores of the server (with RSS) to minimise latencies. You do not want
everything bottlenecked through one core.

All this is, of course, very expensive. But if it saves a single failed
tape-out and prototype run of the chips, it will pay for itself.

Am 07.07.2023 um 19:32 schrieb David Brown:

> RTL simulation for these kinds of chips are /massive/ tasks.  (Note that
> the "36 or more cores" refers to the chips designed at Scott's company -
> not the servers used for simulation, which will have as many cores as
> economically feasible.)  These chips will have billions of gates, which
> need to be simulated.  My guess is that they will be using many
> thousands of processor cores spread across hundreds of machines for
> doing the work, each machine having perhaps half a terabyte of ram.
> Since you are simulating a single chip, each server can only simulate
> part of it (in time and space), and you need a great deal of
> communication between the machines to accurately model the interactions
> between parts.  So there will be very fast networking between the servers.

If you would do that with constant I/Os on the data and not in RAM
you would simulate almost nothing. Scott is lying.

Bonita Montero <Bonita.Montero@gmail.com> wrote:
> Scott is lying.

I am beginning to understand why some people have
killfiled you. I have never killfiled anybody and
never will, but please stop those ridiculous
accusations immediately. It is utter foolishness.

Based on several wise comments, there is absolutely
no reason to doubt Scott Lurndal's sincerity.
I am sure you are the only one who thinks he is lying.

br,
KK

Am 07.07.2023 um 20:09 schrieb Kalevi Kolttonen:

> Bonita Montero <Bonita.Montero@gmail.com> wrote:

>> Scott is lying.

> I am beginning to understand why some people have
> killfiled you. I have never killfiled anybody and
> never will, but please stop those ridiculous
> accusations immediately. It is utter foolishness.

Think about the lag between I/O latency and cache- / memory-latency.
Imagine what a calculation's speed on a data set constantly fetched
from I/O would be.

Bonita Montero <Bonita.Montero@gmail.com> wrote:
> Am 07.07.2023 um 20:09 schrieb Kalevi Kolttonen:
>
>> Bonita Montero <Bonita.Montero@gmail.com> wrote:
>
>>> Scott is lying.
>
>> I am beginning to understand why some people have
>> killfiled you. I have never killfiled anybody and
>> never will, but please stop those ridiculous
>> accusations immediately. It is utter foolishness.
>
> Think about the lag between I/O latency and cache- / memory-latency.
> Imagine what a calculation's speed on a data set constantly fetched
> from I/O would be.

No, I will not. Instead I will freely admit that
those demanding CPU simulations involving high-end
hardware devices are totally outside my very small
set of knowledge. It is specialized expert knowlegde
that only a handful of people have. Ordinary folks
never get access to labs like that.

It is obvious that Scott is one of those experts.
Calling him a liar is stone cold crazy... Oh no!

Again, why would he lie? To show off his skills,
or to brag about the workings of his advanced lab?

I seem to remember that Scott has been working in
the CS field ever since the big Burroughs machines
were hot. It means he has *decades of experience*.

This is my last comment concerning this matter.

br,
KK

Am 07.07.2023 um 20:34 schrieb Kalevi Kolttonen:

> No, I will not. Instead I will freely admit that
> those demanding CPU simulations involving high-end
> hardware devices are totally outside my very small
> set of knowledge. It is specialized expert knowlegde
> that only a handful of people have. Ordinary folks
> never get access to labs like that.

LOL

David Brown <david.brown@hesbynett.no> writes:
>On 07/07/2023 19:01, Bonita Montero wrote:
>> Am 07.07.2023 um 19:00 schrieb Scott Lurndal:
>>
>>> RTL (Register Transfer Language) aka HDL (Hardware Description Language)
>>> aka Verilog.  A hardware description language used
>>> to design advanced (3nm process in our case) processor chips.  With
>>> lots of cores (36 or more) and the aforementioned hardware accelerator
>>> blocks.    The simulations actually simulate every gate clock by clock
>>> under various
>>> directed and randomized test cases to ensure that the very expensive 3nm
>>> process masks create a working chip.    Each block is verified
>>> (simulated)
>>> individual,  groups of blocks are simulated together and various
>>> full-chip simulations (the entire design) are also run.  Then there are
>>> the back-end tasks - floorplanning and pnr (Place and Route) to locate
>>> each
>>> RTL block totaling many billion transistors.  Then there is timing
>>> closure[*],
>>> often with margins in the 10s of picosecond range.
>>
>> And why should that need such large and fast transfers ?
>>
>
>Since you haven't a clue what RTL simulation involves, while Scott
>clearly does, why do you feel qualified to doubt him?
>
>RTL simulation for these kinds of chips are /massive/ tasks. (Note that
>the "36 or more cores" refers to the chips designed at Scott's company -
>not the servers used for simulation, which will have as many cores as
>economically feasible.) These chips will have billions of gates, which
>need to be simulated. My guess is that they will be using many
>thousands of processor cores spread across hundreds of machines for
>doing the work, each machine having perhaps half a terabyte of ram.
>Since you are simulating a single chip, each server can only simulate
>part of it (in time and space), and you need a great deal of
>communication between the machines to accurately model the interactions
>between parts. So there will be very fast networking between the servers.
>
>Then there are the files and databases involved, and output files. The
>design files will be measured in gigabytes, as will the output files,

All correct. The output files consist of log files (debug information
from the test bench and test driver - when simulating a CPU, the log
also contains each disassembled instruction, all input and output registers
and any state changed as a result of instruction execution. This state is
compared against a golden reference model each instruction. There are also
optional 'wave' files, which contain a
clock-by-clock list of signal state changes for post-analysis and debugging
using visualization tools.

Simulating one real-time second of a 2Ghz design results in 2 billion
clocks, for which all the signal state data needs to be stored on disk.

>and there are hundreds of simulation servers accessing these files from
>central repositories. Of course they will need massive bandwidths on
>the file servers. And you will want the lowest practical latencies for
>the file serving - even though file serving takes relatively little
>processing power, you want to spread the different clients around the
>cores of the server (with RSS) to minimise latencies. You do not want
>everything bottlenecked through one core.
>
>All this is, of course, very expensive. But if it saves a single failed
>tape-out and prototype run of the chips, it will pay for itself.

Indeed.

"There are many clever tricks that are being used to lower costs
of designing that chip, but the biggest hinderances to bringing
that design to market is the cost of a mask set. On a foundry
process node, at 90nm to 45nm, mask sets cost on the order of
hundreds of thousands of dollars. At 28nm it moves beyond $1M. With
7nm, the cost increases beyond $10M, and now, as we cross the 3nm
barrier, mask sets will begin to push into the $40M range."

https://www.semianalysis.com/p/the-dark-side-of-the-semiconductor

You don't want to have to spin your design.

kalevi@kolttonen.fi (Kalevi Kolttonen) writes:
>Scott Lurndal <scott@slp53.sl.home> wrote:
>> RTL (Register Transfer Language)
>
>When reading GCC documentation many years ago, I came
>across this RTL:
>
> https://gcc.gnu.org/onlinedocs/gcc-7.1.0/gccint/RTL.html
>
>Does it have anything to do with the RTL you guys use?

Completely unrelated. Here's a sample:

logic [7:0] smb_ctl;
logic [7:0] smb_sta;
logic [31:0] grb_rdata;
logic csr_inf_rd_err;
logic [31:0] elided;

always_ff @(posedge cclk or negedge crst_l) begin
if (!crst_l) begin
smb_re <= 1'b0;
smb_we <= 1'b0;
csr_blk_rd_en <= 1'b0;
csr_blk_wr_en <= 1'b0;
end else if (inf_csr_we &&
(inf_csr_cmd_code == `SMB_CMD_CTL)) begin
smb_re <= inf_csr_wdata[`SMB_CTL_RD_START];
smb_we <= inf_csr_wdata[`SMB_CTL_WR_START];
csr_blk_rd_en <= inf_csr_wdata[`SMB_CTL_BLK_RD_EN];
csr_blk_wr_en <= inf_csr_wdata[`SMB_CTL_BLK_WR_EN];
end else begin
smb_re <= 1'b0;
smb_we <= 1'b0;
csr_blk_rd_en <= csr_blk_rd_en;
csr_blk_wr_en <= csr_blk_wr_en;
end
end

assign smb_ctl = {4'b0, csr_blk_wr_en, csr_blk_rd_en, 2'b0};
assign smb_sta = {2'b0, csr_inf_rd_err, csr_inf_rd_vld,
last_prcl_st[2:0], smb_prcl_err};

// Write Address
always_ff @(posedge cclk or negedge crst_l) begin
if (!crst_l) begin
smb_waddr <= 32'b0;
end else if (inf_csr_we) begin
if (inf_csr_cmd_code == `SMB_CMD_WADDR0)
smb_waddr[7:0] <= inf_csr_wdata;
if (inf_csr_cmd_code == `SMB_CMD_WADDR1)
smb_waddr[15:8] <= inf_csr_wdata;
if (inf_csr_cmd_code == `SMB_CMD_WADDR2)
smb_waddr[23:16] <= inf_csr_wdata;
if (inf_csr_cmd_code == `SMB_CMD_WADDR3)
smb_waddr[31:24] <= inf_csr_wdata;
end
end

On 7/7/2023 8:30 AM, Bonita Montero wrote:
> Am 07.07.2023 um 16:05 schrieb Scott Lurndal:
>
>> Our lab fileservers (using 25gb, 40gb and 100gb network adapters)
>> serve hundreds of high-end multicore servers performing RTL simulations
>> 24x7 using NFS.
>
> I don't believe you at all, that's pure phantasy.

Show me your proof that Scott is lying to us all?

;^/

> But fileservers don't have high CPU-load anyway.
> So manual segementation wouldn't hurt, even more
> because in a LAN-segment you have jumbo frames.
>
>

On 7/7/2023 10:58 AM, Bonita Montero wrote:
> Am 07.07.2023 um 19:32 schrieb David Brown:
>
>> RTL simulation for these kinds of chips are /massive/ tasks.  (Note
>> that the "36 or more cores" refers to the chips designed at Scott's
>> company - not the servers used for simulation, which will have as many
>> cores as economically feasible.)  These chips will have billions of
>> gates, which need to be simulated.  My guess is that they will be
>> using many thousands of processor cores spread across hundreds of
>> machines for doing the work, each machine having perhaps half a
>> terabyte of ram. Since you are simulating a single chip, each server
>> can only simulate part of it (in time and space), and you need a great
>> deal of communication between the machines to accurately model the
>> interactions between parts.  So there will be very fast networking
>> between the servers.
>
> If you would do that with constant I/Os on the data and not in RAM
> you would simulate almost nothing. Scott is lying.
>

Humm... Wow, Bonita, wow... ;^o

Am 07.07.2023 um 21:25 schrieb Chris M. Thomasson:

> Show me your proof that Scott is lying to us all?

It's unplausible that such a simulation isn't done
entirely in RAM but with constant I/O over the network.

On 07/07/2023 20:17, Bonita Montero wrote:
> Am 07.07.2023 um 20:09 schrieb Kalevi Kolttonen:
>
>> Bonita Montero <Bonita.Montero@gmail.com> wrote:
>
>>> Scott is lying.
>
>> I am beginning to understand why some people have
>> killfiled you. I have never killfiled anybody and
>> never will, but please stop those ridiculous
>> accusations immediately. It is utter foolishness.
>
> Think about the lag between I/O latency and cache- / memory-latency.
> Imagine what a calculation's speed on a data set constantly fetched
> from I/O would be.

I've tried to explain things to you, as has Scott. Clearly, it is all
so far beyond you that the Dunning Kruger effect has taken over.

Do you think /everyone/ here is lying, with some grand conspiracy theory
designed to annoy you personally? Or could it simply be that others
know more than you?

I don't have anything like these systems in our company. We have a
couple of servers for which 10 Gb is worth the cost and effort - nothing
like Scott's setup. But I have done a lot of weird things in network
setups, and I have designed electronics with a fair bit of networking -
we buy chips from the same kinds of companies that make devices like
Scott's, albeit much more mundane chips at lower speeds. And I have
done programmable logic design - again, on far smaller scales.

What Scott describes is entirely realistic for labs used in designing
devices like these.

Am 08.07.2023 um 14:51 schrieb David Brown:

> I've tried to explain things to you, as has Scott.  Clearly, it is
> all so far beyond you that the Dunning Kruger effect has taken over.

His reasoning is really 180 degrees opposite of mine,
to an extent that is easily recognizable as fantasy.

On Sat, 8 Jul 2023 04:45:29 +0200
Bonita Montero <Bonita.Montero@gmail.com> wrote:
>Am 07.07.2023 um 21:25 schrieb Chris M. Thomasson:
>
>> Show me your proof that Scott is lying to us all?
>
>It's unplausible that such a simulation isn't done
>entirely in RAM but with constant I/O over the network.

How do you think online multiplayer RPGs work?

Am 08.07.2023 um 16:59 schrieb Muttley@dastardlyhq.com:
> How do you think online multiplayer RPGs work?

The only transfer a small amout of state about the player over the
network. This data needs to be small so that it will reach the other
players very soon. So this is totally different than what was claimed.

On Sat, 8 Jul 2023 18:01:12 +0200
Bonita Montero <Bonita.Montero@gmail.com> wrote:
>Am 08.07.2023 um 16:59 schrieb Muttley@dastardlyhq.com:
>> How do you think online multiplayer RPGs work?
>
>The only transfer a small amout of state about the player over the

There's also a large amount of state about everything else going on
in the simulated world that has to be uploaded to each player which has to be
kept in sync.

On 7/8/2023 9:01 AM, Bonita Montero wrote:
> Am 08.07.2023 um 16:59 schrieb Muttley@dastardlyhq.com:
>> How do you think online multiplayer RPGs work?
>
> The only transfer a small amout of state about the player over the
> network. This data needs to be small so that it will reach the other
> players very soon. So this is totally different than what was claimed.
>
>

I think a lot of them (games) use UDP instead of TCP.

On 7/8/2023 7:19 AM, Bonita Montero wrote:
> Am 08.07.2023 um 14:51 schrieb David Brown:
>
>> I've tried to explain things to you, as has Scott.  Clearly, it is
>> all  so far beyond you that the Dunning Kruger effect has taken over.
>
> His reasoning is really 180 degrees opposite of mine,
> to an extent that is easily recognizable as fantasy.
>
>

Is this a projection?