On 06/07/2023 05:38, Bonita Montero wrote:
> Am 05.07.2023 um 15:49 schrieb Scott Lurndal:
>
>> Your ignorance is showing.  My company makes NICs.  Very powerful
>> NICs.   Which manage streams (the programmable TCAMs allow software
>> to select the packet fields that get CAMd when selecting the target
>> CPU for the interrupt).
>
> Show me the documentation.

I have no idea what NIC's Scott's company makes, but it should not be
too hard to google a bit about "intelligent" NIC's. They range from
relatively simple acceleration such as RSS and TCP/IP checksums to
devices that combine switching, routing, filtering and server facilities
on the NIC card, running their own Linux system on the card.

The simpler TCP/IP offload engines are rarely used with Linux, because
they significantly complicate the network stacks, are much more limited
than networking in software, and modern processors can do the job faster
in software than the devices can do it in hardware. (It's a bit like
hardware RAID cards in that aspect.)

But when you have 100 Gbps and faster NICs, with large multi-core
processors, then it is not at all uncommon to use RSS (and similar) to
hold multiple queues in the NIC, each with their own interrupt that is
then routed to specific CPU cores. The aim is to maximise cache reuse
and reduce latency. More sophisticated cards will do filtering, NAT,
transformations, etc., as well as queueing and interrupt routing based
on content. And these kinds of cards are almost always used with Linux.

Am 06.07.2023 um 13:49 schrieb David Brown:

> I have no idea what NIC's Scott's company makes, but it should
> not be too hard to google a bit about "intelligent" NIC's.

Google for NIC task-offloading and you don't find anything that
supports his statements at least for Linux.

> The simpler TCP/IP offload engines are rarely used with Linux, ...

And where are they actually supportet ?
With Windows, that supports that kind of task-offloading ?

> But when you have 100 Gbps and faster NICs, ...

100GbE is rather a backbone technology.

Bonita Montero <Bonita.Montero@gmail.com> writes:
>Am 05.07.2023 um 15:49 schrieb Scott Lurndal:
>
>> Your ignorance is showing. My company makes NICs. Very powerful
>> NICs. Which manage streams (the programmable TCAMs allow software
>> to select the packet fields that get CAMd when selecting the target
>> CPU for the interrupt).
>
>Show me the documentation.

This is an older generation:

https://doc.dpdk.org/guides-20.11/platform/octeontx2.html

And the current, which builds on the prior generations, going
all the way back to the original MIPS-based Octeons more than a decade ago.

https://www.marvell.com/company/media-kit/marvell-octeon-tx2-and-octeon-fusion-press-kit.html

On Thursday, July 6, 2023 at 5:32:16 PM UTC+3, Scott Lurndal wrote:
> Bonita Montero <Bonita....@gmail.com> writes:
> >Am 05.07.2023 um 15:49 schrieb Scott Lurndal:
> >
> >> Your ignorance is showing. My company makes NICs. Very powerful
> >> NICs. Which manage streams (the programmable TCAMs allow software
> >> to select the packet fields that get CAMd when selecting the target
> >> CPU for the interrupt).
> >
> >Show me the documentation.
> This is an older generation:
>
> https://doc.dpdk.org/guides-20.11/platform/octeontx2.html
>
> And the current, which builds on the prior generations, going
> all the way back to the original MIPS-based Octeons more than a decade ago.
>
> https://www.marvell.com/company/media-kit/marvell-octeon-tx2-and-octeon-fusion-press-kit.html

That's too high end for just about everybody, and I don't think that it really used in any
Marvell-made NIC, not even in the ultra-high-end FastLinQ 45000 Series.

This one more relevant:
marvell-fastLinq-edge-aqc113-aqc113c-aqc113cs-aqc114cs-aqc115c-aqc116c-product-brief.pdf

Michael S <already5chosen@yahoo.com> writes:
>On Thursday, July 6, 2023 at 5:32:16=E2=80=AFPM UTC+3, Scott Lurndal wrote:
>> Bonita Montero <Bonita....@gmail.com> writes:=20
>> >Am 05.07.2023 um 15:49 schrieb Scott Lurndal:=20
>> >=20
>> >> Your ignorance is showing. My company makes NICs. Very powerful=20
>> >> NICs. Which manage streams (the programmable TCAMs allow software=20
>> >> to select the packet fields that get CAMd when selecting the target=20
>> >> CPU for the interrupt).=20
>> >=20
>> >Show me the documentation.
>> This is an older generation:=20
>>=20
>> https://doc.dpdk.org/guides-20.11/platform/octeontx2.html=20
>>=20
>> And the current, which builds on the prior generations, going=20
>> all the way back to the original MIPS-based Octeons more than a decade ag=
>o.=20
>>=20
>> https://www.marvell.com/company/media-kit/marvell-octeon-tx2-and-octeon-f=
>usion-press-kit.html
>
>
>That's too high end for just about everybody, and I don't think that it rea=
>lly used in any
>Marvell-made NIC, not even in the ultra-high-end FastLinQ 45000 Series.

Marvell has a wide range of network interface chips. The OcteonTX
series have both low-end and high-end entries. The low-end (MIPS and ARM64)
Octeons are used in home routers, SOHO routers and some low-end enterprise
routers. The high-end OcteonTx are used in high-end enterprise routers,
cellular and backbone infrastructure.

However, the feature that started this conversation, recieve-side scaling
is widely used and supported by many different NIC vendors, including Intel's
PRO line of network interfaces.

Am 06.07.2023 um 16:31 schrieb Scott Lurndal:

> This is an older generation:
>
> https://doc.dpdk.org/guides-20.11/platform/octeontx2.html
>
> And the current, which builds on the prior generations, going
> all the way back to the original MIPS-based Octeons more than a decade ago.
>
> https://www.marvell.com/company/media-kit/marvell-octeon-tx2-and-octeon-fusion-press-kit.html

That are infrastructure processors. I don't see much sense in
having such >= 25GbE networking in a server since the processing
behind such flows of any server application cost magnitudes more
CPU time than flow segmentation. The devices shown in the PDF
I've checkt confirm that.

On Thursday, July 6, 2023 at 6:01:06 PM UTC+3, Scott Lurndal wrote:
> Michael S <already...@yahoo.com> writes:
> >On Thursday, July 6, 2023 at 5:32:16=E2=80=AFPM UTC+3, Scott Lurndal wrote:
> >> Bonita Montero <Bonita....@gmail.com> writes:=20
> >> >Am 05.07.2023 um 15:49 schrieb Scott Lurndal:=20
> >> >=20
> >> >> Your ignorance is showing. My company makes NICs. Very powerful=20
> >> >> NICs. Which manage streams (the programmable TCAMs allow software=20
> >> >> to select the packet fields that get CAMd when selecting the target=20
> >> >> CPU for the interrupt).=20
> >> >=20
> >> >Show me the documentation.
> >> This is an older generation:=20
> >>=20
> >> https://doc.dpdk.org/guides-20.11/platform/octeontx2.html=20
> >>=20
> >> And the current, which builds on the prior generations, going=20
> >> all the way back to the original MIPS-based Octeons more than a decade ag=
> >o.=20
> >>=20
> >> https://www.marvell.com/company/media-kit/marvell-octeon-tx2-and-octeon-f=
> >usion-press-kit.html
> >
> >
> >That's too high end for just about everybody, and I don't think that it rea> >lly used in any
> >Marvell-made NIC, not even in the ultra-high-end FastLinQ 45000 Series.
> Marvell has a wide range of network interface chips. The OcteonTX
> series have both low-end and high-end entries. The low-end (MIPS and ARM64)
> Octeons are used in home routers, SOHO routers and some low-end enterprise
> routers. The high-end OcteonTx are used in high-end enterprise routers,
> cellular and backbone infrastructure.
>

Exactly. My point is that Octeon chips are not used in NICs.

> However, the feature that started this conversation, recieve-side scaling
> is widely used and supported by many different NIC vendors, including Intel's
> PRO line of network interfaces.

That's, I think, is obvious to everybody except Bonita.
In the past RSS was supported even on non-pro or semi-pro Intel NICs like
82574L that I use to post this message. I didn't check if it's still supported
in this range. Practically, it's no longer needed for 1Gbe since even cheap
today's CPU will handle everything without breaking a sweet. But if they had
in the past, may be they retain it just for sake of check box checked.

On 06/07/2023 16:28, Bonita Montero wrote:
> Am 06.07.2023 um 13:49 schrieb David Brown:
>
>> I have no idea what NIC's Scott's company makes, but it should
>> not be  too hard to google a bit about "intelligent" NIC's.
>
> Google for NIC task-offloading and you don't find anything that
> supports his statements at least for Linux.
>

Which statements, /exactly/, do you disagree with? Try to quote his
words, then perhaps explain in your own words what you think he meant.

(Note that Scott did not mention TOE - /you/ did. He talked about RSS.)

>> The simpler TCP/IP offload engines are rarely used with Linux, ...
>
> And where are they actually supportet ?

They /are/ supported in Linux - they are just not /used/ with Linux,
because they TOE is a crappy solution. Generally you get faster
throughput by disabling TOE.

> With Windows, that supports that kind of task-offloading ?

TOE is used with Windows, because it is a crappy OS for server work.
(Windows is fine for many other aspects - all systems have their good
points and bad points.) Windows' network stack is very limited - it can
only do simple tasks, so it does no harm to have a simple accelerators.

It's like RAID. Windows - even the server versions - does not have
software RAID worth mentioning. So you have to use hardware RAID
solutions. Linux has several different types of software RAID to suit
different needs, with a great deal more flexibility than you get on any
hardware RAID devices, and usually a lot faster.

You only bother with accelerators on Linux systems when you are dealing
with massive amounts of traffic on fast (and expensive) ports. TOE is
not really a consideration here - when RSS is not enough, you have more
advanced pre-filtering and queuing on the NICs. And these all involve
interrupt direction to specific cpus.

I can't imagine anyone using Windows for such systems.

>
>> But when you have 100 Gbps and faster NICs, ...
>
> 100GbE is rather a backbone technology.
>

A dual-port 100 Gb network card is perhaps $800. That is well within
the budget for a high-end server.

On 06/07/2023 17:52, Michael S wrote:
> On Thursday, July 6, 2023 at 6:01:06 PM UTC+3, Scott Lurndal wrote:
>> Michael S <already...@yahoo.com> writes:
>>> On Thursday, July 6, 2023 at 5:32:16=E2=80=AFPM UTC+3, Scott Lurndal wrote:
>>>> Bonita Montero <Bonita....@gmail.com> writes:=20
>>>>> Am 05.07.2023 um 15:49 schrieb Scott Lurndal:=20
>>>>> =20
>>>>>> Your ignorance is showing. My company makes NICs. Very powerful=20
>>>>>> NICs. Which manage streams (the programmable TCAMs allow software=20
>>>>>> to select the packet fields that get CAMd when selecting the target=20
>>>>>> CPU for the interrupt).=20
>>>>> =20
>>>>> Show me the documentation.
>>>> This is an older generation:=20
>>>> =20
>>>> https://doc.dpdk.org/guides-20.11/platform/octeontx2.html=20
>>>> =20
>>>> And the current, which builds on the prior generations, going=20
>>>> all the way back to the original MIPS-based Octeons more than a decade ag=
>>> o.=20
>>>> =20
>>>> https://www.marvell.com/company/media-kit/marvell-octeon-tx2-and-octeon-f=
>>> usion-press-kit.html
>>>
>>>
>>> That's too high end for just about everybody, and I don't think that it rea=
>>> lly used in any
>>> Marvell-made NIC, not even in the ultra-high-end FastLinQ 45000 Series.
>> Marvell has a wide range of network interface chips. The OcteonTX
>> series have both low-end and high-end entries. The low-end (MIPS and ARM64)
>> Octeons are used in home routers, SOHO routers and some low-end enterprise
>> routers. The high-end OcteonTx are used in high-end enterprise routers,
>> cellular and backbone infrastructure.
>>
>
> Exactly. My point is that Octeon chips are not used in NICs.
>

There is steadily more work being done on network cards, in data centres
and wherever you need to handle a lot of traffic. You might not call
them "NICs", since they have a full OS on board, but they are cards that
you put in a server to connect to the network. I don't know the parts
Scott is talking about (I don't know much about /any/ of these, except
what I read about on the net and dream of having an excuse to use them).

<https://www.servethehome.com/what-is-a-dpu-a-data-processing-unit-quick-primer/>

>> However, the feature that started this conversation, recieve-side scaling
>> is widely used and supported by many different NIC vendors, including Intel's
>> PRO line of network interfaces.
>
> That's, I think, is obvious to everybody except Bonita.
> In the past RSS was supported even on non-pro or semi-pro Intel NICs like
> 82574L that I use to post this message. I didn't check if it's still supported
> in this range. Practically, it's no longer needed for 1Gbe since even cheap
> today's CPU will handle everything without breaking a sweet. But if they had
> in the past, may be they retain it just for sake of check box checked.

Michael S <already5chosen@yahoo.com> writes:
>On Thursday, July 6, 2023 at 6:01:06=E2=80=AFPM UTC+3, Scott Lurndal wrote:

>> >Marvell-made NIC, not even in the ultra-high-end FastLinQ 45000 Series.
>> Marvell has a wide range of network interface chips. The OcteonTX=20
>> series have both low-end and high-end entries. The low-end (MIPS and ARM6=
>4)=20
>> Octeons are used in home routers, SOHO routers and some low-end enterpris=
>e=20
>> routers. The high-end OcteonTx are used in high-end enterprise routers,=
>=20
>> cellular and backbone infrastructure.=20
>>=20
>
>Exactly. My point is that Octeon chips are not used in NICs.

Actually, Octeon chips _are_ used in NICs. See "LiquidIO", very
useful in large data centers.

https://www.marvell.com/products/infrastructure-processors/liquidio-smart-nics/liquidio-ii-smart-nics.html

David Brown <david.brown@hesbynett.no> writes:
>On 06/07/2023 17:52, Michael S wrote:
>> On Thursday, July 6, 2023 at 6:01:06 PM UTC+3, Scott Lurndal wrote:
>>> Michael S <already...@yahoo.com> writes:
>>>> Marvell-made NIC, not even in the ultra-high-end FastLinQ 45000 Series.
>>> Marvell has a wide range of network interface chips. The OcteonTX
>>> series have both low-end and high-end entries. The low-end (MIPS and ARM64)
>>> Octeons are used in home routers, SOHO routers and some low-end enterprise
>>> routers. The high-end OcteonTx are used in high-end enterprise routers,
>>> cellular and backbone infrastructure.
>>>
>>
>> Exactly. My point is that Octeon chips are not used in NICs.
>>
>
>There is steadily more work being done on network cards, in data centres
>and wherever you need to handle a lot of traffic. You might not call
>them "NICs", since they have a full OS on board, but they are cards that
>you put in a server to connect to the network. I don't know the parts
>Scott is talking about (I don't know much about /any/ of these, except
>what I read about on the net and dream of having an excuse to use them).
>
><https://www.servethehome.com/what-is-a-dpu-a-data-processing-unit-quick-primer/>

Here's the list:

https://www.marvell.com/products/data-processing-units.html

David Brown <david.brown@hesbynett.no> writes:
>On 06/07/2023 16:28, Bonita Montero wrote:
>> Am 06.07.2023 um 13:49 schrieb David Brown:
>
>>> But when you have 100 Gbps and faster NICs, ...
>>
>> 100GbE is rather a backbone technology.
>>
>
>A dual-port 100 Gb network card is perhaps $800. That is well within
>the budget for a high-end server.

Or one of those 96-core AMD systems. Run 96 guest operating systems
and guarantee each one 1Gb/sec. Bandwidth allocations controlled by the
hypervisor and data center management software dynamically. Perfect for
cloud operators.

On 06/07/2023 19:57, Scott Lurndal wrote:
> David Brown <david.brown@hesbynett.no> writes:
>> On 06/07/2023 16:28, Bonita Montero wrote:
>>> Am 06.07.2023 um 13:49 schrieb David Brown:
>>
>>>> But when you have 100 Gbps and faster NICs, ...
>>>
>>> 100GbE is rather a backbone technology.
>>>
>>
>> A dual-port 100 Gb network card is perhaps $800. That is well within
>> the budget for a high-end server.
>
> Or one of those 96-core AMD systems. Run 96 guest operating systems
> and guarantee each one 1Gb/sec. Bandwidth allocations controlled by the
> hypervisor and data center management software dynamically. Perfect for
> cloud operators.

Or virtual machine hosts with SAN storage for the disk images - you can
never have too much bandwidth for that kind of thing. 100 Gb network is
about 10 GB per second - two good NVM disks could saturate it.

Am 06.07.2023 um 18:37 schrieb David Brown:

> A dual-port 100 Gb network card is perhaps $800.
> That is well within the budget for a high-end server.

And which server application on a server of any size can saturate
a 100GbE-link ?

On 7/5/2023 3:04 PM, Frederick Virchanza Gotham wrote:
> On Wednesday, July 5, 2023 at 4:05:24 PM UTC+1, Bonita Montero wrote:
>> Am 05.07.2023 um 15:28 schrieb Frederick Virchanza Gotham:
>>
>>> I suppose I could use normal sockets but I don't want the overhead of the operating system managing tens of thousands (yes, tens of thousands) of sockets and managing the 3-way handshakes for all those sockets. ...
>>
>> These information is hashed and the time for the lookup is O(1).
>
>
> Yeah but we're talking 40,000 file descriptors for one process.

For some reason this is reminding me of the 50,000 concurrent connection
scenario back on nt 4.0 wrt IOCP vs events.

Am 06.07.2023 um 09:25 schrieb Bonita Montero:

> Even with a billion file descriptors the access time would
> be the same, i.e. O(1).

In theory, but actually there's some cacching effect if the
hashtable fits into the cache:

#include <iostream>
#include <unordered_map>
#include <random>
#include <chrono>

using namespace std;
using namespace chrono;

atomic<size_t> aSum( 0 );

int main()
{ constexpr size_t
TO = 0x100000000,
ROUNDS = 10'000'000;
unordered_map<size_t, size_t> map;
mt19937_64 mt;
for( size_t n = 1, b = 0; n <= TO; n *= 2, b = n )
{
for( size_t i = b; i != n; ++i )
map.emplace( i, i );
uniform_int_distribution<size_t> uid( 0, n - 1 );
size_t sum = 0;
auto start = high_resolution_clock::now();
for( size_t r = ROUNDS; r--; )
sum += map[uid( mt )];
double ns = duration_cast<nanoseconds>( high_resolution_clock::now() -
start ).count() / (double)ROUNDS;
::aSum.fetch_add( sum );
cout << hex << n << ": " << ns << endl;
}
}

This are the results on a AMD 7950X:

1: 6.82776
2: 6.82223
4: 6.82411
8: 6.82552
10: 6.82525
20: 6.82547
40: 6.82349
80: 6.84252
100: 6.84796
200: 10.5177
400: 9.12652
800: 12.9653
1000: 15.0953
2000: 16.1944
4000: 17.4853
8000: 11.3426
10000: 13.4169
20000: 14.9261
40000: 18.6082
80000: 31.3227
100000: 63.0259
200000: 87.1134
400000: 103.188
800000: 141.211
1000000: 200.26
2000000: 308.339
4000000: 178.041
8000000: 225.446
10000000: 514.778
20000000: 223.162
40000000: 230.186
80000000: 834.93
100000000: 202.915

The steps to get to the value are always the same.
But the access time of the memory largely differs
according to the size of the hashtable.

On 06/07/2023 20:50, Bonita Montero wrote:
> Am 06.07.2023 um 18:37 schrieb David Brown:
>
>> A dual-port 100 Gb network card is perhaps $800.
>> That is well within the budget for a high-end server.
>
> And which server application on a server of any size can saturate
> a 100GbE-link ?
>

A file server with a couple of fast SSD's could do it. Remember, these
are 100 Gbps links, not 100 GBps.

If you have a 32 core server, that's an average 3 Gbps per core.

Usually, however, saturation of the link is not the issue - just like
cpu clock speeds, it is often the peaks that matter. You don't
(typically) buy a 100 Gb link because you want to send 36 TB over the
next hour, you buy it so that you can send 1 GB in a tenth of a second.

AMD's latest server chips have 128 cores, and Ampere One's have 192
cores - per socket. Do you think people with a 192 core cpu are going
to be happy with a 10 Gb link? And do you think people would be making
and selling such processors, and servers containing them, if they were
not useful?

Note that I said /high-end/ server. Most servers won't need anything
like that for their application sides. But if you have a cluster, and
storage external to the server, you'll easily find such speeds to be
useful. And plenty of systems in data centres, HPC systems, cloud
hosting, etc., will have links like that - or faster.

Am 07.07.2023 um 10:03 schrieb David Brown:

> A file server with a couple of fast SSD's could do it. ...

No one uses a fileserver with 12,5GB/s.

> If you have a 32 core server, that's an average 3 Gbps per core.

LOL. You have been bathed to hot by your mother.

Am 07.07.2023 um 10:03 schrieb David Brown:

> A file server with a couple of fast SSD's could do it.  Remember, these
> are 100 Gbps links, not 100 GBps.

And one thing which just in my mind: file server's don't have any
additional load beyond I/O, so separating the flows wouldn't hurt.

Bonita Montero <Bonita.Montero@gmail.com> writes:
>Am 07.07.2023 um 10:03 schrieb David Brown:
>
>> A file server with a couple of fast SSD's could do it. ...
>
>No one uses a fileserver with 12,5GB/s.

Your computing experiences seem very limited.

Our lab fileservers (using 25gb, 40gb and 100gb network adapters)
serve hundreds of high-end multicore servers performing RTL simulations
24x7 using NFS.

<Die kindische Beleidigung verschwand>

Am 07.07.2023 um 16:05 schrieb Scott Lurndal:

> Our lab fileservers (using 25gb, 40gb and 100gb network adapters)
> serve hundreds of high-end multicore servers performing RTL simulations
> 24x7 using NFS.

I don't believe you at all, that's pure phantasy.
But fileservers don't have high CPU-load anyway.
So manual segementation wouldn't hurt, even more
because in a LAN-segment you have jumbo frames.

Bonita Montero <Bonita.Montero@gmail.com> wrote:
> Am 07.07.2023 um 16:05 schrieb Scott Lurndal:
>
>> Our lab fileservers (using 25gb, 40gb and 100gb network adapters)
>> serve hundreds of high-end multicore servers performing RTL simulations
>> 24x7 using NFS.
>
> I don't believe you at all, that's pure phantasy.

Do you happen to have any good reasons why he would lie
about their lab?

br,
KK

Am 07.07.2023 um 17:53 schrieb Kalevi Kolttonen:

> Do you happen to have any good reasons why he would lie
> about their lab?

100GbE is a backbone-technology, maybe for linking switches
with further lower speed links. For which application would
you need a fileserver which can supply 12,5GB/s ?

On 07/07/2023 17:30, Bonita Montero wrote:
> Am 07.07.2023 um 16:05 schrieb Scott Lurndal:
>
>> Our lab fileservers (using 25gb, 40gb and 100gb network adapters)
>> serve hundreds of high-end multicore servers performing RTL simulations
>> 24x7 using NFS.
>
> I don't believe you at all, that's pure phantasy.

Do you think Scott is deliberately lying here? That's quite the accusation.

It would appear you know practically nothing about servers or
networking, especially for more demanding uses. That's fine, of course
- no one knows about everything, and most people have little interest in
such things unless they actually need to know about them. However, it
is absurd to suggest that just because /you/ can't imagine what how such
systems might be used, they don't exist. And it is arrogant and
obnoxious to accuse those who /do/ know, and /do/ use such systems, of
lying about it.

> But fileservers don't have high CPU-load anyway.
> So manual segementation wouldn't hurt, even more
> because in a LAN-segment you have jumbo frames.
>
>

Bonita Montero <Bonita.Montero@gmail.com> wrote:
> Am 07.07.2023 um 17:53 schrieb Kalevi Kolttonen:
>
>> Do you happen to have any good reasons why he would lie
>> about their lab?
>
> 100GbE is a backbone-technology, maybe for linking switches
> with further lower speed links. For which application would
> you need a fileserver which can supply 12,5GB/s ?

He did describe his lab setup by saying that there are
"hundreds of high-end multicore servers performing RTL
simulations 24x7 using NFS".

I have no idea what RTL is, it could perhaps be something
to do with CPUs, I don't know. But it is obvious that what
they are doing is no joke.

br,
KK