Last batch of comparisons of OpenVMS to Ubuntu Server 22.04 LTS running on identically configured VirtualBox 7.0.10 VMs (6 CPUs, 16GB RAM) on an Ubuntu Linux host, with an Intel Xeon W-1290P CPU @ 3.70GHz.

I was hoping to be able to compare ZeroMQ speeds, since VSI has a package for it, and the release notes mention the benchmarks. The local benchmarks are, well, embarrassingly slow.

inproc_lat (128 B): Linux: 9.909 us, VMS: 482995.15 us (48743x)
inproc_lat (512 B): Linux: 9.931 us, VMS: 482995.15 us (48635x)
inproc_lat (1024 B): Linux: 9.966 us, VMS: 482995.20 us (48464x)
inproc_lat (4096 B): Linux: 9.966 us, VMS: 482995.15 us (48464x)

inproc_thr (128 B): Linux: 3353294.667 msg/s, VMS: 2325 msg/s (1442x)
inproc_thr (128 B): Linux: 3433.774 Mb/s, VMS: 2.381 Mb/s (1442x)
inproc_thr (512 B): Linux: 2996375.333 msg/s, VMS: 2325 msg/s (1289x)
inproc_thr (512 B): Linux: 12273.153 Mb/s, VMS: 9.523 Mb/s (1289x)
inproc_thr (1024 B): Linux: 2923643.333 msg/s, VMS: 2325 msg/s (1257x)
inproc_thr (1024 B): Linux: 23950.486 Mb/s, VMS: 19.046 Mb/s (1257x)
inproc_thr (4096 B): Linux: 2540187.667 msg/s, VMS: 2325 msg/s (1093x)
inproc_thr (4096 B): Linux: 83236.869 Mb/s, VMS: 76.186 Mb/s (1093x)

I'd like to report the results for ZeroMQ over TCP, but the local_ and remote_ test apps both crash inside the ZeroMQ library trying to connect or bind, respectively.

The more interesting benchmark suite for IPC is one that I forked to port to VMS to test on Itanium a year or so ago:

https://github.com/jhamby/vms-ipc_benchmark/

The tests that worked on VMS are pipe, socketpair, TCP, and UDP. Here are the results with different write sizes. Note that DECC$STREAM_PIPE must be enabled or the pipe() benchmark is 2x-3x slower than these results (closer to the VMS UDP results, actually).

pipe (128 B): Linux: 305 MB/s, VMS: 17 MB/s (17.9x)
pipe (128 B): Linux: 2500748 msg/s, VMS: 139019 msg/s (18.0x)
pipe (256 B): Linux: 574 MB/s, VMS: 34 MB/s (16.9x)
pipe (256 B): Linux: 2350066 msg/s, VMS: 138256 msg/s (17.0x)
pipe (512 B): Linux: 1164 MB/s, VMS: 67 MB/s (17.4x)
pipe (512 B): Linux: 2383168 msg/s, VMS: 136845 msg/s (17.4x)
pipe (1024 B): Linux: 2003 MB/s, VMS: 134 MB/s (14.9x)
pipe (1024 B): Linux: 2051325 msg/s, VMS: 137307 msg/s (14.9x)
pipe (2048 B): Linux: 3609 MB/s, VMS: 263 MB/s (13.7x)
pipe (2048 B): Linux: 1847980 msg/s, VMS: 134540 msg/s (13.7x)
pipe (4096 B): Linux: 5890 MB/s, VMS: 339 MB/s (17.4x)
pipe (4096 B): Linux: 1507907 msg/s, VMS: 86663 msg/s (17.4x)

socketpair (128 B): Linux: 161 MB/s, VMS: 11 MB/s (14.6x)
socketpair (128 B): Linux: 1317178 msg/s, VMS: 88159 msg/s (14.9x)
socketpair (256 B): Linux: 317 MB/s, VMS: 20 MB/s (15.9x)
socketpair (256 B): Linux: 1299375 msg/s, VMS: 79961 msg/s (16.2x)
socketpair (512 B): Linux: 627 MB/s, VMS: 34 MB/s (18.4x)
socketpair (512 B): Linux: 1283596 msg/s, VMS: 69000 msg/s (18.6x)
socketpair (1024 B): Linux: 1186 MB/s, VMS: 51 MB/s (23.3x)
socketpair (1024 B): Linux: 1214167 msg/s, VMS: 52067 msg/s (23.3x)
socketpair (2048 B): Linux: 2147 MB/s, VMS: 96 MB/s (22.4x)
socketpair (2048 B): Linux: 1099505 msg/s, VMS: 49435 msg/s (22.2x)
socketpair (4096 B): Linux: 3755 MB/s, VMS: 141 MB/s (26.6x)
socketpair (4096 B): Linux: 961280 msg/s, VMS: 36015 msg/s (26.7x)

tcp (128 B): Linux: 255 MB/s, VMS: 11 MB/s (23.1x)
tcp (128 B): Linux: 2091401 msg/s, VMS: 88029 msg/s (23.8x)
tcp (256 B): Linux: 502 MB/s, VMS: 20 MB/s (25.1x)
tcp (256 B): Linux: 2055034 msg/s, VMS: 80885 msg/s (25.4x)
tcp (512 B): Linux: 966 MB/s, VMS: 34 MB/s (28.4x)
tcp (512 B): Linux: 1977886 msg/s, VMS: 70080 msg/s (28.2x)
tcp (1024 B): Linux: 1750 MB/s, VMS: 50 MB/s (35.0x)
tcp (1024 B): Linux: 1791894 msg/s, VMS: 50536 msg/s (35.5x)
tcp (2048 B): Linux: 2944 MB/s, VMS: 92 MB/s (32.0x)
tcp (2048 B): Linux: 1507203 msg/s, VMS: 47104 msg/s (32.0x)
tcp (4096 B): Linux: 4635 MB/s, VMS: 173 MB/s (26.8x)
tcp (4096 B): Linux: 1186551 msg/s, VMS: 44168 msg/s (26.9x)

udp (128 B): Linux: 88 MB/s, VMS: 5 MB/s (17.6x)
udp (128 B): Linux: 722460 msg/s, VMS: 40199 msg/s (18.0x)
udp (256 B): Linux: 173 MB/s, VMS: 9 MB/s (19.2x)
udp (256 B): Linux: 708782 msg/s, VMS: 38315 msg/s (18.5x)
udp (512 B): Linux: 357 MB/s, VMS: 19 MB/s (18.8x)
udp (512 B): Linux: 731181 msg/s, VMS: 38281 msg/s (19.1x)
udp (1024 B): Linux: 675 MB/s, VMS: 37 MB/s (18.2x)
udp (1024 B): Linux: 690771 msg/s, VMS: 38146 msg/s (18.1x)
udp (2048 B): Linux: 1267 MB/s, VMS: 72 MB/s (17.6x)
udp (2048 B): Linux: 648895 msg/s, VMS: 37103 msg/s (17.5x)
udp (4096 B): Linux: 2225 MB/s, VMS: 126 MB/s (17.7x)
udp (4096 B): Linux: 569686 msg/s, VMS: 32201 msg/s (17.7x)

There's a lot of room for improvement, to say the least.

Cheers,
Jake Hamby

On Sunday, October 15, 2023 at 10:43:56 PM UTC-7, Jake Hamby wrote:
> Last batch of comparisons of OpenVMS to Ubuntu Server 22.04 LTS running on identically configured VirtualBox 7.0.10 VMs (6 CPUs, 16GB RAM) on an Ubuntu Linux host, with an Intel Xeon W-1290P CPU @ 3.70GHz.

Oops, I meant of course "Part 3". I'm still pondering what might be causing a 17-18x slowdown for VMS relative to Linux in the same VM for pipe() and UDP sockets, with up to 35x slowdown for TCP sockets in the worst case (1K buffer sizes). It's also curious that TCP sockets are faster than socketpair() across the board on Linux, but the same speed on VMS, except for 4K buffers, where TCP pulls ahead for some reason.

This is within the same system, so there's no network or any other device overhead. My first guess as to what might be going on is an excessive number of data copies. My second guess would be an excessive number of context switches, or maybe not enough context switches? My third guess would be low-level IPC code that hasn't been optimized yet for x86, and my fourth guess would be VMS may be doing a lot of cache flushing.

Perhaps I'll add a new IPC benchmark that uses a VMS mailbox and $QIO calls.. I'm curious how many can be batched together and what an optimal number of outstanding requests would be for processing a linear stream of bytes and/or fixed-size records.

Regards,
Jake Hamby

On Monday, October 16, 2023 at 2:28:48 PM UTC-4, Jake Hamby (Solid State Jake) wrote:
> On Sunday, October 15, 2023 at 10:43:56 PM UTC-7, Jake Hamby wrote:
> > Last batch of comparisons of OpenVMS to Ubuntu Server 22.04 LTS running on identically configured VirtualBox 7.0.10 VMs (6 CPUs, 16GB RAM) on an Ubuntu Linux host, with an Intel Xeon W-1290P CPU @ 3.70GHz.
> Oops, I meant of course "Part 3". I'm still pondering what might be causing a 17-18x slowdown for VMS relative to Linux in the same VM for pipe() and UDP sockets, with up to 35x slowdown for TCP sockets in the worst case (1K buffer sizes). It's also curious that TCP sockets are faster than socketpair() across the board on Linux, but the same speed on VMS, except for 4K buffers, where TCP pulls ahead for some reason.
>
> This is within the same system, so there's no network or any other device overhead. My first guess as to what might be going on is an excessive number of data copies. My second guess would be an excessive number of context switches, or maybe not enough context switches? My third guess would be low-level IPC code that hasn't been optimized yet for x86, and my fourth guess would be VMS may be doing a lot of cache flushing.
>
> Perhaps I'll add a new IPC benchmark that uses a VMS mailbox and $QIO calls. I'm curious how many can be batched together and what an optimal number of outstanding requests would be for processing a linear stream of bytes and/or fixed-size records.
>
> Regards,
> Jake Hamby
pipe() has historically been abysmally slow. That isn't a shock to me.

On Monday, October 16, 2023 at 7:28:48 PM UTC+1, Jake Hamby (Solid State Jake) wrote:
> On Sunday, October 15, 2023 at 10:43:56 PM UTC-7, Jake Hamby wrote:
> > Last batch of comparisons of OpenVMS to Ubuntu Server 22.04 LTS running on identically configured VirtualBox 7.0.10 VMs (6 CPUs, 16GB RAM) on an Ubuntu Linux host, with an Intel Xeon W-1290P CPU @ 3.70GHz.
> Oops, I meant of course "Part 3". I'm still pondering what might be causing a 17-18x slowdown for VMS relative to Linux in the same VM for pipe() and UDP sockets, with up to 35x slowdown for TCP sockets in the worst case (1K buffer sizes). It's also curious that TCP sockets are faster than socketpair() across the board on Linux, but the same speed on VMS, except for 4K buffers, where TCP pulls ahead for some reason.
>
> This is within the same system, so there's no network or any other device overhead. My first guess as to what might be going on is an excessive number of data copies. My second guess would be an excessive number of context switches, or maybe not enough context switches? My third guess would be low-level IPC code that hasn't been optimized yet for x86, and my fourth guess would be VMS may be doing a lot of cache flushing.
>
> Perhaps I'll add a new IPC benchmark that uses a VMS mailbox and $QIO calls. I'm curious how many can be batched together and what an optimal number of outstanding requests would be for processing a linear stream of bytes and/or fixed-size records.
>
> Regards,
> Jake Hamby

VMS mailboxes are slow.

You could try ICC$ system services.

On 10/17/2023 5:30 AM, Ian Miller wrote:
> On Monday, October 16, 2023 at 7:28:48 PM UTC+1, Jake Hamby (Solid
> State Jake) wrote:
>> Perhaps I'll add a new IPC benchmark that uses a VMS mailbox and
>> $QIO calls. I'm curious how many can be batched together and what
>> an optimal number of outstanding requests would be for processing a
>> linear stream of bytes and/or fixed-size records.
>
> VMS mailboxes are slow.
>
> You could try ICC$ system services.

That is also my thoughts.

Mailboxes are a solution for sending maybe 100 byte messages at a
rate of maybe 10 per minute.

Trying to stream megabytes through a mailbox will not perform well.

SYS$ICC_* system services may potentially perform well as they are
intended for that kind of traffic. I have never tested their
performance, so I don't know how well they actually perform.

https://www.vajhoej.dk/arne/articles/vmsipc.html#icc has some simple
examples of usage.

Arne

On Tuesday, October 17, 2023 at 4:12:20 PM UTC-7, Arne Vajhøj wrote:
> That is also my thoughts.
>
> Mailboxes are a solution for sending maybe 100 byte messages at a
> rate of maybe 10 per minute.
>
> Trying to stream megabytes through a mailbox will not perform well.
>
> SYS$ICC_* system services may potentially perform well as they are
> intended for that kind of traffic. I have never tested their
> performance, so I don't know how well they actually perform.
>
> https://www.vajhoej.dk/arne/articles/vmsipc.html#icc has some simple
> examples of usage.

The examples on your site are very useful, thanks! Since I had no luck getting the SYSV shared mem + SYSV semaphore IPC benchmark to work (it appears to deadlock with the read/write semaphores), I think I'll try adding a new IPC benchmark modifying the SYSV IPC code to use SYS$ICC_*. It's unfortunate that the feature requires SYSNAM, and also that the maximum message size is 1000 bytes.

POSIX pipe() is fast enough for my purposes (updating bash and the GNV apps). For a distributed server, I'm thinking that JVM-based frameworks would be the best way to optimize message passing within a machine, because you're working with inherently shared memory and kernel threads.

I've been curious about Erlang and the Elixir language, but they're just a bit too far off the beaten path for me. After discovering that the JVM on OpenVMS works best with the system and user WSMAX values set to as large as possible (4000000 pages was what I ended up using, since >2GB working sets apparently aren't allowed by AUTOGEN), I'm fairly impressed with the JVM for running Scala code and something like Akka Actors:

https://doc.akka.io/docs/akka/current/typed/guide/actors-intro.html

What I like even better is the Akka framework has its own clustering facility called Akka Cluster with all the features you'd expect from a failure-tolerant distributed service framework:

https://doc.akka.io/docs/akka/current/typed/cluster.html

There's a Web application framework called Play (terrible name for SEO!) that's built on Akka and I'm definitely interested in learning about:

https://www.playframework.com/

Regards,
Jake

On 10/17/2023 11:20 PM, Jake Hamby (Solid State Jake) wrote:
> I'm fairly impressed with the JVM for running Scala code and something like Akka Actors:
>
> https://doc.akka.io/docs/akka/current/typed/guide/actors-intro.html

Note that usage of Akka does not require programming in Scala - you
can use it from Java code.

Admitted: using it from Scala result in way more elegant code than
from Java.

Akka was pretty hot some years ago. Not so much today.

The switch from Apache license to BSL may not have helped.

> There's a Web application framework called Play (terrible name for SEO!) that's built on Akka and I'm definitely interested in learning about:
>
> https://www.playframework.com/

Play is one of the big users of Scala and Akka.

Play is using the RoR model like many other modern MVC
frameworks (including Grails that I mentioned previously).

Arne