I've become a little obsessed with the question of how well OpenVMS performs relative to Linux inside a VM, under different conditions. My current obsession is the libuv library which provides a popular async I/O abstraction layer implemented for all the different flavors of UNIX that have async I/O, as well as for Windows. What might a VMS version look like? How many cores could it scale up to without too much synchronization overhead?

Alternatively, for existing VMS apps, how might they be sped up on non-VAX hardware? Based on the mailbox copy driver loop in the VMS port of Perl that I spent some time piecing together, I've noticed a few patterns that can't possibly perform well on any hardware newer than Alpha, and maybe not on Alpha either.

My first concern is the use of VAX interlocked queue instructions, because they're system calls on Itanium and x86 (and PALcode on Alpha). They can't possibly run as fast as VMS programmers may be assuming based on what made sense on a VAX 30 years ago. The current hotness is to use lock-free data structures as much as possible.

My next thought was that it makes sense to build your I/O around shared memory and atomic ops as much as possible. I think Linux's io_uring presents a good model for how a user-level VMS I/O completion buffer might look. There's no ability to influence how you issue the async I/O requests, so there's no way to make a circular buffer for submissions, but you have a lot of flexibility in handling the completion events.

I also think it does make sense to have a shared pool of IOSBs and to avoid dynamic memory allocation as much as possible. malloc() vs. stack memory allocation is another area I'm curious about. We're not in the embedded era with 64KB stack sizes or anything like that, so if you know that an array will be a fixed 4KB or 8KB or 64KB max, then why not put it on the stack? That seems like it'd be helpful on anything post-VAX. The circular buffers that io_uring uses are also a good template.

In general, what's the fastest native way to queue and handle async commands? I think setting the event flag to EFN$C_ENF (don't care) and using the AST and AST parameter would be the quickest way to match up a completed I/O to some useful data. There are a lot of possibilities for how to encode things into that 64-bit AST parameter so that it can post the right completion code to the right completion queue.

The problem with ASTs, though, is they don't play well with pthreads, and you might want to use threads with libuv (Node.js is essentially single-threaded, using the default event queue only, but libuv does provide for threads and a few of its calls are threadsafe).

The safest option on VMS, and a popular one, is to use local event flags. I've modified the simplest libuv async benchmark to use pairs of LEF's to ping-pong back and forth and it looks like it's about 1/5 the speed of Linux in the same VM, or 10K/sec instead of 50K. The same test on the bare metal is around 250K per second, so the VM makes a huge difference.

As an aside, I'd avoid KVM/QEMU in favor of VirtualBox to run OpenVMS, for performance reasons. I ran some of the Java benchmarks I posted here recently in KVM and I didn't compute the exact numbers, but most of them seem around half the speed of VirtualBox. I suppose then that it's a good thing I started out with VirtualBox and then only recently copied my .vdi files into .qcow2 format to test in KVM. I have no idea how fast VMware ESXi is.

I think there must be a safe way to wake a POSIX thread that's sleeping on a condition variable without possibly sleeping on the associated mutex. In general, for an async event loop, there'll be only one thread waiting for completion, so I'm thinking it may be safe to use an atomic variable as a counter and then wake the sleeping thread only if the previous value was 0. I'll have to think about the details. The AST callback can't ever block, especially since it's going to be occurring on the kernel thread that issued the request (that's my understanding, anyway), which has an event loop that may or may not be sleeping waiting for a completion event.

The other detail constantly in my mind is that the DECthreads library doesn't have a 1:1 mapping between pthreads and kernel threads. In fact, it only can create kernel threads up to the number of active CPUs, and then it has to multiplex on top of those. That's why I'm thinking it may be best to look at how Linux and others are doing their I/O, so as to avoid falling into patterns that made sense on a single-core VAX in the late 1990s.

I'll add more detail here as I learn more about what works and what doesn't.. This rant and project were largely inspired by watching a talk Mark Russinovich gave in 2004 on Linux vs. Windows kernels, and back then Windows NT's I/O completion ports were seen as better than what Linux had to offer, because you can set up a thread pool and the OS wakes up one thread to handle each response. NT 3.5 introduced the feature in 1994, and Microsoft was granted a patent on it, which didn't expire until 2014.

Ironically, the pendulum today has now swung in the opposite direction, probably due to memory cache contention issues in SMP, to having a single async I/O thread and everything inside of its data structures being non-thread-safe, so Windows' IOCPs don't even seem relevant to me today. In theory, OpenVMS can support the modern paradigm quite well, because it was designed that way 45 years ago. In practice, I think good performance is achievable, especially in a few years when the x86 port is more optimized.

Regards,
Jake Hamby

On 3/11/2023 12:02 pm, Jake Hamby (Solid State Jake) wrote:
> I've become a little obsessed with the question of how well OpenVMS performs relative to Linux inside a VM, under different conditions. My current obsession is the libuv library which provides a popular async I/O abstraction layer implemented for all the different flavors of UNIX that have async I/O, as well as for Windows. What might a VMS version look like? How many cores could it scale up to without too much synchronization overhead?
>
> Alternatively, for existing VMS apps, how might they be sped up on non-VAX hardware? Based on the mailbox copy driver loop in the VMS port of Perl that I spent some time piecing together, I've noticed a few patterns that can't possibly perform well on any hardware newer than Alpha, and maybe not on Alpha either.
>
> My first concern is the use of VAX interlocked queue instructions, because they're system calls on Itanium and x86 (and PALcode on Alpha). They can't possibly run as fast as VMS programmers may be assuming based on what made sense on a VAX 30 years ago. The current hotness is to use lock-free data structures as much as possible.
>
> My next thought was that it makes sense to build your I/O around shared memory and atomic ops as much as possible. I think Linux's io_uring presents a good model for how a user-level VMS I/O completion buffer might look. There's no ability to influence how you issue the async I/O requests, so there's no way to make a circular buffer for submissions, but you have a lot of flexibility in handling the completion events.
>
> I also think it does make sense to have a shared pool of IOSBs and to avoid dynamic memory allocation as much as possible. malloc() vs. stack memory allocation is another area I'm curious about. We're not in the embedded era with 64KB stack sizes or anything like that, so if you know that an array will be a fixed 4KB or 8KB or 64KB max, then why not put it on the stack? That seems like it'd be helpful on anything post-VAX. The circular buffers that io_uring uses are also a good template.
>
> In general, what's the fastest native way to queue and handle async commands? I think setting the event flag to EFN$C_ENF (don't care) and using the AST and AST parameter would be the quickest way to match up a completed I/O to some useful data. There are a lot of possibilities for how to encode things into that 64-bit AST parameter so that it can post the right completion code to the right completion queue.
>
> The problem with ASTs, though, is they don't play well with pthreads, and you might want to use threads with libuv (Node.js is essentially single-threaded, using the default event queue only, but libuv does provide for threads and a few of its calls are threadsafe).
>
> The safest option on VMS, and a popular one, is to use local event flags. I've modified the simplest libuv async benchmark to use pairs of LEF's to ping-pong back and forth and it looks like it's about 1/5 the speed of Linux in the same VM, or 10K/sec instead of 50K. The same test on the bare metal is around 250K per second, so the VM makes a huge difference.
>
> As an aside, I'd avoid KVM/QEMU in favor of VirtualBox to run OpenVMS, for performance reasons. I ran some of the Java benchmarks I posted here recently in KVM and I didn't compute the exact numbers, but most of them seem around half the speed of VirtualBox. I suppose then that it's a good thing I started out with VirtualBox and then only recently copied my .vdi files into .qcow2 format to test in KVM. I have no idea how fast VMware ESXi is.
>
> I think there must be a safe way to wake a POSIX thread that's sleeping on a condition variable without possibly sleeping on the associated mutex. In general, for an async event loop, there'll be only one thread waiting for completion, so I'm thinking it may be safe to use an atomic variable as a counter and then wake the sleeping thread only if the previous value was 0. I'll have to think about the details. The AST callback can't ever block, especially since it's going to be occurring on the kernel thread that issued the request (that's my understanding, anyway), which has an event loop that may or may not be sleeping waiting for a completion event.
>
> The other detail constantly in my mind is that the DECthreads library doesn't have a 1:1 mapping between pthreads and kernel threads. In fact, it only can create kernel threads up to the number of active CPUs, and then it has to multiplex on top of those. That's why I'm thinking it may be best to look at how Linux and others are doing their I/O, so as to avoid falling into patterns that made sense on a single-core VAX in the late 1990s.
>
> I'll add more detail here as I learn more about what works and what doesn't. This rant and project were largely inspired by watching a talk Mark Russinovich gave in 2004 on Linux vs. Windows kernels, and back then Windows NT's I/O completion ports were seen as better than what Linux had to offer, because you can set up a thread pool and the OS wakes up one thread to handle each response. NT 3.5 introduced the feature in 1994, and Microsoft was granted a patent on it, which didn't expire until 2014.
>
> Ironically, the pendulum today has now swung in the opposite direction, probably due to memory cache contention issues in SMP, to having a single async I/O thread and everything inside of its data structures being non-thread-safe, so Windows' IOCPs don't even seem relevant to me today. In theory, OpenVMS can support the modern paradigm quite well, because it was designed that way 45 years ago. In practice, I think good performance is achievable, especially in a few years when the x86 port is more optimized.

Hi Jake.

Your benchmarks are interesting, relevant and incisive. Thanks.

There was a VSI Forum discussion a while back about the relative
profiles of Alpha and X86 CPU modes. Might be of some relevance.

https://forum.vmssoftware.com/viewtopic.php?f=21&t=8654

> Regards,
> Jake Hamby

--
Anyone, who using social-media, forms an opinion regarding anything
other than the relative cuteness or this or that puppy-dog, needs
seriously to examine their critical thinking.

On Friday, November 3, 2023 at 1:02:33 AM UTC, Jake Hamby (Solid State Jake) wrote:
> I've become a little obsessed with the question of how well OpenVMS performs relative to Linux inside a VM, under different conditions. My current obsession is the libuv library which provides a popular async I/O abstraction layer implemented for all the different flavors of UNIX that have async I/O, as well as for Windows. What might a VMS version look like? How many cores could it scale up to without too much synchronization overhead?
>
> Alternatively, for existing VMS apps, how might they be sped up on non-VAX hardware? Based on the mailbox copy driver loop in the VMS port of Perl that I spent some time piecing together, I've noticed a few patterns that can't possibly perform well on any hardware newer than Alpha, and maybe not on Alpha either.
>
> My first concern is the use of VAX interlocked queue instructions, because they're system calls on Itanium and x86 (and PALcode on Alpha). They can't possibly run as fast as VMS programmers may be assuming based on what made sense on a VAX 30 years ago. The current hotness is to use lock-free data structures as much as possible.
>
> My next thought was that it makes sense to build your I/O around shared memory and atomic ops as much as possible. I think Linux's io_uring presents a good model for how a user-level VMS I/O completion buffer might look. There's no ability to influence how you issue the async I/O requests, so there's no way to make a circular buffer for submissions, but you have a lot of flexibility in handling the completion events.
>
> I also think it does make sense to have a shared pool of IOSBs and to avoid dynamic memory allocation as much as possible. malloc() vs. stack memory allocation is another area I'm curious about. We're not in the embedded era with 64KB stack sizes or anything like that, so if you know that an array will be a fixed 4KB or 8KB or 64KB max, then why not put it on the stack? That seems like it'd be helpful on anything post-VAX. The circular buffers that io_uring uses are also a good template.
>
> In general, what's the fastest native way to queue and handle async commands? I think setting the event flag to EFN$C_ENF (don't care) and using the AST and AST parameter would be the quickest way to match up a completed I/O to some useful data. There are a lot of possibilities for how to encode things into that 64-bit AST parameter so that it can post the right completion code to the right completion queue.
>
> The problem with ASTs, though, is they don't play well with pthreads, and you might want to use threads with libuv (Node.js is essentially single-threaded, using the default event queue only, but libuv does provide for threads and a few of its calls are threadsafe).
>
> The safest option on VMS, and a popular one, is to use local event flags. I've modified the simplest libuv async benchmark to use pairs of LEF's to ping-pong back and forth and it looks like it's about 1/5 the speed of Linux in the same VM, or 10K/sec instead of 50K. The same test on the bare metal is around 250K per second, so the VM makes a huge difference.
>
> As an aside, I'd avoid KVM/QEMU in favor of VirtualBox to run OpenVMS, for performance reasons. I ran some of the Java benchmarks I posted here recently in KVM and I didn't compute the exact numbers, but most of them seem around half the speed of VirtualBox. I suppose then that it's a good thing I started out with VirtualBox and then only recently copied my .vdi files into .qcow2 format to test in KVM. I have no idea how fast VMware ESXi is.
>
> I think there must be a safe way to wake a POSIX thread that's sleeping on a condition variable without possibly sleeping on the associated mutex. In general, for an async event loop, there'll be only one thread waiting for completion, so I'm thinking it may be safe to use an atomic variable as a counter and then wake the sleeping thread only if the previous value was 0. I'll have to think about the details. The AST callback can't ever block, especially since it's going to be occurring on the kernel thread that issued the request (that's my understanding, anyway), which has an event loop that may or may not be sleeping waiting for a completion event.
>
> The other detail constantly in my mind is that the DECthreads library doesn't have a 1:1 mapping between pthreads and kernel threads. In fact, it only can create kernel threads up to the number of active CPUs, and then it has to multiplex on top of those. That's why I'm thinking it may be best to look at how Linux and others are doing their I/O, so as to avoid falling into patterns that made sense on a single-core VAX in the late 1990s.
>
> I'll add more detail here as I learn more about what works and what doesn't. This rant and project were largely inspired by watching a talk Mark Russinovich gave in 2004 on Linux vs. Windows kernels, and back then Windows NT's I/O completion ports were seen as better than what Linux had to offer, because you can set up a thread pool and the OS wakes up one thread to handle each response. NT 3.5 introduced the feature in 1994, and Microsoft was granted a patent on it, which didn't expire until 2014.
>
> Ironically, the pendulum today has now swung in the opposite direction, probably due to memory cache contention issues in SMP, to having a single async I/O thread and everything inside of its data structures being non-thread-safe, so Windows' IOCPs don't even seem relevant to me today. In theory, OpenVMS can support the modern paradigm quite well, because it was designed that way 45 years ago. In practice, I think good performance is achievable, especially in a few years when the x86 port is more optimized.
>
> Regards,
> Jake Hamby

Have you looked at the Fast I/O routines? https://docs.vmssoftware.com/vsi-openvms-io-user-s-reference-manual/#ch10.html

On 11/2/2023 9:02 PM, Jake Hamby (Solid State Jake) wrote:
> I've become a little obsessed with the question of how well OpenVMS
> performs relative to Linux inside a VM, under different conditions.
> My current obsession is the libuv library which provides a popular
> async I/O abstraction layer implemented for all the different flavors
> of UNIX that have async I/O, as well as for Windows. What might a VMS
> version look like? How many cores could it scale up to without too
> much synchronization overhead?
>
> Alternatively, for existing VMS apps, how might they be sped up on
> non-VAX hardware? Based on the mailbox copy driver loop in the VMS
> port of Perl that I spent some time piecing together, I've noticed a
> few patterns that can't possibly perform well on any hardware newer
> than Alpha, and maybe not on Alpha either.

The normal assumption regarding speed of disk IO would be that:

RMS record IO ($GET and $PUT) < RMS block IO ($READ and $WRITE) <
$QIO(W) < $IO_PERFORM(W) < memory mapped file

(note that assumption and fact are spelled differently)

All of it can relative easily be done async as VMS has done async IO
since forever.

> My first concern is the use of VAX interlocked queue instructions,
> because they're system calls on Itanium and x86 (and PALcode on
> Alpha). They can't possibly run as fast as VMS programmers may be
> assuming based on what made sense on a VAX 30 years ago. The current
> hotness is to use lock-free data structures as much as possible.

These constructs are certainly going to be relative
more costly on newer system than on VAX.

But I doubt that it matters much for overall performance.

> I also think it does make sense to have a shared pool of IOSBs and to
> avoid dynamic memory allocation as much as possible. malloc() vs.
> stack memory allocation is another area I'm curious about. We're not
> in the embedded era with 64KB stack sizes or anything like that, so
> if you know that an array will be a fixed 4KB or 8KB or 64KB max,
> then why not put it on the stack? That seems like it'd be helpful on
> anything post-VAX. The circular buffers that io_uring uses are also a
> good template.

I have never seen VMS code dynamic allocate an IOSB in heap - either
stack or static heap.

But again then I don't think location of IOSB or reuse of IOSB's will
matter much for overall performance.

Arne

On Friday, November 3, 2023 at 10:08:11 AM UTC-4, Arne Vajhøj wrote:
> I have never seen VMS code dynamic allocate an IOSB in heap - either
> stack or static heap.
>
> Arne

For network code, it was pretty common for me to allocate from the heap
composite structures that included both a buffer and IOSB since the number
of pending I/O requests depended upon the number of concurrent client
connections.

On 11/3/23 11:08 AM, David Jones wrote:
> On Friday, November 3, 2023 at 10:08:11 AM UTC-4, Arne Vajhøj wrote:
>> I have never seen VMS code dynamic allocate an IOSB in heap - either
>> stack or static heap.

The alternative being what? A finite pool of global buffers?

> For network code, it was pretty common for me to allocate from the heap
> composite structures that included both a buffer and IOSB since the number
> of pending I/O requests depended upon the number of concurrent client
> connections.

And moreover, an IOSB on the stack is kind of a non-starter for
asynchronous code since it will very likely have gone out of scope
before the completion routine needs to check its contents. It's
completely normal and convenient, though, (and quite common) if you're
calling the versions of system services that end it "w."

In article <ff7f2845-84cc-4c59-a007-1b388c82543fn@googlegroups.com>,
Jake Hamby (Solid State Jake) <jake.hamby@gmail.com> wrote:
>I've become a little obsessed with the question of how well
>OpenVMS performs relative to Linux inside a VM, under different
>conditions. My current obsession is the libuv library which
>provides a popular async I/O abstraction layer implemented for
>all the different flavors of UNIX that have async I/O, as well
>as for Windows. What might a VMS version look like? How many
>cores could it scale up to without too much synchronization
>overhead?
>
>Alternatively, for existing VMS apps, how might they be sped up
>on non-VAX hardware? Based on the mailbox copy driver loop in
>the VMS port of Perl that I spent some time piecing together,
>I've noticed a few patterns that can't possibly perform well on
>any hardware newer than Alpha, and maybe not on Alpha either.
>
>My first concern is the use of VAX interlocked queue
>instructions, because they're system calls on Itanium and x86
>(and PALcode on Alpha). They can't possibly run as fast as VMS
>programmers may be assuming based on what made sense on a VAX
>30 years ago. The current hotness is to use lock-free data
>structures as much as possible.

I don't know that that's the current hotness so much as trying
to structure problems so that the vast majority of access are
local to a core, so that either locks are unnecessary, or
taking a lock is just an uncontended write to an owned cache
line.

Consider the problem of a per-core queue with work-stealing;
the specifics of what's in the queue don't matter so much as the
overall structure of the problem: items in the queue might
represent IO requests, or they may represent runnable threads,
or whatever. Anyway, a core will usually process things on
its own local queue, but if it runs out of things to do, it
may choose some victim and steal some work from it.
Conventionally, this will involve everyone taking locks on
these queues, but that's ok: we expect that work stealing is
pretty rare, and that usually a core is simply locking its own
queue, which will be an uncontended write on a cacheline it
already owns. In any event, _most of the time_ the overhead
of using a "lock" will be minimal (assuming relatively simply
spinlocks or MCS locks or something like that).

>My next thought was that it makes sense to build your I/O
>around shared memory and atomic ops as much as possible. I
>think Linux's io_uring presents a good model for how a
>user-level VMS I/O completion buffer might look. There's no
>ability to influence how you issue the async I/O requests, so
>there's no way to make a circular buffer for submissions, but
>you have a lot of flexibility in handling the completion
>events.

I disagree with the first sentence. Contended atomic
operations become a scalability bottleneck very quickly; this
is generally why spinlocks behave so poorly ones you have more
than a handful of cores racing against one. Better would be to
structure the problem to minimize interaction between components
at all, and there are a number of techniques one can take
advantage of here. For example, one might have a thread (or
async task or whatever) per IO device, set up a number of IOs in
some worker thread, and then transfer the descriptors for those
IOs to the the per-device thread _en masse_; by doing so, one
amortizes the expensive step of synchronizing between components
across many operations.

A thing to bear in mind with async systems is that they run the
risk of unbounded growth if there isn't some way to inject
backpressure into the system. Without hysteresis, you hit an
inflection point where the amount of work queued balloons to the
point the system becomes unresponsive.

>I also think it does make sense to have a shared pool of IOSBs
>and to avoid dynamic memory allocation as much as possible.
>malloc() vs. stack memory allocation is another area I'm
>curious about. We're not in the embedded era with 64KB stack
>sizes or anything like that, so if you know that an array will
>be a fixed 4KB or 8KB or 64KB max, then why not put it on the
>stack? That seems like it'd be helpful on anything post-VAX.
>The circular buffers that io_uring uses are also a good template.

I can think of several (potential) issues with this direction.
First, stack allocations are inherently transient; if you share
a pointer to something on your stack, then that pointer is
invalidated if you return from whatever function call owns the
frame that data is allocated on. Which isn't to say that you
shouldn't share things allocated on your stack, just that one
must exercise caution to when doing so to ensure that you don't
return while there's still a live pointer to the data on the
relevant stack frame.

Second, just because it doesn't involve an explicit `malloc`
doesn't mean that stack allocation is free, particularly for
large objects: it may be that you encur the overhead of one
(or perhaps more than one) fault as the OS grows the stack on
your behalf.

I think the more general advice is, "minimize allocations in
the hot path." There's nothing magical about heap memory versus
stack memory; they're both just memory. If you preallocate
space in the heap, accessing it will be pretty fast just like
accessing space on an allocated stack frame (yeah yeah yeah,
we can talk about addressing modes and frame/sp-relative
accesses and so forth here, but that's really getting into the
weeds). But if, say, you heap allocated a per-CPU "arena" for
temporary space when your program started up and treated it
kind of like a stack, I don't think you'd see any significant
performance variation versus just making stack allocations. It
may even be slightly more efficient, particularly if you can
arrange for the OS to preallocate your memory and you wire it
into your address space (e.g., `mlock` or equivalent).

>In general, what's the fastest native way to queue and handle
>async commands? I think setting the event flag to EFN$C_ENF
>(don't care) and using the AST and AST parameter would be the
>quickest way to match up a completed I/O to some useful data.
>There are a lot of possibilities for how to encode things into
>that 64-bit AST parameter so that it can post the right
>completion code to the right completion queue.
>
>The problem with ASTs, though, is they don't play well with
>pthreads, and you might want to use threads with libuv (Node.js
>is essentially single-threaded, using the default event queue
>only, but libuv does provide for threads and a few of its calls
>are threadsafe).

Generally, event-driven callbacks will be faster than
synchronous threading (at a minimum, you avoid register save and
restore and stack switching, and you may have better cache
locality, too). However, there's no reason you can't multiplex
such callbacks onto multiple operating system-provided threads,
to get multicore parallelism....Indeed, you almost have to since
the practically no OS gives you any other primitive.

In OS design, there is always a tension between interrupts and
polling in terms of which has more desirable characteristics
under different criteria; generally speaking, you'll have lower
latency with polling but at the expense of burning computational
resources (since you burn a core spinning on the polling
operation). So if you have a core to spare, _a_ way to get
faster performance in an async system is to dedicate it to
polling for IO completions and marking tasks runnable (e.g.,
by handing them out to per-core executors or something). Task-
handling cores would just spin on their executor queues or
something like that.

>The safest option on VMS, and a popular one, is to use local
>event flags. I've modified the simplest libuv async benchmark
>to use pairs of LEF's to ping-pong back and forth and it looks
>like it's about 1/5 the speed of Linux in the same VM, or
>10K/sec instead of 50K. The same test on the bare metal is
>around 250K per second, so the VM makes a huge difference.

Is the code available somewhere to poke at?

>As an aside, I'd avoid KVM/QEMU in favor of VirtualBox to run
>OpenVMS, for performance reasons. I ran some of the Java
>benchmarks I posted here recently in KVM and I didn't compute
>the exact numbers, but most of them seem around half the speed
>of VirtualBox. I suppose then that it's a good thing I started
>out with VirtualBox and then only recently copied my .vdi files
>into .qcow2 format to test in KVM. I have no idea how fast
>VMware ESXi is.

I actually find this rather surprising; KVM at least is very
good and is the hypervisor of choice for the hyperscalers (well,
except MSFT, which uses Hyper-V). ESXi is pretty zippy.

>I think there must be a safe way to wake a POSIX thread that's
>sleeping on a condition variable without possibly sleeping on
>the associated mutex. In general, for an async event loop,
>there'll be only one thread waiting for completion, so I'm
>thinking it may be safe to use an atomic variable as a counter
>and then wake the sleeping thread only if the previous value
>was 0. I'll have to think about the details.
>
>The AST callback
>can't ever block, especially since it's going to be occurring
>on the kernel thread that issued the request (that's my
>understanding, anyway), which has an event loop that may or may
>not be sleeping waiting for a completion event.

On Friday, November 3, 2023 at 3:17:06 AM UTC-7, Ian Miller wrote:
> Have you looked at the Fast I/O routines? https://docs.vmssoftware.com/vsi-openvms-io-user-s-reference-manual/#ch10.html

No, I hadn't seen the Fast I/O routines and they look very relevant, thanks!

I'm going to have to take a little time to read everyone's replies, which I greatly appreciate. I've been hacking away on libuv to build it with Clang as C++ (with exceptions and RTTI disabled), in order to be able to use C++11's <atomic> header file, which is present, in place of <stdatomic.h> and a C11 compiler, which isn't. Then I had to change "_Atomic int" to "atomic_int" everywhere. It's all fairly ugly, but I'll complain about specifics another time.

The good news is that I do now have a naive port of libuv that should be able to run with a little more polishing. The last hurdle to get it running was remembering to add "/Threads" to the link line since it apparently didn't notice it was compiling a multithreaded program. The previous hurdle was changing a select() to a poll() call in the benchmark runner, because I was getting an error about select() being called on a non-socket. libuv itself was using poll(), so that one confused me for a while. SEARCH is my friend (just remember to reverse the argument order from "grep").

The hilariously bad news is that the UNIX way of doing things, in the case of one thread waking up another, is to send a single "\0" byte down a pipe to wake up the other thread. The first benchmark I started looking at, alphabetically, and also in terms of involving the least effort, is "async1" to "async8" (1, 2, 4, and 8 worker threads, respectively). Once I got "async1" to run by fixing the link line in my descrip.mms, it gave me just barely over 50 round-trip pings/sec. Not 50K. 50.

I think some of you may have guessed the same thing as I have about what's likely going on. The user-mode threads scheduler isn't thinking to look to wake up another thread in the same process to get a message from it. The main and worker thread(s) are both taking turns waiting on poll() and reading one byte from each other via a pipe. The pipe isn't the issue. The CPU is practically idle. Both threads are giving up the process quantum and sleeping until the process comes back around and notices one of the threads can wake up. At least I suspect it's something like that.

I'm glad I started out thinking about using local event flags for the same purpose, because it looks like that's actually the only way to have fast wakeup of any particular thread within a process. I'll take 10000 pings/s over 50 pings/s any day. The benchmark isn't representative of real work anyway.

It'd definitely be doable to implement a proper VMS port of libuv, and thereby enable a reasonable-speed Node.js plus any other programs using that library as an abstraction layer. The trick would be to leverage the VMS APIs, use channel numbers instead of fds, use async I/O with either a LEF or an AST callback (but not both, presumably), and use VMS APIs for file I/O, mailboxes/pipes, terminals, and sockets (including for DNS resolution).

Regards,
Jake Hamby

Something else just occurred to me. While file I/O performance is important, as well as local IPC, I've also been keeping in mind something else: simultaneous socket connections. One benefit of everyone going to RESTful APIs is you're not keeping a long-lived TCP socket open like you would with other types of protocols. I believe both Google and Apple's push notifications for alerts work with an always-open socket. You don't have to send keepalive packets very often, if at all. But that work can be handled by front-ends..

There have been a lot of posts on my LinkedIn feed from IBM mainframers and they are more excited than ever about IBM's latest hardware which is very fast at running COBOL, Java, Go, C++, Db2, CICS, IMS, TSO/E, ISPF, Python. Technologies from every decade.

Mainframes are great for all the applications that are difficult to shard. Giant databases of bank transactions. I think software engineers want every problem to be something you can shard and parallelize to run in an AWS or Google cluster, but you can't do that with billions of lines of COBOL. But the reason everyone's still using that COBOL is that there's no real gain from rewriting it because the problems can't be solved in some new way. So having extremely powerful cores, terabytes of RAM for caching, high-speed dedicated disk and network I/O, and all the other features are helpful.

So it may be the case that the network front-end can fan out to hundreds or thousands of users and then request from backend services through a vastly smaller number of sockets, and by having two or three layers, including caching, you don't have the "C10K" problem. But the C10K problem has been on my mind ever since I saw the acronym this week: https://en.wikipedia.org/wiki/C10k_problem

There's a ~65535 limit on TCP sockets due to limited port numbers (including with IPv6), but you can have many different IP addresses, especially with IPv6. It looks like Linux servers can scale up to a few hundred thousand simultaneous connections. The "net.core.somaxconn" sysctl on my Ubuntu server is set to 4096, for comparison with the hardcoded SOMAXCONN value of 1024 in VMS's socket.h (but only if !defined _XOPEN_SOURCE_EXTENDED, which looks like a typo bug in the header file since it has an "#ifdef _XOPEN_SOURCE_EXTENDED" block embedded in the middle of the "if !defined" block.

My big gripe about VMS header files is you can't define _XOPEN_SOURCE_EXTENDED without breaking inclusion of basic standard C headers, because it causes _ANSI_C_SOURCE to be defined and then they stop including the prototypes that other headers need for inline functions. But you must define _XOPEN_SOURCE_EXTENDED to get the newer definitions of "struct iovec" for using readv/writev, so you have to define _XOPEN_SOURCE_EXTENDED after a few other VMS headers, but before you include either socket.h or uio.h.

In theory, not defining any feature flags is supposed to give you all of the VMS extensions, right? The _XOPEN_* feature flags are supposed to limit the environment, not necessarily to extend it. So I'd prefer to get the newer versions of everything by default. But not having a single set of flag definitions that works to include all the VMS header files and get all the newer features is a bit absurd. I already have to #define (_USE_STD_STAT, _POSIX_EXIT, _SOCKADDR_LEN), for goodness sake!

On 2023-11-04 01:35, Jake Hamby (Solid State Jake) wrote:
> There's a ~65535 limit on TCP sockets due to limited port numbers (including with IPv6), but you can have many different IP addresses, especially with IPv6. It looks like Linux servers can scale up to a few hundred thousand simultaneous connections. The "net.core.somaxconn" sysctl on my Ubuntu server is set to 4096, for comparison with the hardcoded SOMAXCONN value of 1024 in VMS's socket.h (but only if !defined _XOPEN_SOURCE_EXTENDED, which looks like a typo bug in the header file since it has an "#ifdef _XOPEN_SOURCE_EXTENDED" block embedded in the middle of the "if !defined" block.

Nitpick: There is actually no such limit. There is for UDP, but TCP
sockets are identified by the combination of local and remote addresses
and ports. So you can have an infinite number of socket, actually.
But you cannot listen on more than 64K sockets.

Basically, the same local port number can exist for any number of TCP
sockets. The most classical example: Think of your HTTP server. It
accepts connections on port 80. Any number of connections that come in
all use port 80. So there is only ever one local port number used. But
you can have tens of thousands of connections, potentially.

As for your testing - it's all interesting and fun. But I do worry a bit
that you are using solutions and paradigms designed on the available
tools and capabilities of Linux systems, and trying to make them perform
on VMS, while VMS might actually be able to do much better using other
solutions and paradigms.
But I noticied you at least are playing around with event flags and
ASTs, so you might be doing better than that.

Johnny

On 2023-11-03 15:08, Arne Vajhøj wrote:
> On 11/2/2023 9:02 PM, Jake Hamby (Solid State Jake) wrote:
>> I've become a little obsessed with the question of how well OpenVMS
>> performs relative to Linux inside a VM, under different conditions.
>> My current obsession is the libuv library which provides a popular
>> async I/O abstraction layer implemented for all the different flavors
>> of UNIX that have async I/O, as well as for Windows. What might a VMS
>> version look like? How many cores could it scale up to without too
>> much synchronization overhead?
>>
>> Alternatively, for existing VMS apps, how might they be sped up on
>> non-VAX hardware? Based on the mailbox copy driver loop in the VMS
>> port of Perl that I spent some time piecing together, I've noticed a
>> few patterns that can't possibly perform well on any hardware newer
>> than Alpha, and maybe not on Alpha either.
>
> The normal assumption regarding speed of disk IO would be that:
>
> RMS record IO ($GET and $PUT) < RMS block IO ($READ and $WRITE) <
> $QIO(W) < $IO_PERFORM(W) < memory mapped file
>
> (note that assumption and fact are spelled differently)

I'm not sure I have ever understood why people think memory mapped files
would be faster than a QIO under VMS.

With memory mapped I/O, what you essentially get is that I/O transfers
go directly from/to disk to user memory with a single operation. There
are no intermediate buffers, no additional copying. Which is what you
pretty much always have otherwise on Unix systems.

However, a QIO under VMS is already a direct communication between the
physical memory and the device with no intermediate buffers, additional
copying or whatever, unless I'm confused (and VMS changed compared to
RSX here...).

So how would memory mapped I/O be any faster? You basically cannot be
any faster than one DMA transfer. In fact, with memory mapped I/O, you
might be also hitting the page fault handling, and a reading in of a
full page, which might be more than you needed, causing some overhead as
well.

Also, what do $IO_PERFORM do, that could possibly make it faster than QIO?

I'm really curious about this topic...

> All of it can relative easily be done async as VMS has done async IO
> since forever.

Indeed. And it does it in a most wonderful way. Unfortunately, most
people never make full use of it, and it's often not that easy to use
from higher level languages.

>> My first concern is the use of VAX interlocked queue instructions,
>> because they're system calls on Itanium and x86 (and PALcode on
>> Alpha). They can't possibly run as fast as VMS programmers may be
>> assuming based on what made sense on a VAX 30 years ago. The current
>> hotness is to use lock-free data structures as much as possible.
>
> These constructs are certainly going to be relative
> more costly on newer system than on VAX.
>
> But I doubt that it matters much for overall performance.

Agreed. I doubt a lot of performance is hanging around these bits.

>> I also think it does make sense to have a shared pool of IOSBs and to
>> avoid dynamic memory allocation as much as possible. malloc() vs.
>> stack memory allocation is another area I'm curious about. We're not
>> in the embedded era with 64KB stack sizes or anything like that, so
>> if you know that an array will be a fixed 4KB or 8KB or 64KB max,
>> then why not put it on the stack? That seems like it'd be helpful on
>> anything post-VAX. The circular buffers that io_uring uses are also a
>> good template.
>
> I have never seen VMS code dynamic allocate an IOSB in heap - either
> stack or static heap.

I have, but I don't think it matters much where your IOSB is. Not sure
what the problem is here that is being addressed...

> But again then I don't think location of IOSB or reuse of IOSB's will
> matter much for overall performance.

Me neither. But a common trick (at least in my corner of the world) is
to have the IOSB as a part of some larger structure with context and
various information. Which means that when the AST is called, you
actually get the address of this context, since the address of the IOSB
is passed on to the AST handler.
No need to have anything locked, managed and shared around this. Each
IOSB as well as the whole data structure can be pretty private from the
start.

Johnny

On 2023-11-03 22:45, Dan Cross wrote:
> In article <ff7f2845-84cc-4c59-a007-1b388c82543fn@googlegroups.com>,
> Jake Hamby (Solid State Jake) <jake.hamby@gmail.com> wrote:
>> I've become a little obsessed with the question of how well
>> OpenVMS performs relative to Linux inside a VM, under different
>> conditions. My current obsession is the libuv library which
>> provides a popular async I/O abstraction layer implemented for
>> all the different flavors of UNIX that have async I/O, as well
>> as for Windows. What might a VMS version look like? How many
>> cores could it scale up to without too much synchronization
>> overhead?
>>
>> Alternatively, for existing VMS apps, how might they be sped up
>> on non-VAX hardware? Based on the mailbox copy driver loop in
>> the VMS port of Perl that I spent some time piecing together,
>> I've noticed a few patterns that can't possibly perform well on
>> any hardware newer than Alpha, and maybe not on Alpha either.
>>
>> My first concern is the use of VAX interlocked queue
>> instructions, because they're system calls on Itanium and x86
>> (and PALcode on Alpha). They can't possibly run as fast as VMS
>> programmers may be assuming based on what made sense on a VAX
>> 30 years ago. The current hotness is to use lock-free data
>> structures as much as possible.
>
> I don't know that that's the current hotness so much as trying
> to structure problems so that the vast majority of access are
> local to a core, so that either locks are unnecessary, or
> taking a lock is just an uncontended write to an owned cache
> line.
>
> Consider the problem of a per-core queue with work-stealing;
> the specifics of what's in the queue don't matter so much as the
> overall structure of the problem: items in the queue might
> represent IO requests, or they may represent runnable threads,
> or whatever. Anyway, a core will usually process things on
> its own local queue, but if it runs out of things to do, it
> may choose some victim and steal some work from it.
> Conventionally, this will involve everyone taking locks on
> these queues, but that's ok: we expect that work stealing is
> pretty rare, and that usually a core is simply locking its own
> queue, which will be an uncontended write on a cacheline it
> already owns. In any event, _most of the time_ the overhead
> of using a "lock" will be minimal (assuming relatively simply
> spinlocks or MCS locks or something like that).

I generally agree with everything you posted. But I think it's
unfortunate that you bring in cache lines into this, as it's somewhat
incorrect to talk about "owning" a cache line, and cache lines are not
really that relevant at all in this context.

I think you already know this - but multiple CPUs can have the same
memory cached, and that's just fine. Noone "owns" it. The exciting
moment is when you write data. Cached or not (it don't matter). When
writing, anything in any cache on any other CPU needs to be either
invalidated or updated with the new data. Most common is that it just
gets invalidated.

But that's all there is to it. Now, it is indeed very costly if you have
many CPUs trying to spin-lock on the same data, because each one will be
hitting the same memory, causing a big pressure on that memory address.
Of course, each CPU will then cache the memory address, but as soon as
anyone tries to take it, then all other CPUs cache entry will be
invalidated, and each one will need to read out from main memory again,
which scales very badly with many CPUs.

So, lockless algorithms, or clever locks that spread out over many
addresses, can help a lot here, since memory is so slow compared to
cache. Cache invalidation is costly. But essential for correctly working
software.
(And then we had the PDP-11/74, which had to implement cache coherency
between CPUs without any hardware support...)

Johnny

In article <ui59v2$rn5$3@news.misty.com>,
Johnny Billquist <bqt@softjar.se> wrote:
>On 2023-11-03 22:45, Dan Cross wrote:
>> In article <ff7f2845-84cc-4c59-a007-1b388c82543fn@googlegroups.com>,
>> Jake Hamby (Solid State Jake) <jake.hamby@gmail.com> wrote:
>>> I've become a little obsessed with the question of how well
>>> OpenVMS performs relative to Linux inside a VM, under different
>>> conditions. My current obsession is the libuv library which
>>> provides a popular async I/O abstraction layer implemented for
>>> all the different flavors of UNIX that have async I/O, as well
>>> as for Windows. What might a VMS version look like? How many
>>> cores could it scale up to without too much synchronization
>>> overhead?
>>>
>>> Alternatively, for existing VMS apps, how might they be sped up
>>> on non-VAX hardware? Based on the mailbox copy driver loop in
>>> the VMS port of Perl that I spent some time piecing together,
>>> I've noticed a few patterns that can't possibly perform well on
>>> any hardware newer than Alpha, and maybe not on Alpha either.
>>>
>>> My first concern is the use of VAX interlocked queue
>>> instructions, because they're system calls on Itanium and x86
>>> (and PALcode on Alpha). They can't possibly run as fast as VMS
>>> programmers may be assuming based on what made sense on a VAX
>>> 30 years ago. The current hotness is to use lock-free data
>>> structures as much as possible.
>>
>> I don't know that that's the current hotness so much as trying
>> to structure problems so that the vast majority of access are
>> local to a core, so that either locks are unnecessary, or
>> taking a lock is just an uncontended write to an owned cache
>> line.
>>
>> Consider the problem of a per-core queue with work-stealing;
>> the specifics of what's in the queue don't matter so much as the
>> overall structure of the problem: items in the queue might
>> represent IO requests, or they may represent runnable threads,
>> or whatever. Anyway, a core will usually process things on
>> its own local queue, but if it runs out of things to do, it
>> may choose some victim and steal some work from it.
>> Conventionally, this will involve everyone taking locks on
>> these queues, but that's ok: we expect that work stealing is
>> pretty rare, and that usually a core is simply locking its own
>> queue, which will be an uncontended write on a cacheline it
>> already owns. In any event, _most of the time_ the overhead
>> of using a "lock" will be minimal (assuming relatively simply
>> spinlocks or MCS locks or something like that).
>
>I generally agree with everything you posted. But I think it's
>unfortunate that you bring in cache lines into this, as it's somewhat
>incorrect to talk about "owning" a cache line, and cache lines are not
>really that relevant at all in this context.

This is an odd thing to say. It's quite common to talk about
core ownership of cache lines in both MESI and MOESI, which are
the cache coherency protocols in use on modern x86 systems from
both Intel and AMD, respectively. Indeed, the "O" in "MOESI" is
for "owned" (https://en.wikipedia.org/wiki/MOESI_protocol).

Cache lines and their ownership are massively relevant in the
context of mutual exclusion, atomic operations, and working on
shared memory generally.

>I think you already know this - but multiple CPUs can have the same
>memory cached, and that's just fine. Noone "owns" it. The exciting
>moment is when you write data. Cached or not (it don't matter). When
>writing, anything in any cache on any other CPU needs to be either
>invalidated or updated with the new data. Most common is that it just
>gets invalidated.

This is incorrect, or at least conflating several things that
make it sufficiently inaccurate overall as to be be effectively
incorrect.

Note that when we talk about ownership in this context, we talk
about ownership of a _cache line_, which is different than
ownership of _memory_. Recalling that the context is x86, we
can review the cache coherency protocols in use on
multiprocessor x86 systems, MESI and MOESI, which are based on
driving a state machine around snooping on accesses to "cache
lines", which are in turn relatively fine divisions of aligned
RAM regions (e.g., 64 bytes or 128 bytes are common). As
different processors in the system operate on memory, the cache
units use the coherency protocol to synchronize the state of
each processor's cache with respect to others. In these
protocols, we may think of their being a dedicated bus for cache
coherency operations. In both MESI and MOESI, when a cache line
is in the "exclusive" state for a CPU, we say that that CPU
"owns" that cache line. In MOESI, there is as mentioned above
also a dedicated "OWNED" state.

So "ownership" is very much a part of the model, and is used in
both the Intel SDM and AMD APM when discussing cache coherency
protocols.

>But that's all there is to it. Now, it is indeed very costly if you have
>many CPUs trying to spin-lock on the same data, because each one will be
>hitting the same memory, causing a big pressure on that memory address.

More accurately, you'll be pushing the associated cache line
through a huge number of state transitions as different cores
vye for exclusive ownership of the line in order to write the
lock value. Beyond a handful of cores, the generated cache
coherency traffic begins to dominate, and overall throughput
_decreases_.

This is of paramount importance when we start talking about
things like synchronization primitives, which are based on
atomic updates to shared memory; cache-inefficient algorithms
simply do not scale beyond a handful of actors, and why things
like MCS locks or CHM locks are used on many-core machines.
See, for example:
https://people.csail.mit.edu/nickolai/papers/boyd-wickizer-locks.pdf

This isn't to say that one cannot use naive spinlocks, simply
that one must be careful when doing so. For example, a spin
lock protecting a per-core work queue that is rarely contended
may be just fine if the only other writer is a single entity
that is dispatching work to available threads.

>Of course, each CPU will then cache the memory address, but as soon as
>anyone tries to take it, then all other CPUs cache entry will be
>invalidated, and each one will need to read out from main memory again,
>which scales very badly with many CPUs.

Eh, close. The description on Wikipedia for MESI isn't bad:
https://en.wikipedia.org/wiki/MESI_protocol

And the description of MOESI in the AMD APM is pretty good (sec. 7.3
of vol 2):
https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/programmer-references/24593.pdf

>So, lockless algorithms, or clever locks that spread out over many
>addresses, can help a lot here, since memory is so slow compared to
>cache. Cache invalidation is costly. But essential for correctly working
>software.

Yes and no. The point I have been trying to make is that
minimizing the need for cache coherency traffic scales;
"lockless" algorithms are still almost always based on atomic
operations and may collapse under load, as the pointed out by
the Boyd-Wickizer paper. If you want scalability, you have to
come up with some way to avoid doing that. E.g., MCS locks,
while a fairly traditional mutual exclusion primitive, may be
far more efficient than a "lockless" protocol.

>(And then we had the PDP-11/74, which had to implement cache coherency
>between CPUs without any hardware support...)

Sounds rough. :-/

- Dan C.

On 11/4/23 6:11 AM, Johnny Billquist wrote:

> I'm not sure I have ever understood why people think memory mapped files
> would be faster than a QIO under VMS.

I might've missed it but I haven't seen anyone say that. It's that
global sections are faster than mailboxes. The I/O API may be a
consideration but is secondary to the nature of the device.

> With memory mapped I/O, what you essentially get is that I/O transfers
> go directly from/to disk to user memory with a single operation. There
> are no intermediate buffers, no additional copying. Which is what you
> pretty much always have otherwise on Unix systems.
>
> However, a QIO under VMS is already a direct communication between the
> physical memory and the device with no intermediate buffers, additional
> copying or whatever, unless I'm confused (and VMS changed compared to
> RSX here...).
>
> So how would memory mapped I/O be any faster? You basically cannot be
> any faster than one DMA transfer. In fact, with memory mapped I/O, you
> might be also hitting the page fault handling, and a reading in of a
> full page, which might be more than you needed, causing some overhead as
> well.

I doubt that QIO with most device drivers is a simple as you make it out
to be. Last I heard they use Intermediate Request Packets (IRPs) and
other system resources in addition to the buffer space in the user
program. And in any case, QIO is limited to transfer sizes that these
days are considered pretty small.

> Also, what do $IO_PERFORM do, that could possibly make it faster than QIO?
>
> I'm really curious about this topic...

There's a pretty good explanation in the manual:

https://docs.vmssoftware.com/vsi-openvms-programming-concepts-manual-volume-ii/#FAST_IO_PATH

On 11/4/2023 1:06 PM, Craig A. Berry wrote:
>
> On 11/4/23 6:11 AM, Johnny Billquist wrote:
>
>> I'm not sure I have ever understood why people think memory mapped
>> files would be faster than a QIO under VMS.
>
> I might've missed it but I haven't seen anyone say that. It's that
> global sections are faster than mailboxes. The I/O API may be a
> consideration but is secondary to the nature of the device.

I guess that I as usual will have to plead guilty.

I wrote:

# The normal assumption regarding speed of disk IO would be that:
# # RMS record IO ($GET and $PUT) < RMS block IO ($READ and $WRITE) <
# $QIO(W) < $IO_PERFORM(W) < memory mapped file
# # (note that assumption and fact are spelled differently)

Arne

On 11/4/23 1:42 PM, Arne Vajhøj wrote:
> On 11/4/2023 1:06 PM, Craig A. Berry wrote:
>>
>> On 11/4/23 6:11 AM, Johnny Billquist wrote:
>>
>>> I'm not sure I have ever understood why people think memory mapped
>>> files would be faster than a QIO under VMS.
>>
>> I might've missed it but I haven't seen anyone say that. It's that
>> global sections are faster than mailboxes. The I/O API may be a
>> consideration but is secondary to the nature of the device.
>
> I guess that I as usual will have to plead guilty.
>
> I wrote:
>
> # The normal assumption regarding speed of disk IO would be that:
> #
> # RMS record IO ($GET and $PUT) < RMS block IO ($READ and $WRITE) <
> # $QIO(W) < $IO_PERFORM(W) < memory mapped file
> #
> # (note that assumption and fact are spelled differently)
>

Actually, I'm guilty of forgetting I read that message. I had in mind
Jake's original problem of IPC, not disk I/O, when I responded to
Johnny's remark about memory mapped files. For disk I/O, yes, it's
almost certain that using virtual memory primitives to synchronize
integral pages between disk and memory will be faster than any other I/O
method; that's why pretty much every database product on every platform
does it.

In article <ui64is$3go9i$1@dont-email.me>,
Craig A. Berry <craigberry@nospam.mac.com> wrote:
>On 11/4/23 1:42 PM, Arne Vajhøj wrote:
>> On 11/4/2023 1:06 PM, Craig A. Berry wrote:
>>>
>>> On 11/4/23 6:11 AM, Johnny Billquist wrote:
>>>
>>>> I'm not sure I have ever understood why people think memory mapped
>>>> files would be faster than a QIO under VMS.
>>>
>>> I might've missed it but I haven't seen anyone say that. It's that
>>> global sections are faster than mailboxes. The I/O API may be a
>>> consideration but is secondary to the nature of the device.
>>
>> I guess that I as usual will have to plead guilty.
>>
>> I wrote:
>>
>> # The normal assumption regarding speed of disk IO would be that:
>> #
>> # RMS record IO ($GET and $PUT) < RMS block IO ($READ and $WRITE) <
>> # $QIO(W) < $IO_PERFORM(W) < memory mapped file
>> #
>> # (note that assumption and fact are spelled differently)
>
>Actually, I'm guilty of forgetting I read that message. I had in mind
>Jake's original problem of IPC, not disk I/O, when I responded to
>Johnny's remark about memory mapped files. For disk I/O, yes, it's
>almost certain that using virtual memory primitives to synchronize
>integral pages between disk and memory will be faster than any other I/O
>method; that's why pretty much every database product on every platform
>does it.

Everyone starts out thinking that, but most are wrong:
https://db.cs.cmu.edu/mmap-cidr2022/

- Dan C.

On 11/4/2023 7:11 AM, Johnny Billquist wrote:
> On 2023-11-03 15:08, Arne Vajhøj wrote:
>> On 11/2/2023 9:02 PM, Jake Hamby (Solid State Jake) wrote:
>>> I've become a little obsessed with the question of how well OpenVMS
>>> performs relative to Linux inside a VM, under different conditions.
>>> My current obsession is the libuv library which provides a popular
>>> async I/O abstraction layer implemented for all the different flavors
>>> of UNIX that have async I/O, as well as for Windows. What might a VMS
>>> version look like? How many cores could it scale up to without too
>>> much synchronization overhead?
>>>
>>> Alternatively, for existing VMS apps, how might they be sped up on
>>> non-VAX hardware? Based on the mailbox copy driver loop in the VMS
>>> port of Perl that I spent some time piecing together, I've noticed a
>>> few patterns that can't possibly perform well on any hardware newer
>>> than Alpha, and maybe not on Alpha either.
>>
>> The normal assumption regarding speed of disk IO would be that:
>>
>> RMS record IO ($GET and $PUT) < RMS block IO ($READ and $WRITE) <
>> $QIO(W) < $IO_PERFORM(W) < memory mapped file
>>
>> (note that assumption and fact are spelled differently)
>
> I'm not sure I have ever understood why people think memory mapped files
> would be faster than a QIO under VMS.

Very few layers.

Large degree of freedom to the OS about how to read.

> With memory mapped I/O, what you essentially get is that I/O transfers
> go directly from/to disk to user memory with a single operation. There
> are no intermediate buffers, no additional copying. Which is what you
> pretty much always have otherwise on Unix systems.
>
> However, a QIO under VMS is already a direct communication between the
> physical memory and the device with no intermediate buffers, additional
> copying or whatever, unless I'm confused (and VMS changed compared to
> RSX here...).

XFC?

> So how would memory mapped I/O be any faster? You basically cannot be
> any faster than one DMA transfer. In fact, with memory mapped I/O, you
> might be also hitting the page fault handling, and a reading in of a
> full page, which might be more than you needed, causing some overhead as
> well.

Fewer layers to go through. More freedom to read ahead.

> Also, what do $IO_PERFORM do, that could possibly make it faster than QIO?

$QIO(W) is original. $IO_PERFORM(W) was added much later.

$IO_PERFORM(W) is called fast path IO. The name and the fact
that it was added later hint at it being faster.

That name has always give me associations to a strategy of
doing lots of checks upfront and then skip layers
and checks when doing the actual reads/writes. But I
have no idea if that is actually what it does.

Arne

On 2023-11-04 21:44:24 +0000, Arne Vajhøj said:

> $QIO(W) is original. $IO_PERFORM(W) was added much later.
>
> $IO_PERFORM(W) is called fast path IO. The name and the fact that it
> was added later hint at it being faster.
>
> That name has always give me associations to a strategy of doing lots
> of checks upfront and then skip layers and checks when doing the actual
> reads/writes. But I have no idea if that is actually what it does.

Careful with the "fast path IO" names, here.

From somebody long ago and far away, an overview of Fast Path:
http://h41379.www4.hpe.com/wizard/wiz_2358.html

I wouldn't expect much of a performance benefit to Fast Path on a
uniprocessor configuration. If anything.

Fast I/O may well be useful here, though. Fast I/O is effectively a
version of the $qio read and write paths, reworked for 64-bit support
and for speed.

Quoth the doc: "In total, Fast I/O services eliminate four spinlock
acquisitions per I/O (two for the MMG spinlock and two for the SCHED
spinlock). The reduction in CPU cost per I/O is 20 percent for
uniprocessor systems and 10 percent for multiprocessor systems."

Additionally, use of buffer objects can usually improve somewhat upon
that Fast I/O performance, though how much depends on the details of
local system memory activity.

Fast Path and Fast I/O were, are, and will always remain stupid feature names.

If I really need performance (or often times, if I want simpler app
coding), I'll use memory mapped sections and keep the data in memory.
There are tradeoffs there around locking and interprocess notifications
of course, and as has been mentioned. Conversely, using the file system
manages the memory caches and the interlocking and notifications for
me. TANSTAAFL, etc.

PS: IIRC, there are some discussions around here about potential
benefits of shutting off some of the in-memory caching when using SSDs,
too. ODS-2 and ODS-5 and the I/O stack all well pre-date SSD storage,
and current speeds and feeds, and have different trade-offs.

--
Pure Personal Opinion | HoffmanLabs LLC

On 11/4/2023 9:53 PM, Stephen Hoffman wrote:
> On 2023-11-04 21:44:24 +0000, Arne Vajhøj said:
>
>> $QIO(W) is original. $IO_PERFORM(W) was added much later.
>>
>> $IO_PERFORM(W) is called fast path IO. The name and the fact that it
>> was added later hint at it being faster.
>>
>> That name has always give me associations to a strategy of doing lots
>> of checks upfront and then skip layers and checks when doing the
>> actual reads/writes. But I have no idea if that is actually what it does.
>
> Careful with the "fast path IO" names, here.
>
> From somebody long ago and far away, an overview of Fast Path:
> http://h41379.www4.hpe.com/wizard/wiz_2358.html
>
> I wouldn't expect much of a performance benefit to Fast Path on a
> uniprocessor configuration. If anything.
>
> Fast I/O may well be useful here, though. Fast I/O is effectively a
> version of the $qio read and write paths, reworked for 64-bit support
> and for speed.
>
> Quoth the doc: "In total, Fast I/O services eliminate four spinlock
> acquisitions per I/O (two for the MMG spinlock and two for the SCHED
> spinlock). The reduction in CPU cost per I/O is 20 percent for
> uniprocessor systems and 10 percent for multiprocessor systems."
>
> Additionally, use of buffer objects can usually improve somewhat upon
> that Fast I/O performance, though how much depends on the details of
> local system memory activity.
>
> Fast Path and Fast I/O were, are, and will always remain stupid feature
> names.

So $IO_FASTPATH(W) is for "Fast Path" and for speeding
up IO on multi-processors while $IO_SETUP + $IO_PERFORM(W) +
$IO_CLEANUP are for "Fast IO" and for reducing spinlock overhead?

And no relation - $QIO(W) or $IO_PERFORM(W) and fast path on
or fast path off are 4 valid options?

Arne

On 11/3/2023 8:35 PM, Jake Hamby (Solid State Jake) wrote:
> Something else just occurred to me. While file I/O performance is
> important, as well as local IPC, I've also been keeping in mind
> something else: simultaneous socket connections. One benefit of
> everyone going to RESTful APIs is you're not keeping a long-lived TCP
> socket open like you would with other types of protocols.

HTTP does support keep-alive.

And RESTful web services do use that. Especially for high volume
server to server setups.

> There have been a lot of posts on my LinkedIn feed from IBM
> mainframers and they are more excited than ever about IBM's latest
> hardware which is very fast at running COBOL, Java, Go, C++, Db2,
> CICS, IMS, TSO/E, ISPF, Python. Technologies from every decade.
>
> Mainframes are great for all the applications that are difficult to
> shard. Giant databases of bank transactions. I think software
> engineers want every problem to be something you can shard and
> parallelize to run in an AWS or Google cluster, but you can't do that
> with billions of lines of COBOL. But the reason everyone's still
> using that COBOL is that there's no real gain from rewriting it
> because the problems can't be solved in some new way.

There are banks switching from Cobol on mainframe to newer
technologies. Newer technologies that reduce cost significantly
and are more online/24x7 friendly. The benefits from changing
are there.

But so are the cost. The migration will cost a fortune. All the
bugs introduced when migrating such large code bases will have
a huge indirect cost. And the lost profits from not adding new features
while migrating could be huge also.

So only a few CIO's chose to actually migrate. Most CIO's chose to
create new stuff using new technologies and put new technologies in
front of the mainframe instead of replacing the mainframe. A part of
those in the latter category has a "we will migrate in
<current year + 5 years>" roadmap.

Arne

On 11/4/23 4:41 PM, Dan Cross wrote:
> In article <ui64is$3go9i$1@dont-email.me>,
> Craig A. Berry <craigberry@nospam.mac.com> wrote:
>> For disk I/O, yes, it's
>> almost certain that using virtual memory primitives to synchronize
>> integral pages between disk and memory will be faster than any other I/O
>> method; that's why pretty much every database product on every platform
>> does it.
>
> Everyone starts out thinking that, but most are wrong:
> https://db.cs.cmu.edu/mmap-cidr2022/

Interesting article. I don't buy its conclusions as an indication of
what most databases do. Its main point seems to be that mapped memory
makes transactions difficult because, "due to transparent paging, the OS
can flush a dirty page to secondary storage at any time, irrespective of
whether the writing transaction has committed." Well, yeah, you have to
lock pages in memory, and apparently mlock() doesn't guarantee that it
does what is says on the tin. They also seem to be surprised by the fact
that paging to a remote storage system is slow. Well, duh. So there
are complexities and difficulties arising from using mapped memory.

But there are also complexities and difficulties with their recommended
alternative of "traditional buffer pools" and libaio or io_uring. Stack
Overflow is full of articles about problems, including broken
implementations and security problems, with those APIs. I don't doubt
they can be useful if the complexities and difficulties are managed
correctly, but the same is true of mapped memory.

Also noteworthy is that in the long list of databases whose
implementations they considered, SQLite is the only major player they
mention. SQLite has an unusual storage model (data typing imposed at run
time, IIRC) and was never intended as a multi-user OLTP system, so its
I/O choices aren't much of a guide to what databases in general do.

If the authors of the article could show evidence that the research
teams at Oracle, Microsoft, and IBM have come to the same conclusions
they did and don't use mapped memory in their products anymore, that
would be interesting, but they make only an oblique reference to SQL
Server in a context that implies it does use mapped memory.

I suspect all of the major databases use any and every I/O mechanism
available in different situations, chosen by a variety of engine
choices, run-time heuristics, and configuration options. Mapped memory
may not be the only game in town besides file I/O like it once was, but
I'm just not buying that it's been entirely eclipsed.

On 11/5/2023 9:28 AM, Arne Vajhøj wrote:
> On 11/3/2023 8:35 PM, Jake Hamby (Solid State Jake) wrote:
>> Something else just occurred to me. While file I/O performance is
>> important, as well as local IPC, I've also been keeping in mind
>> something else: simultaneous socket connections. One benefit of
>> everyone going to RESTful APIs is you're not keeping a long-lived TCP
>> socket open like you would with other types of protocols.
>
> HTTP does support keep-alive.
>
> And RESTful web services do use that. Especially for high volume
> server to server setups.
>
>> There have been a lot of posts on my LinkedIn feed from IBM
>> mainframers and they are more excited than ever about IBM's latest
>> hardware which is very fast at running COBOL, Java, Go, C++, Db2,
>> CICS, IMS, TSO/E, ISPF, Python. Technologies from every decade.
>>
>> Mainframes are great for all the applications that are difficult to
>> shard. Giant databases of bank transactions. I think software
>> engineers want every problem to be something you can shard and
>> parallelize to run in an AWS or Google cluster, but you can't do that
>> with billions of lines of COBOL. But the reason everyone's still
>> using that COBOL is that there's no real gain from rewriting it
>> because the problems can't be solved in some new way.
>
> There are banks switching from Cobol on mainframe to newer
> technologies. Newer technologies that reduce cost significantly
> and are more online/24x7 friendly. The benefits from changing
> are there.
>
> But so are the cost. The migration will cost a fortune. All the
> bugs introduced when migrating such large code bases will have
> a huge indirect cost. And the lost profits from not adding new features
> while migrating could be huge also.
>
> So only a few CIO's chose to actually migrate. Most CIO's chose to
> create new stuff using new technologies and put new technologies in
> front of the mainframe instead of replacing the mainframe. A part of
> those in the latter category has a "we will migrate in
> <current year + 5 years>" roadmap.

And, so the story goes.

In 1980 I was a COBOL Programmer/Systems Analyst on IBM and Univac
Mainframes. Trade journals were already saying "COBOL is dead".
And yet, it went on. Two of the largest ISes in the country (probably
the world) were COBOL. Still are today and there is no plan or sign that
they will ever be replaced with another language. There was a third.
Contractor opted to not renew and the program died. Not because of any
flaws in COBOL but because academia refuses to teach it even as an
elective. System belonged to the contractor so stayed with them. New
system written from scratch in god only knows what language, Some
language du jour. The new system is slow, cumbersome, error prone and
lacks many of the features that the old system had.

We have so many "colleges" teaching trade school courses (like diesel
mechanics, HVAC welding and even motorcycle mechanics)I really wish
trade schools would step up to the plate ad start teaching IT and in
particular thing like COBOL, Fortran and PL/I. They are not going away.

bill

On 11/5/2023 10:58 AM, bill wrote:
> In 1980 I was a COBOL Programmer/Systems Analyst on IBM and Univac
> Mainframes.  Trade journals were already saying "COBOL is dead".
> And yet, it went on.  Two of the largest ISes in the country (probably
> the world) were COBOL. Still are today and there is no plan or sign that
> they will ever be replaced with another language.  There was a third.
> Contractor opted to not renew and the program died.  Not because of any
> flaws in COBOL but because academia refuses to teach it even as an
> elective.  System belonged to the contractor so stayed with them.  New
> system written from scratch in god only knows what language, Some
> language du jour.  The new system is slow, cumbersome, error prone and
> lacks many of the features that the old system had.
>
> We have so many "colleges" teaching trade school courses (like diesel
> mechanics, HVAC welding and even motorcycle mechanics)I really wish
> trade schools would step up to the plate ad start teaching IT and in
> particular thing like COBOL, Fortran and PL/I.  They are not going away.

There is not much point, because there will not be jobs
for them.

There are a lot of Cobol and PL/I code in production doing
usually highly business critical stuff.

But if cost and risk are too high to rewrite to C++
or Java or C# or whatever, then the risk of
rewriting it in Cobol is also too high.

So instead the new functionality is put in
secondary systems using newer technology and
only changes that has to be done in the core
are done there.

All the less critical but developer time consuming
code in UI and reporting are long gone.

As a result the demand is small.

It is not zero but despite the frequent "we have
a problem" announcements, then it seems like they can
keep and hire+train the people they need for
maintenance.

Arne

On 11/5/2023 12:37 PM, John Dallman wrote:
> In article <kqpsd9F6p36U1@mid.individual.net>, bill.gunshannon@gmail.com
> (bill) wrote:
>
>> I really wish trade schools would step up to the plate and start
>> teaching IT and in particular thing like COBOL, Fortran and PL/I.
>> They are not going away.
>
> However, recruiting young people onto those courses, rather than ones in
> mobile app creation, might be quite hard.

Not every student studying computers is a budding computer scientist.
Most schools with CS programs also have CIS programs and a lot of
students go that route.

> You'd also need relevant
> compilers, operating systems and hardware.

Just like VMS has a student program IBM and UNISYS, the two biggest
mainframers left, offer programs for people to familiarize with their
world. There are also Open Source COBOL and Fortran systems available.
PL/i being the red headed step child. (As a side note, the recent
edition of the CACM has an article that claims IBM intended PL/I to
take over the computing world from the likes of COBOL, Fortran and
ALGOL. Funny considering that they kept it hidden in their corporate
bowels for so long.)

>
> I'm not saying this is a bad idea, but if it was easy, it might well
> already be happening.

The problem goes back to marketing. Academia has held it down for so
long that people don't know about it. It will take a sell but right now
it is hard to find the right person to sell it. Maybe this whole loan
debt thing will finally provide the push it needs.

>
> I learned the basics of Fortran and Cobol as an undergraduate in the
> early eighties, and have never had any desire to do more.

And I also started with COBOL and Fortran (but no college, that came
like documentation, at the very end. I got my degree 4 years before
I retired.) and while I have done (and still do) a lot of other kinds
of programming, network, OS, embedded systems, RTOS, etc. I always found
the COBOL and Fortran to be the work I enjoy the most.

bill