Historically, every file-intensive application running under Unix got the benefit the system's
buffer cache without explicit tuning done by the programmer. The similar mechanism under
VMS, the virtual file cache, wasn't wasn't added until much later after VMS's reputation for
abysmal I/O performance was firmly entrenched. With the VFC in place, I/O performance is
now merely poor.

For SQLite, my VFS (Virtual File System) module uses RMS, but opens the file with the user
I/O option -- reads and writes are done with SYS$QIO. The VFS implements it's own buffer
cache as well. Since the VFS only gets what spills out of SQLite's own page cache, the VFS
cache doesn't show much effect with my benchmark program until sized to 8 Mbyte or so
(~1.75:1 speedup with spinning rust, 1.25:1 with solid state).

On Wednesday, August 30, 2023 at 7:24:56 AM UTC-4, Ian Miller wrote:
> I'm having trouble with this link http://rlgsc.com/r/20220506.html - page not found

Not sure what the problem is. I just clicked on the hyperlink in the Google groups feed and it worked without a problem.

- Bob Gezelter, http://www.rlgsc.com

On Wednesday, August 30, 2023 at 7:24:56 AM UTC-4, Ian Miller wrote:
> I'm having trouble with this link http://rlgsc.com/r/20220506.html - page not found
Ian,

Just checked the www server logs on this end. No 404s associated with that URL (just the usual 404s on favicon).

If the problem repeats, please let me know via email.

- Bob Gezelter, http://www.rlgsc.com

Den 2023-08-30 kl. 14:18, skrev Bob Gezelter:
> On Wednesday, August 30, 2023 at 7:24:56 AM UTC-4, Ian Miller wrote:
>> I'm having trouble with this link http://rlgsc.com/r/20220506.html - page not found
> Ian,
>
> Just checked the www server logs on this end. No 404s associated with that URL (just the usual 404s on favicon).
>
> If the problem repeats, please let me know via email.
>
> - Bob Gezelter, http://www.rlgsc.com

Opens fine over here also.

It does forward to:
http://www.rlgsc.com/ieee/longisland/2022/revisiting-mass-storage-presumptions.html

But still opens fine...

In article <ucm01o$2ehk3$2@dont-email.me>,
Arne Vajhøj <arne@vajhoej.dk> wrote:
>On 8/29/2023 7:30 PM, Arne Vajhøj wrote:
>> My expectation would be that:
>...
>> * the CPU's running VMS today are so fast and the network cards
>>   supported by VMS so slow that OS overhead is not a huge problem
>>
>> But I could be wrong.
>
>I took a quick look at:
> https://www.islandco.com/hp-parts/hp-integrity-network-options
>
>It seems like the most common netword card speed is 1 Gbit, but
>that 10 Gbit cards are available.
>
>I don't see 400 Gbps being possible. Not enough slots in an Itanium
>to be able to put enough 10 Gbit cards in to achieve that.

No one doing 400 Gbps is using 40 x 10Gbps cards to do so,
unless it's some kind of a "hold my beer" dare.

They're using something like Mellanox ConnectX-6 adapters in
whatever configuration, or Chelsio terminator parts, or similar.
I'm sure VMS does not support these, but could be made to.

Itanium has nothing to do with it.

- Dan C.

On Wednesday, August 30, 2023 at 1:40:31 PM UTC+1, Jan-Erik Söderholm wrote:
> Den 2023-08-30 kl. 14:18, skrev Bob Gezelter:
> > On Wednesday, August 30, 2023 at 7:24:56 AM UTC-4, Ian Miller wrote:
> >> I'm having trouble with this link http://rlgsc.com/r/20220506.html - page not found
> > Ian,
> >
> > Just checked the www server logs on this end. No 404s associated with that URL (just the usual 404s on favicon).
> >
> > If the problem repeats, please let me know via email.
> >
> > - Bob Gezelter, http://www.rlgsc.com
> Opens fine over here also.
>
> It does forward to:
> http://www.rlgsc.com/ieee/longisland/2022/revisiting-mass-storage-presumptions.html
>
> But still opens fine...
that works for me

On 2023-08-29, Johnny Billquist <bqt@softjar.se> wrote:
> On 2023-08-29 19:25, Simon Clubley wrote:
>> On 2023-08-29, Single Stage to Orbit <alex.buell@munted.eu> wrote:
>>> On Thu, 1970-01-01 at 00:00 +0000, Simon Clubley wrote:
>>>>> Very much FreeBSD here for some years, after decades first with
>>>>> dec,
>>>>> then Sun. Forms the basic of at least some proprietary offerings,
>>>>> as
>>>>> well as millions of embedded devices. Linux is still a unix,
>>>>> and runs the majority of web sites of the world, so if anything,
>>>>> unix has won the os wars...
>>>>>
>>>>
>>>> Yes, very much so. (And I can't believe Arne thinks the *BSDs have no
>>>> serious users... :-) ).
>>>
>>> Netflix picked FreeBSD as it could chuck out data at 400GB/s. Linux was
>>> not even close.
>>
>> 10 years from now (assuming the economy hasn't collapsed by then :-) ):
>>
>> 400GB/s ??? Is that all ??? Amateurs!!! :-)
>
> Well. I have this friend of mine, who installed 40 Gb/s at his moms
> place in 2007...
>
>> On a more serious note, I wonder what the maximum rate VMS is capable
>> of emitting data at if it was using the fastest network hardware
>> available.
>
> What a weird question. VMS in itself don't have any limits. It's all
> always just about the hardware.

As you have just been educated by multiple people, this statement is
nonsense and shows you have no understanding of the tradeoffs involved
in OS design and how those tradeoffs have changed over time.

You have also just quoted a message above that says two very conceptually
similar designs (Linux and FreeBSD) still manage to have very different
performance outcomes.

> Some software might be able to squeeze more out of the same hardware,
> but just spin up faster hardware, and you'll get higher throughput. But
> any system will basically just be limited by the speed of the network
> hardware, if that item is fixed. You can't go above that. But there are
> no reasons why you wouldn't be able to get to that point.
>

Why do you say that ? There will always be OS overheads. The only question
is how large are those overheads ?

> These are the kind of questions that sometimes make me wonder if you
> know anything at all about computers. But then you do some other posts
> which clearly demonstrate that you do understand some stuff.
>

The problem with you Johnny is that you appear to be a very arrogant
person who assumes that if you don't understand what a person is
saying, then you assume that person is talking nonsense.

You never seem to consider the possibility that the lack of understanding
might be at your end instead of in the person whose comments you are reading.

I, OTOH, enjoy being pointed to new ideas and knowledge by others in
this newsgroup. Look at the recent ALGOL discussions for one example.
I found out quite a bit about systems I simply didn't know about until then.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 2023-08-29, bill <bill.gunshannon@gmail.com> wrote:
> On 8/29/2023 3:18 PM, Johnny Billquist wrote:
>> On 2023-08-29 19:25, Simon Clubley wrote:
>>> On 2023-08-29, Single Stage to Orbit <alex.buell@munted.eu> wrote:
>>>> On Thu, 1970-01-01 at 00:00 +0000, Simon Clubley wrote:
>>>>>> Very much FreeBSD here for some years, after decades first with
>>>>>> dec,
>>>>>> then Sun. Forms the basic of at least some proprietary offerings,
>>>>>> as
>>>>>> well as millions of embedded devices. Linux is still a unix,
>>>>>> and runs the majority of web sites of the world, so if anything,
>>>>>> unix has won the os wars...
>>>>>>
>>>>>
>>>>> Yes, very much so. (And I can't believe Arne thinks the *BSDs have no
>>>>> serious users... :-) ).
>>>>
>>>> Netflix picked FreeBSD as it could chuck out data at 400GB/s. Linux was
>>>> not even close.
>>>
>>> 10 years from now (assuming the economy hasn't collapsed by then :-) ):
>>>
>>> 400GB/s ??? Is that all ??? Amateurs!!! :-)
>>
>> Well. I have this friend of mine, who installed 40 Gb/s at his moms
>> place in 2007...
>>
>>> On a more serious note, I wonder what the maximum rate VMS is capable
>>> of emitting data at if it was using the fastest network hardware
>>> available.
>>
>> What a weird question. VMS in itself don't have any limits. It's all
>> always just about the hardware.
>
> Not really. VMS has always been notoriously slow with I/O and I assume
> that's what Simon was hinting at.
>

Yes, I was.

>> Some software might be able to squeeze more out of the same hardware,
>> but just spin up faster hardware, and you'll get higher throughput. But
>> any system will basically just be limited by the speed of the network
>> hardware, if that item is fixed. You can't go above that. But there are
>> no reasons why you wouldn't be able to get to that point.
>
> You are forgetting the overhead of the drivers and any underlying O/S
> code that has to be called to do it.
>

Yes. And the tradeoffs in the OS design directly affect the level of
overhead.

>>
>> These are the kind of questions that sometimes make me wonder if you
>> know anything at all about computers. But then you do some other posts
>> which clearly demonstrate that you do understand some stuff.
>
> I think it was just a subtle poke in the ribs for VMS.
>

Yes. :-)

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

In article <uclgav$q0b$1@news.misty.com>,
Johnny Billquist <bqt@softjar.se> wrote:
>On 2023-08-29 19:25, Simon Clubley wrote:
>[snip]
>> On a more serious note, I wonder what the maximum rate VMS is capable
>> of emitting data at if it was using the fastest network hardware
>> available.
>
>What a weird question. VMS in itself don't have any limits. It's all
>always just about the hardware.
>Some software might be able to squeeze more out of the same hardware,
>but just spin up faster hardware, and you'll get higher throughput. But
>any system will basically just be limited by the speed of the network
>hardware, if that item is fixed. You can't go above that. But there are
>no reasons why you wouldn't be able to get to that point.

I don't really understand this line of reasoning. At some point
you're running on the fastest hardware available; at that point,
what do you do? It seems perfectly reasonable to try and
quantify the overhead due to the software stack, and if possible
to optimize it, particularly if your chosen software platform
can't saturate the NIC at line rate. And even if it can, if it
requires all of the CPU resources on your machine to do so, then
you're starving the platform for cycles that would otherwise go
to the software that all of those hungry network clients want to
talk to.

"Throw more hardware at it" works well until it doesn't, and
when that happens you've hit a wall that you have to get around
some other way.

- Dan C.

On 2023-08-29, Johnny Billquist <bqt@softjar.se> wrote:
> On 2023-08-29 22:27, bill wrote:
>> On 8/29/2023 3:18 PM, Johnny Billquist wrote:
>>>
>>> What a weird question. VMS in itself don't have any limits. It's all
>>> always just about the hardware.
>>
>> Not really.  VMS has always been notoriously slow with I/O and I assume
>> that's what Simon was hinting at.
>
> So? It just means that other systems might achieve higher rate of I/O
> throughput than VMS on a specific piece of hardware. Nothing prevents me
> from throwing faster hardware on the problem until I saturate the
> network, no matter which OS I'm using.
>

Why would you waste serious money on extra hardware to overcome OS
limitations instead of just using a more efficient OS for the task at hand ?

>>> Some software might be able to squeeze more out of the same hardware,
>>> but just spin up faster hardware, and you'll get higher throughput.
>>> But any system will basically just be limited by the speed of the
>>> network hardware, if that item is fixed. You can't go above that. But
>>> there are no reasons why you wouldn't be able to get to that point.
>>
>> You are forgetting the overhead of the drivers and any underlying O/S
>> code that has to be called to do it.
>
> Irrelevant. That just means I need to throw some more hardware on the
> problem. Not that the OS somehow prevents me from achieving a higher
> throughput.
>

Why would you waste serious money on extra hardware to overcome OS
limitations instead of just using a more efficient OS for the task at hand ?

>>> These are the kind of questions that sometimes make me wonder if you
>>> know anything at all about computers. But then you do some other posts
>>> which clearly demonstrate that you do understand some stuff.
>>
>> I think it was just a subtle poke in the ribs for VMS.
>
> I think it was a display of complete ignorance on behalf of Simon.
> Possibly just not thinking before writing. Who knows...?
>

That's seriously rich coming from you Johnny. Seriously rich.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 2023-08-30, Ian Miller <gxys@uk2.net> wrote:
> I'm having trouble with this link http://rlgsc.com/r/20220506.html - page not found
>

Works OK for me (and now I see what Bob looks like. :-)).

Is your work firewall getting in the way ?

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

In article <ucneo1$2o3hj$3@dont-email.me>,
Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> wrote:
>On 2023-08-29, Johnny Billquist <bqt@softjar.se> wrote:
>> On 2023-08-29 22:27, bill wrote:
>>> On 8/29/2023 3:18 PM, Johnny Billquist wrote:
>>>>
>>>> What a weird question. VMS in itself don't have any limits. It's all
>>>> always just about the hardware.
>>>
>>> Not really.� VMS has always been notoriously slow with I/O and I assume
>>> that's what Simon was hinting at.
>>
>> So? It just means that other systems might achieve higher rate of I/O
>> throughput than VMS on a specific piece of hardware. Nothing prevents me
>> from throwing faster hardware on the problem until I saturate the
>> network, no matter which OS I'm using.
>
>Why would you waste serious money on extra hardware to overcome OS
>limitations instead of just using a more efficient OS for the task at hand ?

Just throwing this out there....

One reason may be that optimizing software is hard, expensive,
and can take a long time. If you're working on a $n$-month
hardware refresh rate anyway, then you may be better off just
waiting for the next hardware generation.

Also, over time some optimizations either turn into
pessimizations or force you into a design shape that makes
future change difficult. An example of this is the Unix system
interface; it was deliberately designed to be highly synchronous
as it was thought that asynch interfaces were cumbersome for
"workaday" programming. However, we now have languages with
large, asynchronous runtimes; squeezing these through the
soda straw of a Unix-style system interface can be challenging.

Anyway, I agree with you that looking at system overhead is
worthwhile, but I can see an argument for opting for a
hardware solution to the problem as well.

- Dan C.

On 8/30/2023 9:04 AM, Simon Clubley wrote:
> On 2023-08-29, Johnny Billquist <bqt@softjar.se> wrote:
>> On 2023-08-29 22:27, bill wrote:
>>> On 8/29/2023 3:18 PM, Johnny Billquist wrote:
>>>>
>>>> What a weird question. VMS in itself don't have any limits. It's all
>>>> always just about the hardware.
>>>
>>> Not really. VMS has always been notoriously slow with I/O and I assume
>>> that's what Simon was hinting at.
>>
>> So? It just means that other systems might achieve higher rate of I/O
>> throughput than VMS on a specific piece of hardware. Nothing prevents me
>> from throwing faster hardware on the problem until I saturate the
>> network, no matter which OS I'm using.
>>
>
> Why would you waste serious money on extra hardware to overcome OS
> limitations instead of just using a more efficient OS for the task at hand ?
>
>>>> Some software might be able to squeeze more out of the same hardware,
>>>> but just spin up faster hardware, and you'll get higher throughput.
>>>> But any system will basically just be limited by the speed of the
>>>> network hardware, if that item is fixed. You can't go above that. But
>>>> there are no reasons why you wouldn't be able to get to that point.
>>>
>>> You are forgetting the overhead of the drivers and any underlying O/S
>>> code that has to be called to do it.
>>
>> Irrelevant. That just means I need to throw some more hardware on the
>> problem. Not that the OS somehow prevents me from achieving a higher
>> throughput.
>>
>
> Why would you waste serious money on extra hardware to overcome OS
> limitations instead of just using a more efficient OS for the task at hand ?

Well, that sort of depends upon the "task at hand", doesn't it?

Perhaps the OS has other features one might be depending upon.

>>>> These are the kind of questions that sometimes make me wonder if you
>>>> know anything at all about computers. But then you do some other posts
>>>> which clearly demonstrate that you do understand some stuff.
>>>
>>> I think it was just a subtle poke in the ribs for VMS.
>>
>> I think it was a display of complete ignorance on behalf of Simon.
>> Possibly just not thinking before writing. Who knows...?

Here now, what's this? I'm officially in charge of poking fun at Simon. Are
you usurping my job?

> That's seriously rich coming from you Johnny. Seriously rich.

You tell him Simon ...

:-)

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

With regards to mass storage, I will note that using recent generation HPE storage arrays and rx2800 i6 servers, I did not find it difficult to push mass storage load to the point that the storage controllers were "heavily" loaded and not "amused".

To make the storage controllers happy, I had to throttle things back a bit.

A fair number of client matters in my consulting practice have related to performance. In so far as mass storage I/O is concerned, my experience over the past three decades has been that almost all OpenVMS "mass storage" performance problems have root causes of poor parameters, not any particular quirk of OpenVMS or RMS. The poor parameter choices run the gamut including page limits; file organization, RMS parameters, disk volume configuration, and many others. I have seen a fairly broad sample over the years. A reconsideration can often increase performance significantly.

That is not to say, as I did in my Ph.D. dissertation and the referenced paper that OpenVMS can do better. Even in out present fairly well-resourced environments, compute cycles are better used to productive work, not unnecessary overhead. We can and should do better.

- Bob Gezelter, http://www.rlgsc.com

On 8/30/2023 4:53 AM, Johnny Billquist wrote:
> Not sure how easy it is to dodge RMS under VMS. In RSX, you can just do
> the QIOs to the ACP yourself and go around the whole thing, which makes
> I/O way faster. Of course, since files still have this structure thing,
> most of the time you are still going to have to pay for it somewhere.
> But if you are happy with just raw disk blocks, the basic I/O do not
> have near as much penalty. Admitted, the ODS-1 (as well as ODS-2)
> structure have some inherent limitations that carry some cost as well.
> So you could improve things some by doing some other implementation on
> the file system level.
> But mainly, no matter what the file system design is, you are still
> going to have the pain of RMS, which is the majority of the cost. And
> you'll never get away from this as long as you use VMS.

SYS$QIO(W) for files works fine on VMS too.

But a bit of a hassle to use.

There are two alternative ways to to bypass RMS:
* SYS$IO_PERFORM(W) - the "fast I/O" thingy
* SYS$CRMPSC - mapping the file to memory

Arne

On 8/30/2023 9:04 AM, Simon Clubley wrote:
> On 2023-08-29, Johnny Billquist <bqt@softjar.se> wrote:
>> On 2023-08-29 22:27, bill wrote:
>>> On 8/29/2023 3:18 PM, Johnny Billquist wrote:
>>>> What a weird question. VMS in itself don't have any limits. It's all
>>>> always just about the hardware.
>>>
>>> Not really.  VMS has always been notoriously slow with I/O and I assume
>>> that's what Simon was hinting at.
>>
>> So? It just means that other systems might achieve higher rate of I/O
>> throughput than VMS on a specific piece of hardware. Nothing prevents me
>> from throwing faster hardware on the problem until I saturate the
>> network, no matter which OS I'm using.
>
> Why would you waste serious money on extra hardware to overcome OS
> limitations instead of just using a more efficient OS for the task at hand ?

I think we need to know then what is meant by VMS "limit".

A) Is the limit what VMS can handle if enough hardware is thrown at it?

B) Or is the limit what VMS can do cost efficiently without any extra
hardware cost?

The original question was:

# I wonder what the maximum rate VMS is capable
# of emitting data at if it was using the fastest network hardware
# available.

I read that as #A.

Obviously #B is more relevant for what tasks VMS will be chosen
for, but that was not the question asked as I read it.

Arne

On 2023-08-30 15:04, Simon Clubley wrote:
> On 2023-08-29, Johnny Billquist <bqt@softjar.se> wrote:
>> On 2023-08-29 22:27, bill wrote:
>>> On 8/29/2023 3:18 PM, Johnny Billquist wrote:
>>>>
>>>> What a weird question. VMS in itself don't have any limits. It's all
>>>> always just about the hardware.
>>>
>>> Not really.  VMS has always been notoriously slow with I/O and I assume
>>> that's what Simon was hinting at.
>>
>> So? It just means that other systems might achieve higher rate of I/O
>> throughput than VMS on a specific piece of hardware. Nothing prevents me
>> from throwing faster hardware on the problem until I saturate the
>> network, no matter which OS I'm using.
>>
>
> Why would you waste serious money on extra hardware to overcome OS
> limitations instead of just using a more efficient OS for the task at hand ?

Because you already have the software for that OS?

>>>> Some software might be able to squeeze more out of the same hardware,
>>>> but just spin up faster hardware, and you'll get higher throughput.
>>>> But any system will basically just be limited by the speed of the
>>>> network hardware, if that item is fixed. You can't go above that. But
>>>> there are no reasons why you wouldn't be able to get to that point.
>>>
>>> You are forgetting the overhead of the drivers and any underlying O/S
>>> code that has to be called to do it.
>>
>> Irrelevant. That just means I need to throw some more hardware on the
>> problem. Not that the OS somehow prevents me from achieving a higher
>> throughput.
>>
>
> Why would you waste serious money on extra hardware to overcome OS
> limitations instead of just using a more efficient OS for the task at hand ?

Because you already have the software for that OS?

>>>> These are the kind of questions that sometimes make me wonder if you
>>>> know anything at all about computers. But then you do some other posts
>>>> which clearly demonstrate that you do understand some stuff.
>>>
>>> I think it was just a subtle poke in the ribs for VMS.
>>
>> I think it was a display of complete ignorance on behalf of Simon.
>> Possibly just not thinking before writing. Who knows...?
>>
>
> That's seriously rich coming from you Johnny. Seriously rich.

I certainly am guilty of posting before thinking sometimes. That don't
make your question this time any less broken.

Johnny

On 2023-08-30 14:52, Simon Clubley wrote:
> On 2023-08-29, Johnny Billquist <bqt@softjar.se> wrote:
>> On 2023-08-29 19:25, Simon Clubley wrote:
>>> On 2023-08-29, Single Stage to Orbit <alex.buell@munted.eu> wrote:
>>>> On Thu, 1970-01-01 at 00:00 +0000, Simon Clubley wrote:
>>>>>> Very much FreeBSD here for some years, after decades first with
>>>>>> dec,
>>>>>> then Sun. Forms the basic of at least some proprietary offerings,
>>>>>> as
>>>>>> well as millions of embedded devices. Linux is still a unix,
>>>>>> and runs the majority of web sites of the world, so if anything,
>>>>>> unix has won the os wars...
>>>>>>
>>>>>
>>>>> Yes, very much so. (And I can't believe Arne thinks the *BSDs have no
>>>>> serious users... :-) ).
>>>>
>>>> Netflix picked FreeBSD as it could chuck out data at 400GB/s. Linux was
>>>> not even close.
>>>
>>> 10 years from now (assuming the economy hasn't collapsed by then :-) ):
>>>
>>> 400GB/s ??? Is that all ??? Amateurs!!! :-)
>>
>> Well. I have this friend of mine, who installed 40 Gb/s at his moms
>> place in 2007...
>>
>>> On a more serious note, I wonder what the maximum rate VMS is capable
>>> of emitting data at if it was using the fastest network hardware
>>> available.
>>
>> What a weird question. VMS in itself don't have any limits. It's all
>> always just about the hardware.
>
> As you have just been educated by multiple people, this statement is
> nonsense and shows you have no understanding of the tradeoffs involved
> in OS design and how those tradeoffs have changed over time.

Ok. I challenge you to point out what the limit is in VMS here then. Is
it some check deep down in the code that basically says:

if (thoughput > limit) sleep(sometime);

to make sure we don't get above that limit? Or what is it?

> You have also just quoted a message above that says two very conceptually
> similar designs (Linux and FreeBSD) still manage to have very different
> performance outcomes.

Both systems can push whatever network card/interface to its limit if
you throw enough resources at it. Just as VMS can. Or RSX for that
matter. This really should not be hard to understand.

>> Some software might be able to squeeze more out of the same hardware,
>> but just spin up faster hardware, and you'll get higher throughput. But
>> any system will basically just be limited by the speed of the network
>> hardware, if that item is fixed. You can't go above that. But there are
>> no reasons why you wouldn't be able to get to that point.
>>
>
> Why do you say that ? There will always be OS overheads. The only question
> is how large are those overheads ?

Yes. And that was not the question. Maybe you should go back and check
what question you actually wrote.

>> These are the kind of questions that sometimes make me wonder if you
>> know anything at all about computers. But then you do some other posts
>> which clearly demonstrate that you do understand some stuff.
>>
>
> The problem with you Johnny is that you appear to be a very arrogant
> person who assumes that if you don't understand what a person is
> saying, then you assume that person is talking nonsense.

Well. Since you were/are talking nonsense here, what am I do to?

Johnny

On 2023-08-30 15:02, Dan Cross wrote:
> In article <uclgav$q0b$1@news.misty.com>,
> Johnny Billquist <bqt@softjar.se> wrote:
>> On 2023-08-29 19:25, Simon Clubley wrote:
>> [snip]
>>> On a more serious note, I wonder what the maximum rate VMS is capable
>>> of emitting data at if it was using the fastest network hardware
>>> available.
>>
>> What a weird question. VMS in itself don't have any limits. It's all
>> always just about the hardware.
>> Some software might be able to squeeze more out of the same hardware,
>> but just spin up faster hardware, and you'll get higher throughput. But
>> any system will basically just be limited by the speed of the network
>> hardware, if that item is fixed. You can't go above that. But there are
>> no reasons why you wouldn't be able to get to that point.
>
> I don't really understand this line of reasoning. At some point
> you're running on the fastest hardware available; at that point,
> what do you do? It seems perfectly reasonable to try and
> quantify the overhead due to the software stack, and if possible
> to optimize it, particularly if your chosen software platform
> can't saturate the NIC at line rate. And even if it can, if it
> requires all of the CPU resources on your machine to do so, then
> you're starving the platform for cycles that would otherwise go
> to the software that all of those hungry network clients want to
> talk to.

But that is a different question. We are not talking about maximising
the amount of work done over some time (with whatever unit of
measurement of work we might be using).

The question was "I wonder what the maximum rate VMS is capable of
emitting data at if it was using the fastest network hardware available"?

And the answer to that will end up - the speed data can be delivered by
that network hardware.

Now, if you instead want to talk about how cost efficient it might be,
or what the user experience for 100 users sitting trying to do
interactive editing on the machine at that same time, then we might have
more things to talk about. But that was not the question.

> "Throw more hardware at it" works well until it doesn't, and
> when that happens you've hit a wall that you have to get around
> some other way.

Well. Then ask a question about that.

Johnny

On Wednesday, August 30, 2023 at 4:58:45 PM UTC-4, Arne Vajhøj wrote:
> On 8/30/2023 4:53 AM, Johnny Billquist wrote:
> > Not sure how easy it is to dodge RMS under VMS. In RSX, you can just do
> > the QIOs to the ACP yourself and go around the whole thing, which makes
> > I/O way faster. Of course, since files still have this structure thing,
> > most of the time you are still going to have to pay for it somewhere.
> > But if you are happy with just raw disk blocks, the basic I/O do not
> > have near as much penalty. Admitted, the ODS-1 (as well as ODS-2)
> > structure have some inherent limitations that carry some cost as well.
> > So you could improve things some by doing some other implementation on
> > the file system level.
> > But mainly, no matter what the file system design is, you are still
> > going to have the pain of RMS, which is the majority of the cost. And
> > you'll never get away from this as long as you use VMS.
> SYS$QIO(W) for files works fine on VMS too.
>
> But a bit of a hassle to use.
>
> There are two alternative ways to to bypass RMS:
> * SYS$IO_PERFORM(W) - the "fast I/O" thingy
> * SYS$CRMPSC - mapping the file to memory
>
> Arne
Arne,

One can bypass RMS, but it is not RMS that is the inherent problem. In my experience, it is not so much using RMS, but using RMS poorly that is the source of most problems.

As I noted in another post in this thread, increasing buffer factors and block sizes often virtually eliminates "RMS" performance problems. File extensions are costly, extending files by large increments also reduces overhead, increasing performance.

Not solely an OpenVMS problem. Originally dealt with this problem on IBM's OS/360 when I was a high school student. Reduced the cost of a production job by multiple orders of magnitude by simply increasing a blocking factor from 1 to 100. Yield? Two orders of magnitude reduction in CPU time consumed..

The other day I needed to copy a bare partition on a linux system. When refreshing my recollection of dd, noted that dd, for historical reasons, copies one block at a time. Increased performance by reading/writing a megabyte at a time.

It is often not the depth of the stack that matters, but how often one traverses the stack from top to bottom.

- Bob Gezelter, http://www.rlgsc.com

On Wednesday, August 30, 2023 at 11:00:38 AM UTC-4, Bob Gezelter wrote:
> With regards to mass storage, I will note that using recent generation HPE storage arrays and rx2800 i6 servers, I did not find it difficult to push mass storage load to the point that the storage controllers were "heavily" loaded and not "amused".
>
> To make the storage controllers happy, I had to throttle things back a bit.

I'm curious as to what the maximum real-world throughput of those arrays
is. And also what was making the storage controllers unhappy.

The 1.8GByte/sec mass storage write performance I mentioned upthread is
pretty much happening on "parts bin"" hardware without any tuning.

The 128TB file servers I was building 7.5 years ago (yikes!) do 700Mbyte/sec
I/O (averaged over a 24-hour period) and they cost under $3000 (not includ-
ing bare 8TB disk drives) to build with new parts back then.

I'm wondering how that compares with what HPE is doing. Obviously one
can build all-flash arrays of incredible speed (and price 8-) or low-cost arrays
using low-performance shingled drives.

On Tuesday, August 29, 2023 at 10:25:31 AM UTC-7, Simon Clubley wrote:

(snip)

> 400GB/s ??? Is that all ??? Amateurs!!! :-)
> On a more serious note, I wonder what the maximum rate VMS is capable
> of emitting data at if it was using the fastest network hardware
> available.
I am not sure what hardware can do now.

Traditionally, Ethernet was much faster than processors, such that the
shared media could handle the load.

That is less obvious now, but a 400Gb/s network doesn't mean that one host
can go that fast.

Otherwise, there are many stories of running old OS in emulation on modern
hardware, running into problems that never would have occurred years ago.

One is that emulators often do synchronous I/O. The I/O interrupt occurs almost
immediately, as seen by the OS. Some OS assume that there is time in between.

It is, then, possible that surprises will be found when running I/O at higher speed.

It might be useful to say which host architecture you were asking about.
I am sure no-one thought about 400Gb/s Ethernet for VAX.

On Thursday, August 31, 2023 at 12:35:05 AM UTC-4, gah4 wrote:
> On Tuesday, August 29, 2023 at 10:25:31 AM UTC-7, Simon Clubley wrote:
>
> (snip)
> > 400GB/s ??? Is that all ??? Amateurs!!! :-)
> > On a more serious note, I wonder what the maximum rate VMS is capable
> > of emitting data at if it was using the fastest network hardware
> > available.
> I am not sure what hardware can do now.
>
> Traditionally, Ethernet was much faster than processors, such that the
> shared media could handle the load.
>
> That is less obvious now, but a 400Gb/s network doesn't mean that one host
> can go that fast.
>
> Otherwise, there are many stories of running old OS in emulation on modern
> hardware, running into problems that never would have occurred years ago.
>
> One is that emulators often do synchronous I/O. The I/O interrupt occurs almost
> immediately, as seen by the OS. Some OS assume that there is time in between.
>
> It is, then, possible that surprises will be found when running I/O at higher speed.
>
> It might be useful to say which host architecture you were asking about.
> I am sure no-one thought about 400Gb/s Ethernet for VAX.
gah4,

Ethernet (and other CSMA/CD networking approaches) in a configuration with more than a single, full duplex connection connecting two adapters are essentially limited to a maximum effective utilization of 30% before contention backoff becomes unacceptable.

Active switches, as opposed to hubs, can increase this threshold as each cable is physically connected to an interface and a switch port.

Instant I/O completion has uncovered all manner of bugs and deficiencies over the years. Most such problems are at the driver level, where a driver fails to ensure that data structures are correctly prepared for an immediate interrupt when the "Go" bit is triggered. On occasion, an application has been written to run very I/O bound with the presumption that I/O delay will slow it down, e.g., OpenVMS COPY, RSX-11 PIP, linux dd. Combine a greedy application with a high dispatch priority and voila, monopolized machine for the duration. On OpenVMS, put a process at a base priority above most users, say 6-7, and run a large COPY from one instantaneous mass storage device to another. Other users see an effectively dead machine.

Virtual machine-level synchronous I/O is a gargantuan performance impediment on many levels. My aforementioned Ph.D. dissertation on I/O performance, one of the main conclusions was that unnecessary serialization at any point in the I/O stack was a performance cliff. Serialization obstructs lower-level optimization and workload management. The mathematics could not be more definitive.

- Bob Gezelter, http://www.rlgsc.com

On 2023-08-30, Bob Gezelter <gezelter@rlgsc.com> wrote:
>
> The other day I needed to copy a bare partition on a linux system. When refreshing my recollection of dd, noted that dd, for historical reasons, copies one block at a time. Increased performance by reading/writing a megabyte at a time.
>

:-)

Adjusting dd command line options for efficiency is one of the first things
you learn to do when you start using dd for "serious" things. :-)

I learnt that back in the late 1990s when it comes to dd...

BTW, did you know you can send dd a signal to tell you how far along
in the copy process it is and to tell you how fast the copy is going ?

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 2023-08-31, gah4 <gah4@u.washington.edu> wrote:
> On Tuesday, August 29, 2023 at 10:25:31?AM UTC-7, Simon Clubley wrote:
>
> (snip)
>
>> 400GB/s ??? Is that all ??? Amateurs!!! :-)
>
>> On a more serious note, I wonder what the maximum rate VMS is capable
>> of emitting data at if it was using the fastest network hardware
>> available.
>
> It might be useful to say which host architecture you were asking about.
> I am sure no-one thought about 400Gb/s Ethernet for VAX.
>

My question was in the context of the comment that FreeBSD was chosen
over Linux for pushing data at a huge rate through some _very_ high
speed network interfaces.

Given that VMS is known for having slower I/O performance, I was wondering
just _how_ much slower than Linux it would have been on the same hardware.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.