If the emulated executive mode is a major reason for the kernel overheads
we are seeing on x86-64 VMS, what would be involved in moving RMS from
executive mode directly into kernel mode ?

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 2023-05-02 15:24, Simon Clubley wrote:
> If the emulated executive mode is a major reason for the kernel overheads
> we are seeing on x86-64 VMS, what would be involved in moving RMS from
> executive mode directly into kernel mode ?

I have a hard time seeing why it would cost very much to deal with the
different modes. I can definitely believe that code hasn't been
optimized in general, though.

Johnny

On 2023-05-04, Johnny Billquist <bqt@softjar.se> wrote:
> On 2023-05-02 15:24, Simon Clubley wrote:
>> If the emulated executive mode is a major reason for the kernel overheads
>> we are seeing on x86-64 VMS, what would be involved in moving RMS from
>> executive mode directly into kernel mode ?
>
> I have a hard time seeing why it would cost very much to deal with the
> different modes.

In which case Johnny, you have no clue about the overheads involved here
when emulating a processor mode in software.

I also suspect you have no clue either about why I keep asking about kernel
overhead on AMD systems and what the significance of that question is.

> I can definitely believe that code hasn't been
> optimized in general, though.
>

At best, that's only a small contribution to the numbers that Mark
is seeing. Also, don't forget that the E/S emulation code is new code,
not ported existing code.

A percentage is a ratio and unless there's any evidence that better
compilers would affect the code running on one side of that ratio much
more than the other side, then the overall percentage should still remain
roughly the same even with better compilers.

I also note that VSI have not responded to Mark's question after the
better part of a week. That's not a good look.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

> At best, that's only a small contribution to the numbers that Mark
> is seeing. Also, don't forget that the E/S emulation code is new code,
> not ported existing code.
>

Hence it being referred to as potentially unoptimized. First revision and
generation code almost never has a fraction of the performance gains
that future generations will. Then there's also the fact that, well, when trying
to write something correct, you're not worried about performance or optimization
at first - that can come later, so you do it in the cleanest 'correct' way, then get
down into what can be shortened, optimized, restructured, merged, etc.

> A percentage is a ratio and unless there's any evidence that better
> compilers would affect the code running on one side of that ratio much
> more than the other side, then the overall percentage should still remain
> roughly the same even with better compilers.

Compilers aren't magic. Rewriting functions to act specific ways can do
a lot compilers can't.

I just squeezed about 50 seconds of runtime off a current project by
changing the way a specific data set is handled in an internal function -
but outside that function everything still appears and functions exactly the
same - including to already compiled binary code that links/calls it as the
interfaces have not changed.

Anything, of course, that

> I also note that VSI have not responded to Mark's question after the
> better part of a week. That's not a good look.
> Simon.

Not the largest company in the world, and for all we know, they may still be
analyzing the usage/functionality and reported information/reproducing it.

That is, if they've even got the spare cycles to throw at it compared to what's
already on deck from paying customers.

On Tuesday, May 2, 2023 at 9:24:03 AM UTC-4, Simon Clubley wrote:
> If the emulated executive mode is a major reason for the kernel overheads
> we are seeing on x86-64 VMS, what would be involved in moving RMS from
> executive mode directly into kernel mode ?

Nah, have a link time option to run RMS in user mode! I'm mostly serious.
Moving it to kernel mode you still need to probe the RABs, FABs, XABs and anything those point to.
The protection exec or kernel mode offers is minimal. It makes sure that broken user mode doesn't stomp on data block buffers and corrupts the underlying file impacting downstream usage of those files. Possibly worse, in user mode buggy programs could impact other, shared, users of those files, specially when global buffers are involved.
Well tough luck, write better program, test them better!
Exec mode RMS might help the testing but when is the last time your programs accidently tried to touch an rms buffer and got an accvio?
It (exec or kernel) mode only protects against accidental data corruption (bugs).
It doesn't protect against malicious corruption as anyone with write access can overwrite data.
Regular file protection handles that. Only allow trusted users, or subsystems holding the right identifier, access to critical files.
Admittedly RMS does use CMKRNL for some odd things like borrowing/stealing ENQLM to get out of a tight spot, but in general user mode could be just fine.

fwiw ... back in the day (30 years) ago, while in VMS Engineering I had access to usermode RMS for debugging purposes. It did exists at the time with minimal restrictions. It was rarely - if ever - used. Mostly we worked of crashdumps and 'live' ANAL/SYSTEM to look at the data structures to debug RMS.

Hein.

On 5/4/2023 10:24 AM, Hein RMS van den Heuvel wrote:
> On Tuesday, May 2, 2023 at 9:24:03 AM UTC-4, Simon Clubley wrote:
>> If the emulated executive mode is a major reason for the kernel overheads
>> we are seeing on x86-64 VMS, what would be involved in moving RMS from
>> executive mode directly into kernel mode ?
>
> Nah, have a link time option to run RMS in user mode! I'm mostly serious.
> Moving it to kernel mode you still need to probe the RABs, FABs, XABs and anything those point to.
> The protection exec or kernel mode offers is minimal. It makes sure that broken user mode doesn't stomp on data block buffers and corrupts the underlying file impacting downstream usage of those files. Possibly worse, in user mode buggy programs could impact other, shared, users of those files, specially when global buffers are involved.
> Well tough luck, write better program, test them better!
> Exec mode RMS might help the testing but when is the last time your programs accidently tried to touch an rms buffer and got an accvio?
> It (exec or kernel) mode only protects against accidental data corruption (bugs).
> It doesn't protect against malicious corruption as anyone with write access can overwrite data.
> Regular file protection handles that. Only allow trusted users, or subsystems holding the right identifier, access to critical files.
> Admittedly RMS does use CMKRNL for some odd things like borrowing/stealing ENQLM to get out of a tight spot, but in general user mode could be just fine.
>
> fwiw ... back in the day (30 years) ago, while in VMS Engineering I had access to usermode RMS for debugging purposes. It did exists at the time with minimal restrictions. It was rarely - if ever - used. Mostly we worked of crashdumps and 'live' ANAL/SYSTEM to look at the data structures to debug RMS.
>
> Hein.
>

I have to agree with Hein on this subject. I don't see what inner modes do for
accessing and updating data.

I've got an old database product, similar in some ways to RMS, that runs in user
mode. The only inner modes used are for system wide locks, so that required
privs are not assigned to the users. Works just fine.

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

On 5/4/2023 8:15 AM, Simon Clubley wrote:
> On 2023-05-04, Johnny Billquist <bqt@softjar.se> wrote:
>> I can definitely believe that code hasn't been
>> optimized in general, though.
>
> At best, that's only a small contribution to the numbers that Mark
> is seeing.

Sure about that?

The difference between LLVM x86-64 default and -O3 on other platforms
for CPU intensive integer math is a factor 5.

VMS may be very different from "CPU intensive integer math", but
I would not start with an assumption that it would only be a small
contribution.

> A percentage is a ratio and unless there's any evidence that better
> compilers would affect the code running on one side of that ratio much
> more than the other side, then the overall percentage should still remain
> roughly the same even with better compilers.

Are you sure about that?

The WASD download site claims:

<quote>
All the x86-64 object modules have been native-compiled using VSI C
X7.4-726 (GEM 50X23) on OpenVMS x86_64 V9.2
</quote>

That gives me the impression that the application code (which
must be approx. the code running in user mode) is native compiled
aka optimized. While VMS (code running in inner modes) is not.

Arne

On 5/4/2023 8:15 AM, Simon Clubley wrote:
> I also note that VSI have not responded to Mark's question after the
> better part of a week. That's not a good look.

VSI has to prioritize.

They have ISV's and customers waiting for tools and middleware
to kick off migration projects.

Getting those migration projects underway is important
for VSI.

Based on what has been posted so far then the current
performance on x86-64 is more than fine for the migration
project.

It seems even likely that the current performance on x86-64
will be good enough for production with current applications - if
performance was OK on Alpha and Itanium, then it will be OK
on x86-64.

The day VMS starts competing with Linux and the comparison is
no longer XYZ running on VMS x84-64 vs XYZ running on VMS Alpha
or Itanium but XYZ running on VMS x86-64 vs XYZ running on
Linux x86-64, then the efficiency becomes important for VSI.

So it is an important issue but it is not an urgent issue.

If VSI is lucky then the problem goes away with the native
build of VMS x86-64.

If not then they will need to measure where the time
is actually spent and fix what that is.

Guesses on where that time could be spent make for
great discussions on usenet, but for actually doing
something measuring will be needed.

Arne

On 2023-05-02 13:24:00 +0000, Simon Clubley said:

> If the emulated executive mode is a major reason for the kernel
> overheads we are seeing on x86-64 VMS, what would be involved in moving
> RMS from executive mode directly into kernel mode ?

If you're going to blue-sky this, move RMS and the XQP into a user-mode
process, preferably with zero-copy buffering.

Whether a solitary process or replicated processes would depend on
complexity, aggregate system load, and available cores.

Yeah, I know, I'm making a mach-ery of the existing two-by-two version
of the four-mode design.

But pragmatically, VSI has a few billion projects in the queue ahead of
re-architecting the RMS and XQP giblets.

--
Pure Personal Opinion | HoffmanLabs LLC

On 04/05/2023 21:57, Stephen Hoffman wrote:
> On 2023-05-02 13:24:00 +0000, Simon Clubley said:
>
>> If the emulated executive mode is a major reason for the kernel
>> overheads we are seeing on x86-64 VMS, what would be involved in
>> moving RMS from executive mode directly into kernel mode ?
>
> If you're going to blue-sky this, move RMS and the XQP into a user-mode
> process, preferably with zero-copy buffering.
>
> Whether a solitary process or replicated processes would depend on
> complexity, aggregate system load, and available cores.
>
> Yeah, I know, I'm making a mach-ery of the existing two-by-two version
> of the four-mode design.
>
> But pragmatically, VSI has a few billion projects in the queue ahead of
> re-architecting the RMS and XQP giblets.
>
>

I am sure Hein is available

--
Chris

> I just squeezed about 50 seconds of runtime off a current project
> [...]

As they say on "More or Less", is that a big number?

"50 seconds" out of a minute? An hour? A day? A month? ...

http://www.isthatabignumber.com/

On Friday, May 5, 2023 at 1:14:40 AM UTC-4, Steven Schweda wrote:
> > I just squeezed about 50 seconds of runtime off a current project
> > [...]
>
> As they say on "More or Less", is that a big number?
>
> "50 seconds" out of a minute? An hour? A day? A month? ...
>
> http://www.isthatabignumber.com/

50 seconds out of a 57 second runtime for that function. :)

Den 2023-05-05 kl. 07:41, skrev Gary Sparkes:
> On Friday, May 5, 2023 at 1:14:40 AM UTC-4, Steven Schweda wrote:
>>> I just squeezed about 50 seconds of runtime off a current project
>>> [...]
>>
>> As they say on "More or Less", is that a big number?
>>
>> "50 seconds" out of a minute? An hour? A day? A month? ...
>>
>> http://www.isthatabignumber.com/
>
> 50 seconds out of a 57 second runtime for that function. :)

That is not bad, for a single run of that function.

But still, and in line with the question from Steven, did that
function run once each minute? Or once each day/week/month?

On 2023-05-04, Stephen Hoffman <seaohveh@hoffmanlabs.invalid> wrote:
> On 2023-05-02 13:24:00 +0000, Simon Clubley said:
>
>> If the emulated executive mode is a major reason for the kernel
>> overheads we are seeing on x86-64 VMS, what would be involved in moving
>> RMS from executive mode directly into kernel mode ?
>
> If you're going to blue-sky this, move RMS and the XQP into a user-mode
> process, preferably with zero-copy buffering.
>
> Whether a solitary process or replicated processes would depend on
> complexity, aggregate system load, and available cores.
>
> Yeah, I know, I'm making a mach-ery of the existing two-by-two version
> of the four-mode design.
>

Really ?

I thought you might have been making a mica-ery of the situation. :-)

(After posting, I remembered that the released Prism documents talk about
MICA running its version of RMS in user mode.)

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 2023-05-04, Arne Vajhøj <arne@vajhoej.dk> wrote:
>
> The WASD download site claims:
>
><quote>
> All the x86-64 object modules have been native-compiled using VSI C
> X7.4-726 (GEM 50X23) on OpenVMS x86_64 V9.2
></quote>
>
> That gives me the impression that the application code (which
> must be approx. the code running in user mode) is native compiled
> aka optimized. While VMS (code running in inner modes) is not.
>

Interesting, thanks. So, maybe yes, or maybe no. Or it could simply be
limitations in the E/S emulation technique coming to light, which
compilers may not be able to do anything about.

It will be nice to get a proper production release of VMS so that we
know for sure, but it doesn't look like that is going to happen this
year. :-(

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 2023-05-04, Chris Townley <news@cct-net.co.uk> wrote:
> On 04/05/2023 21:57, Stephen Hoffman wrote:
>> On 2023-05-02 13:24:00 +0000, Simon Clubley said:
>>
>>> If the emulated executive mode is a major reason for the kernel
>>> overheads we are seeing on x86-64 VMS, what would be involved in
>>> moving RMS from executive mode directly into kernel mode ?
>>
>> If you're going to blue-sky this, move RMS and the XQP into a user-mode
>> process, preferably with zero-copy buffering.
>>
>> Whether a solitary process or replicated processes would depend on
>> complexity, aggregate system load, and available cores.
>>
>> Yeah, I know, I'm making a mach-ery of the existing two-by-two version
>> of the four-mode design.
>>
>> But pragmatically, VSI has a few billion projects in the queue ahead of
>> re-architecting the RMS and XQP giblets.
>>
>>
>
> I am sure Hein is available
>

Who is he ? :-)

On a more serious note, I've received private email pointing out that
there may be an issue with PPFs if running RMS in user mode.

I also wonder how global buffers might be affected by user-mode RMS.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 2023-05-05 12:22:56 +0000, Simon Clubley said:

> On a more serious note, I've received private email pointing out that
> there may be an issue with PPFs if running RMS in user mode.

User mode in a different process is (ideally) immune to user mode
shenanigans in a nefarious process, and anything that needs to be
protected within the local process can be in a mode inaccessible to
user mode.

> I also wonder how global buffers might be affected by user-mode RMS.

Either the buffers are mapped and accessible in virtual memory, or not.

(Reducing the numbers of buffer copies has not been a focus for
OpenVMS. There have been previous discussions here concerning the
network stack latency, too.
https://groups.google.com/g/comp.os.vms/c/AAkAPK5_CZ4/m/2RkjDkm2AQAJ )

And there's an open question around how beneficial RMS global buffers
might be in practice, given current SSD transfer speeds are approaching
those of DDR3 SDRAM memory. Upgrades from HDDs to SSDs can hide a whole
lot of app performance issues.

But again, VSI is not in a position to re-architect OpenVMS now, nor in
the forseeable future.

--
Pure Personal Opinion | HoffmanLabs LLC

Chris Townley>> I am sure Hein is available

Nah, happily retired.

On Friday, May 5, 2023 at 6:19:10 PM UTC-4, Stephen Hoffman wrote:
> On 2023-05-05 12:22:56 +0000, Simon Clubley said:
>
> > On a more serious note, I've received private email pointing out that
> > there may be an issue with PPFs if running RMS in user mode.
> User mode in a different process is (ideally) immune to user mode

PPFs would be challenging. Executing RMS in a slave process - dedicated or shared - could solve a lot.
One could perhaps also think of a hybrid solution where usermode is used wherever possible but when for example the PPF flag is seen in the IFI/ISI a dive into the normal exec mode RMS is made .

> > I also wonder how global buffers might be affected by user-mode RMS.
> Either the buffers are mapped and accessible in virtual memory, or not.

Global buffers can be mapped in usermode, the penalty for a bad actor would get magnified as it could kill an unsuspecting other process, which is why I also indicated them as potentially challenging. There is currently also a global buffer lock optimization implemented with system lock. That particular optimization is an enormous win in simple 'get next' processing and cache hits. without that (pre 7.3), every entry into a buffer would at lease convert a lock to up, and down again on exit. with the optimization the system lock guarantees that the right version of bucket data is in the buffer. That is a massive locking (and thus kernel mode or even MPsync time) win.

Hoff> And there's an open question around how beneficial RMS global buffers
Hoff> might be in practice, given current SSD transfer speeds are approaching

Indeed. But that lock optimization I just mentioned is nowadays more important with the caching XFC and faster and faster (caching) IO, for that alone global buffers as a big win.

Hoff> But again, VSI is not in a position to re-architect OpenVMS now, nor in the forseeable future.

Correct.

Also, we just don't know yet where we are whether there is a problem big enough to worry about.
Brute power may well come to the rescue. Let's wait and see the results when X86 VMS is build with optimizing compilers before speculating about the need for a solution.

Some places (C-RTL, COBRTL, SORT) already use their own usermode code avoiding SYS$GET, SYS$PUT for simple unshared sequential files for that very performance reason of avoiding the CMEXEC and PROBES. But for those simple files that overhead is disproportional as they pretty much deal with 1 known buffer of data and a predictable offset in that buffer. Once one gets into indexed file territory the actual work grows a lot often involving multiple buffer visits and buffer searches. With that the the entrance price into the service becomes relatively much less important than work performed for the service.

Cheers,
Hein.

Den 2023-05-06 kl. 05:26, skrev Hein RMS van den Heuvel:
>
> Chris Townley>> I am sure Hein is available
>
> Nah, happily retired.
>
> On Friday, May 5, 2023 at 6:19:10 PM UTC-4, Stephen Hoffman wrote:
>> On 2023-05-05 12:22:56 +0000, Simon Clubley said:
>>
>>> On a more serious note, I've received private email pointing out that
>>> there may be an issue with PPFs if running RMS in user mode.
>> User mode in a different process is (ideally) immune to user mode
>
> PPFs would be challenging. Executing RMS in a slave process - dedicated or shared - could solve a lot.
> One could perhaps also think of a hybrid solution where usermode is used wherever possible but when for example the PPF flag is seen in the IFI/ISI a dive into the normal exec mode RMS is made .
>
>>> I also wonder how global buffers might be affected by user-mode RMS.
>> Either the buffers are mapped and accessible in virtual memory, or not.
>
> Global buffers can be mapped in usermode, the penalty for a bad actor would get magnified as it could kill an unsuspecting other process, which is why I also indicated them as potentially challenging. There is currently also a global buffer lock optimization implemented with system lock. That particular optimization is an enormous win in simple 'get next' processing and cache hits. without that (pre 7.3), every entry into a buffer would at lease convert a lock to up, and down again on exit. with the optimization the system lock guarantees that the right version of bucket data is in the buffer. That is a massive locking (and thus kernel mode or even MPsync time) win.
>
>
> Hoff> And there's an open question around how beneficial RMS global buffers
> Hoff> might be in practice, given current SSD transfer speeds are approaching
>
> Indeed. But that lock optimization I just mentioned is nowadays more important with the caching XFC and faster and faster (caching) IO, for that alone global buffers as a big win.
>
> Hoff> But again, VSI is not in a position to re-architect OpenVMS now, nor in the forseeable future.
>
> Correct.
>
> Also, we just don't know yet where we are whether there is a problem big enough to worry about.
> Brute power may well come to the rescue. Let's wait and see the results when X86 VMS is build with optimizing compilers before speculating about the need for a solution.
>
> Some places (C-RTL, COBRTL, SORT) already use their own usermode code avoiding SYS$GET, SYS$PUT for simple unshared sequential files for that very performance reason of avoiding the CMEXEC and PROBES. But for those simple files that overhead is disproportional as they pretty much deal with 1 known buffer of data and a predictable offset in that buffer. Once one gets into indexed file territory the actual work grows a lot often involving multiple buffer visits and buffer searches. With that the the entrance price into the service becomes relatively much less important than work performed for the service.
>
> Cheers,
> Hein.
>
>

About RMS Global Buffers. What files are these usually used for?
I have got an (maybe wrong) impression that it is mostly indexed
files "databases".

As an example, Rdb is not using RMS for the data/storage. It is
used for some Rdb files like the backup files and "export" files,
but for that I do not see can gain much from faster global buffers...

On 5/6/2023 6:39 AM, Jan-Erik Söderholm wrote:
> About RMS Global Buffers. What files are these usually used for?
> I have got an (maybe wrong) impression that it is mostly indexed
> files "databases".
>
> As an example, Rdb is not using RMS for the data/storage. It is
> used for some Rdb files like the backup files and "export" files,
> but for that I do not see can gain much from faster global buffers...

I am far from an expert in RMS (unlike Hein), but my very basic
understanding is that global buffers work for both sequential
and index-sequential files.

But to provide value in the form of better performance then:
- the same file need to be accessed by multiple processes
- the access need to go through RMS

Which may have been a common scenario 35 years ago, but
not so much today.

An indexed-sequential file being updated by multiple
processes running some Cobol/Basic/Pascal application
designed 35 years ago is probably the most likely
combo to match those criteria.

Arne

On Saturday, May 6, 2023 at 10:44:58 AM UTC-4, Arne Vajhøj wrote:
> On 5/6/2023 6:39 AM, Jan-Erik Söderholm wrote:
> > About RMS Global Buffers. What files are these usually used for?
> > I have got an (maybe wrong) impression that it is mostly indexed
> > files "databases".

Correct.

> But to provide value in the form of better performance then:
> - the same file need to be accessed by multiple processes
> - the access need to go through RMS

Correct - with write sharing, so not useful for say SYLOGIN.COM.

One more requirement for performance gain through global buffers is repeated access.
You also want to file to remain open as t5he established global buffers are dropped with the last accessor closing it.
One may go as far as creating a helper program to keep files open.

As an example repeated indexed file inserts would go through each index tree top to bottom and the top two layers can often be cached nicely.
With the 7.3 global buffer syslck method even a single process performing sequential gets to a shared file can benefit a lot.
Without global buffer each get would unlock the last record, lock the bucket, locate the next record, lock that, return it, unlock the bucket.
With Global buffers the bucket lock on happens on first entry into the bucket and thus saves almost half the lock activity.

One more easy global buffer win is cluster wide sharing. Whereas with single server access the XFC does a great job caching file, better than Global Buffers really as it is more dynamic, the XFC caching falls apart with cross cluster write shared access any write intend for a file from the 'other' node will flush the buffers for that file everywhere else. The other node then starts building its cache for the file only to tapped on the should for a flush a couple of buffers into the process. RMS Global buffer to NOT have this issue as each buckets is protected from being stale through a per-bucket lock, not per file.

And yes, the classic 'thick', 'cobol' application is a prime candidate but think also about SYSUAF.DAT, RIGHTSLIST.DAT,...

Cheers,
Hein.

On 5/6/2023 10:43 AM, Arne Vajhøj wrote:
> On 5/6/2023 6:39 AM, Jan-Erik Söderholm wrote:
>> About RMS Global Buffers. What files are these usually used for?
>> I have got an (maybe wrong) impression that it is mostly indexed
>> files "databases".
>>
>> As an example, Rdb is not using RMS for the data/storage. It is
>> used for some Rdb files like the backup files and "export" files,
>> but for that I do not see can gain much from faster global buffers...
>
> I am far from an expert in RMS (unlike Hein), but my very basic
> understanding is that global buffers work for both sequential
> and index-sequential files.

I believe that is true.

> But to provide value in the form of better performance then:
> - the same file need to be accessed by multiple processes
> - the access need to go through RMS

Yes.

> Which may have been a common scenario 35 years ago, but
> not so much today.

It's still very common today.

However, global buffers are no longer very helpful. Why? Because of the
caching added to VMS somewhere around V7 or V8.

The CODIS application has one file that just about every process has open. One
would consider such a scenario to benefit from global buffering. However, DAS
(the database product) didn't offer global buffers. When most of your data is
in cache, things can get rather fast. What was observed was performance quite
fast with caching, and a real slog without.

> An indexed-sequential file being updated by multiple
> processes running some Cobol/Basic/Pascal application
> designed 35 years ago is probably the most likely
> combo to match those criteria.

And caching has satisfied that need.

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

On Friday, May 5, 2023 at 4:01:08 AM UTC-4, Jan-Erik Söderholm wrote:
> But still, and in line with the question from Steven, did that
> function run once each minute? Or once each day/week/month?

Constantly. Data ingestion/processing run each loop. Total main
runtime to results is now 20 seconds at most for a single iteration
of the entire program, but it is constantly pulling that data/running
it and updating the display each time. So it was a *massive*
improvement - leaning more on native data structures than the
textbook clean implementation. Can't really say much more
beyond that as it's a current employer project, but just one of
the many examples of optimization that should be thought of
and/or considered instead of just relying on the compiler.

VSI probably has a lot of big easy wins like this coming in the
future, but went with a "textbook" / "robust" POC-level implementation
first. I suspect there are possibly a lot of gains to be had, from
source-level optimization/testing/profiling/etc.

On Friday, May 5, 2023 at 8:17:02 AM UTC-4, Simon Clubley wrote:
> It will be nice to get a proper production release of VMS so that we
> know for sure, but it doesn't look like that is going to happen this
> year. :-(
> Simon.

Isn't that what V9.2 is? :)

On 5/6/2023 12:51 PM, Dave Froble wrote:
> On 5/6/2023 10:43 AM, Arne Vajhøj wrote:
>> On 5/6/2023 6:39 AM, Jan-Erik Söderholm wrote:
>>> About RMS Global Buffers. What files are these usually used for?
>>> I have got an (maybe wrong) impression that it is mostly indexed
>>> files "databases".
>>>
>>> As an example, Rdb is not using RMS for the data/storage. It is
>>> used for some Rdb files like the backup files and "export" files,
>>> but for that I do not see can gain much from faster global buffers...
>>
>> I am far from an expert in RMS (unlike Hein), but my very basic
>> understanding is that global buffers work for both sequential
>> and index-sequential files.
>
> I believe that is true.
>
>> But to provide value in the form of better performance then:
>> - the same file need to be accessed by multiple processes
>> - the access need to go through RMS
>
> Yes.
>
>> Which may have been a common scenario 35 years ago, but
>> not so much today.
>
> It's still very common today.

> The CODIS application has one file that just about every process has
> open.  One would consider such a scenario to benefit from global
> buffering.  However, DAS (the database product) didn't offer global
> buffers.  When most of your data is in cache, things can get rather
> fast.  What was observed was performance quite fast with caching, and a
> real slog without.

In the IT world in general data access has moved to client
server models and to relational databases.

I believe that even though VMS still got a large number of
older applications running then it has also at least partly
moved.

> However, global buffers are no longer very helpful. Why? Because of
> the caching added to VMS somewhere around V7 or V8.

>> An indexed-sequential file being updated by multiple
>> processes running some Cobol/Basic/Pascal application
>> designed 35 years ago is probably the most likely
>> combo to match those criteria.
>
> And caching has satisfied that need.

That is a good point.

XFC (VMS 7.3) may have taken care of a lot of the
caching needs.

Arne