On 2023-11-04 18:06, Craig A. Berry wrote:
>
> On 11/4/23 6:11 AM, Johnny Billquist wrote:
>
>> I'm not sure I have ever understood why people think memory mapped
>> files would be faster than a QIO under VMS.
>
> I might've missed it but I haven't seen anyone say that. It's that
> global sections are faster than mailboxes. The I/O API may be a
> consideration but is secondary to the nature of the device.

I think I've seen plenty of suggestions that memory mapped I/O would be
a performance enhancer in VMS. Probably based on the fact that it is
under Unix.

>> With memory mapped I/O, what you essentially get is that I/O transfers
>> go directly from/to disk to user memory with a single operation. There
>> are no intermediate buffers, no additional copying. Which is what you
>> pretty much always have otherwise on Unix systems.
>>
>> However, a QIO under VMS is already a direct communication between the
>> physical memory and the device with no intermediate buffers,
>> additional copying or whatever, unless I'm confused (and VMS changed
>> compared to RSX here...).
>>
>> So how would memory mapped I/O be any faster? You basically cannot be
>> any faster than one DMA transfer. In fact, with memory mapped I/O, you
>> might be also hitting the page fault handling, and a reading in of a
>> full page, which might be more than you needed, causing some overhead
>> as well.
>
> I doubt that QIO with most device drivers is a simple as you make it out
> to be. Last I heard they use Intermediate Request Packets (IRPs) and
> other system resources in addition to the buffer space in the user
> program. And in any case, QIO is limited to transfer sizes that these
> days are considered pretty small.

I admit that I based this on how the QIO is processed in RSX. I wouldn't
expect VMS to do it much differently, but I haven't looked at the VMS code.

The IRP is basically the internal storage/representation of the QIO that
the kernel use, unless I'm confused. And yes, that have to be allocated.
(In RSX you have a bunch of these preallocated for efficiency). Size
limitation I can possibly see a bit of a thing with. I guess it's still 64K.

But that's pretty much it.

>> Also, what do $IO_PERFORM do, that could possibly make it faster than
>> QIO?
>>
>> I'm really curious about this topic...
>
> There's a pretty good explanation in the manual:
>
> https://docs.vmssoftware.com/vsi-openvms-programming-concepts-manual-volume-ii/#FAST_IO_PATH

Thanks. Very enlightning. So basically:
1. Preallocated IRPs.
2. Constantly locked memory, thus avoiding the unlock/lock bits.
3. Quick I/O completion.

It does make it sound like QIO in VMS would use an intermediate buffer.
Which is perhaps the most interesting bit. So IOPERFORM is closer to
what QIO does under RSX.

Thanks.

Johnny

On 2023-11-04 20:03, Craig A. Berry wrote:
> On 11/4/23 1:42 PM, Arne Vajhøj wrote:
>> On 11/4/2023 1:06 PM, Craig A. Berry wrote:
>>>
>>> On 11/4/23 6:11 AM, Johnny Billquist wrote:
>>>
>>>> I'm not sure I have ever understood why people think memory mapped
>>>> files would be faster than a QIO under VMS.
>>>
>>> I might've missed it but I haven't seen anyone say that. It's that
>>> global sections are faster than mailboxes. The I/O API may be a
>>> consideration but is secondary to the nature of the device.
>>
>> I guess that I as usual will have to plead guilty.
>>
>> I wrote:
>>
>> # The normal assumption regarding speed of disk IO would be that:
>> #
>> # RMS record IO ($GET and $PUT) < RMS block IO ($READ and $WRITE) <
>> # $QIO(W) < $IO_PERFORM(W) < memory mapped file
>> #
>> # (note that assumption and fact are spelled differently)
>>
>
>
> Actually, I'm guilty of forgetting I read that message.  I had in mind
> Jake's original problem of IPC, not disk I/O, when I responded to
> Johnny's remark about memory mapped files. For disk I/O, yes, it's
> almost certain that using virtual memory primitives to synchronize
> integral pages between disk and memory will be faster than any other I/O
> method; that's why pretty much every database product on every platform
> does it.

And that is what I am questioning. Why would memory mapped I/O to disk
be any faster than IO_PERFORM? After all, both cases will just be DMA
from disk to your memory without any intermediate copying.

With IO_PERFORM you need to issue the request to do a read. With memory
mapped I/O you need to do a system call to instead first tell which part
of the file you want mapped, and then hit a page fault to get the read
to happen.

Why pretty much every database product on every platform use memory
mapped I/O is most likely becase "every platform" is basically
Unix/Linux or Windows. In which case "normal" I/O always goes through
another layer of buffering and copying, making memory mapped I/O very
supierior. That isn't nearly the same situation as under VMS.

Johnny

On 2023-11-04 22:41, Dan Cross wrote:
> In article <ui64is$3go9i$1@dont-email.me>,
> Craig A. Berry <craigberry@nospam.mac.com> wrote:
>> On 11/4/23 1:42 PM, Arne Vajhøj wrote:
>>> On 11/4/2023 1:06 PM, Craig A. Berry wrote:
>>>>
>>>> On 11/4/23 6:11 AM, Johnny Billquist wrote:
>>>>
>>>>> I'm not sure I have ever understood why people think memory mapped
>>>>> files would be faster than a QIO under VMS.
>>>>
>>>> I might've missed it but I haven't seen anyone say that. It's that
>>>> global sections are faster than mailboxes. The I/O API may be a
>>>> consideration but is secondary to the nature of the device.
>>>
>>> I guess that I as usual will have to plead guilty.
>>>
>>> I wrote:
>>>
>>> # The normal assumption regarding speed of disk IO would be that:
>>> #
>>> # RMS record IO ($GET and $PUT) < RMS block IO ($READ and $WRITE) <
>>> # $QIO(W) < $IO_PERFORM(W) < memory mapped file
>>> #
>>> # (note that assumption and fact are spelled differently)
>>
>> Actually, I'm guilty of forgetting I read that message. I had in mind
>> Jake's original problem of IPC, not disk I/O, when I responded to
>> Johnny's remark about memory mapped files. For disk I/O, yes, it's
>> almost certain that using virtual memory primitives to synchronize
>> integral pages between disk and memory will be faster than any other I/O
>> method; that's why pretty much every database product on every platform
>> does it.
>
> Everyone starts out thinking that, but most are wrong:
> https://db.cs.cmu.edu/mmap-cidr2022/

*Thank you*. Finally someone who actually gives this some thought
instead of just repeating what everyone else says.

Johnny

On 2023-11-04 22:44, Arne Vajhøj wrote:
> On 11/4/2023 7:11 AM, Johnny Billquist wrote:
>> On 2023-11-03 15:08, Arne Vajhøj wrote:
>>> On 11/2/2023 9:02 PM, Jake Hamby (Solid State Jake) wrote:
>>>> I've become a little obsessed with the question of how well OpenVMS
>>>> performs relative to Linux inside a VM, under different conditions.
>>>> My current obsession is the libuv library which provides a popular
>>>> async I/O abstraction layer implemented for all the different flavors
>>>> of UNIX that have async I/O, as well as for Windows. What might a VMS
>>>> version look like? How many cores could it scale up to without too
>>>> much synchronization overhead?
>>>>
>>>> Alternatively, for existing VMS apps, how might they be sped up on
>>>> non-VAX hardware? Based on the mailbox copy driver loop in the VMS
>>>> port of Perl that I spent some time piecing together, I've noticed a
>>>> few patterns that can't possibly perform well on any hardware newer
>>>> than Alpha, and maybe not on Alpha either.
>>>
>>> The normal assumption regarding speed of disk IO would be that:
>>>
>>> RMS record IO ($GET and $PUT) < RMS block IO ($READ and $WRITE) <
>>> $QIO(W) < $IO_PERFORM(W) < memory mapped file
>>>
>>> (note that assumption and fact are spelled differently)
>>
>> I'm not sure I have ever understood why people think memory mapped
>> files would be faster than a QIO under VMS.
>
> Very few layers.

Well. The point is that QIO (and even more IO_PERFORM) actually do not
have many layers. It's very different from I/O operations in a Unix system.

> Large degree of freedom to the OS about how to read.

There aren't really that many ways to read from a disk. Bascially, there
is just one. Beyond that, it's about layers inside the OS. But we
already said here now that we don't want those layers...

>> So how would memory mapped I/O be any faster? You basically cannot be
>> any faster than one DMA transfer. In fact, with memory mapped I/O, you
>> might be also hitting the page fault handling, and a reading in of a
>> full page, which might be more than you needed, causing some overhead
>> as well.
>
> Fewer layers to go through. More freedom to read ahead.

Read ahead is something that the system can easily do both for normal
I/O and memory mapped I/O. It's just a question of speculative reads
assuming some pattern by the program. Most commonly that you are reading
files sequentially from start to finish.

>> Also, what do $IO_PERFORM do, that could possibly make it faster than
>> QIO?
>
> $QIO(W) is original. $IO_PERFORM(W) was added much later.
>
> $IO_PERFORM(W) is called fast path IO. The name and the fact
> that it was added later hint at it being faster.
>
> That name has always give me associations to a strategy of
> doing lots of checks upfront and then skip layers
> and checks when doing the actual reads/writes. But I
> have no idea if that is actually what it does.

I think we covered this elsewhere. IO_PERFORM is really the one without
any layers in VMS, it seems. QIO looks like it might actually have one
or two layers in there.

But IO_PERFORM I can't see would be any slower than memory mapped I/O.
And it might in fact actually be faster in some situations.

Johnny

On 11/6/2023 5:58 AM, Johnny Billquist wrote:
> On 2023-11-05 16:58, bill wrote:
>
>> We have so many "colleges" teaching trade school courses (like diesel
>> mechanics, HVAC welding and even motorcycle mechanics)I really wish
>> trade schools would step up to the plate ad start teaching IT and in
>> particular thing like COBOL, Fortran and PL/I.  They are not going away.
>
> Academia should not teach languages. If they do, they are clearly not
> doing the right thing.

I hear this all the time. Believe it or not, in a way, it is a debate
that has been going on for centuries. Should college teach trades or
just liberal arts and leave the trades to others? Like it or not, the
majority of college students are there with a belief that they will
learn something that will enhance their future earnings and not just to
expand their minds.

As for teaching languages. Every program I have ever seen taught
languages.

>
> They should teach methods, principles, concepts, ideas.

They teach that, too, but without detailed knowledge of a language
it really doesn't do much for the student.

>
> The language is just a tool. You need to learn and use different tools
> all the time. That you could/should learn at the place where it is
> used/needed.

The old OJT idea. But most places expect when they hire you you will
hit the ground running. Thus the reason for this latest craze for
"certification". HR places a value on them. The government requires
them. I really see little value in something you learned over the
weekend in a boot camp and probably forgot by Teusday.

bill

> And if you have all the teachings from academia, that
> should be an easy thing.

Not necessarily easy, but doable. But, how many hiring managers are
going to be willing to wait for you to learn something they expected
you to learn in college before you can provide any value to the company?

bill

On 2023-11-05, Dan Cross <cross@spitfire.i.gajendra.net> wrote:
>
> That's your real problem. No one is selling your favorite
> technologies to the younger generation, and the players in the
> mainframe space make it sort of ridiculously hard to casually
> gain exposure.
>

The only public access I have ever seen to mainframes is via the
Master The Mainframe program (which I was pointed to by someone here).

I have never seen another legal way of gaining z/OS experience for free
(although a _really_ old MVS version is legally available for free).

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 11/6/2023 8:35 AM, Simon Clubley wrote:
> On 2023-11-05, Dan Cross <cross@spitfire.i.gajendra.net> wrote:
>>
>> That's your real problem. No one is selling your favorite
>> technologies to the younger generation, and the players in the
>> mainframe space make it sort of ridiculously hard to casually
>> gain exposure.
>>
>
> The only public access I have ever seen to mainframes is via the
> Master The Mainframe program (which I was pointed to by someone here).

Master the Mainframe was fun. I did it until I ran out of subjects I
was interested in. Somehow Linux on a Mainframe just doesn't float my
boat.

>
> I have never seen another legal way of gaining z/OS experience for free
> (although a _really_ old MVS version is legally available for free).
>

z/OS is not the only mainframe. I have UNISYS OS2200 running on this
PC just like the VMS Student Edition.

bill

On 11/6/2023 6:31 AM, Johnny Billquist wrote:
> On 2023-11-04 22:44, Arne Vajhøj wrote:
>> On 11/4/2023 7:11 AM, Johnny Billquist wrote:
>>> So how would memory mapped I/O be any faster? You basically cannot be
>>> any faster than one DMA transfer. In fact, with memory mapped I/O,
>>> you might be also hitting the page fault handling, and a reading in
>>> of a full page, which might be more than you needed, causing some
>>> overhead as well.
>>
>> Fewer layers to go through. More freedom to read ahead.
>
> Read ahead is something that the system can easily do both for normal
> I/O and memory mapped I/O. It's just a question of speculative reads
> assuming some pattern by the program. Most commonly that you are reading
> files sequentially from start to finish.

$QIO(w) and $IO_PERFORM(W) could.

But at least for $QIO(W) then I would be very surprised if it did. It is
from before VIOC/XFC so originally it did not have anywhere to
store read ahead data. VMS engineering could have changed
behavior when VIOC/XFC was introduced. But I doubt it.

$IO_PERFORM(W) maybe. But I think the spirit is still
"do exactly what the developer asked for and nothing more".

Arne

In article <uiaq3u$fun1$2@dont-email.me>,
Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> wrote:
>On 2023-11-05, Dan Cross <cross@spitfire.i.gajendra.net> wrote:
>>
>> That's your real problem. No one is selling your favorite
>> technologies to the younger generation, and the players in the
>> mainframe space make it sort of ridiculously hard to casually
>> gain exposure.
>>
>
>The only public access I have ever seen to mainframes is via the
>Master The Mainframe program (which I was pointed to by someone here).
>
>I have never seen another legal way of gaining z/OS experience for free
>(although a _really_ old MVS version is legally available for free).

Yeah, same with VM/CMS.

It sounds like Master the Mainframe will get you an account for
a little while, then you lose access. That makes it hard to
play around with it to gain familiarity. Oh well.

- Dan C.

In article <0c8ed1d9-ae12-4086-b1f8-0d647b5f5891n@googlegroups.com>,
Jake Hamby (Solid State Jake) <jake.hamby@gmail.com> wrote:
>On Sunday, November 5, 2023 at 10:43:09 AM UTC-8, Dan Cross wrote:
>> In article <kqq5kd...@mid.individual.net>,
>> bill <bill.gu...@gmail.com> wrote:
>> >On 11/5/2023 1:18 PM, Arne Vajhøj wrote:
>> >> [snip]
>> >> But that does mean that it would make sense for colleges
>> >> to spew out hundreds or thousands of people with solid
>> >> PL/I skills.
>> >
>> >No one says hundreds of thousands. But if 1% of CS and CIS
>> >students were offered a course in COBOL or Fortran or PL/I
>> >it would provide a pool of about 2000 candidates a year. A
>> >little better than 0.
>> I wonder why you're so focused on confronting this in the
>> university system? What is so different about a single semester
>> course and a short-term job with OJT and an option to hire full
>> time at the end for new graduates?
>>
>> If these organizations are so eager to hire programmers with
>> COBOL and Fortran experience, why don't they take charge of the
>> situation and provide the experience themselves?
>
>IBM has a very successful program for self-taught instruction for university students, originally called "Master the Mainframe", which they've
>opened up to the general public under the name "IBM Z Xplore". It's quite fun. I highly recommend it. The challenges walk you through how to
>use VS Code to submit COBOL, JCL, and HLASM, how to use ISPF on a 3270 terminal, how to use the UNIX shell via ssh, how to use their modern
>Python integration, and one challenge uses the LinuxONE community cloud account and walks you through setting up Apache there to serve some
>files.
>
>IBM also has a variety of badges and certificates you can get by taking courses in their catalog on COBOL, LinuxONE, REXX, z/OS system
>administration, and more. Many of those courses give you Web remote access to a time-limited Windows VM on an intranet that connects to their
>student account they create for you automatically when you sign up for the course, with a pre-configured Windows 3270 emulator. The Z Xplore
>courses also give you a student account but show you how to connect from your own PC/Linux/Mac. I'm still working through the Z Xplore
>courses.

Sounds very time limited. :-(

- Dan C.

In article <kqqluvF6p35U3@mid.individual.net>,
bill <bill.gunshannon@gmail.com> wrote:
>On 11/5/2023 4:50 PM, John Dallman wrote:
>> In article <ui8nnq$995$1@reader2.panix.com>,
>> cross@spitfire.i.gajendra.net (Dan Cross) wrote:
>>
>>> If these organizations are so eager to hire programmers with
>>> COBOL and Fortran experience, why don't they take charge of the
>>> situation and provide the experience themselves?
>>
>> Management short-terminism, I think.
>
>Maybe, to some extent.
>
>But I provided an example of academia doing all it could to make the
>effort fail.

I don't think academia cares whether COBOL succeeds (whatever
that means) or fails. I think academia is more concerned with
research, and making the most of their finite time teaching
students to give them as much of the continually-expanding field
as possible.

>And I say again, I really doubt my University is somehow
>unique. If the professors at my University take action to quash COBOL
>I have to believe others are doing it to. It has been a long time
>since the academics decided rather than preparing their students for
>the world after graduation they preferred to steer them in another
>direction.

There are at least two logical fallacies here; first, that your
experience with a sample set size of one extends to anything
else.

Second, it's a false dichotomy to assert that, because your
university is not teaching students COBOL that it is not
"preparing their students for the world after graduation."

In particular, it may be the case that some number of students
can get a good job slinging COBOL after they graduate; it is
also likely that those same students can get a different jobs
that pay equally well. It is also likely true that other
students can similarly get other jobs. That is, the false
dichotomy is the apparant assumption that not getting the COBOL
job somehow means they don't have a job at all; you've shared
no evidence that would show that.

>> Incidentally, the situation for Fortran is rather different from COBOL
>> and PL/I. Academic computer scientists don't usually touch it, but
>> physicists, computational chemists and the like still use it heavily. So
>> there are still people coming onto the job market who know it.
>
>Yes, but how well (and I don't mean syntax)? Who is teaching them?
>If they are not getting this training from CS Departments are they
>getting any of the fundamentals or just syntax?

Why do you think they would learn this better in a CS undergrad
program than anywhere else?

The important thing I would like to see from students who've
gone through a CS undergraduate education is familiarity with
basic theory and concepts (algorithms; data structures; basic
complexity theory and the ability to apply that to working
code), and exposure to a smattering of research topics to give
them an indication that there is a bigger world out there.
Beyond that? I could care less if they learn a particular
language as a student. Is it really that hard for a competent
person to pick up COBOL?

>Some of the worst business programs I ever had to work with were in
>Fortran and written by engineering faculty who needed something to do
>during the summer.
>
>Note, I said some of the worst. The worst was COBOL written by
>government contractors who obviously had no real COBOL experience
>and definitely no supervision. But then, low bidder and all that.
>(Think about it. Budget tracking with 6 figure sums and all the
>intermediate results were unsigned!!)

Honestly? This sounds more like a process failure than anything
else. Who reviewed their code? What kind of change management
systems are in place to submit to the production repository and
go through the testing pipeline to get into prod?

- Dan C.

In article <kqs6nnF6p35U7@mid.individual.net>,
bill <bill.gunshannon@gmail.com> wrote:
>On 11/6/2023 5:58 AM, Johnny Billquist wrote:
>> On 2023-11-05 16:58, bill wrote:
>>
>>> We have so many "colleges" teaching trade school courses (like diesel
>>> mechanics, HVAC welding and even motorcycle mechanics)I really wish
>>> trade schools would step up to the plate ad start teaching IT and in
>>> particular thing like COBOL, Fortran and PL/I.  They are not going away.
>>
>> Academia should not teach languages. If they do, they are clearly not
>> doing the right thing.
>
>I hear this all the time. Believe it or not, in a way, it is a debate
>that has been going on for centuries. Should college teach trades or
>just liberal arts and leave the trades to others? Like it or not, the
>majority of college students are there with a belief that they will
>learn something that will enhance their future earnings and not just to
>expand their minds.

....and what makes you think that learning code will expand their
future earnings more than learning any other language?

>As for teaching languages. Every program I have ever seen taught
>languages.

Chemists have to learn how to do lab work, too. Learning a
language in the course of one's undergraduate education is a
means to an end, not the end itself.
>> They should teach methods, principles, concepts, ideas.
>
>They teach that, too, but without detailed knowledge of a language
>it really doesn't do much for the student.
>
>>
>> The language is just a tool. You need to learn and use different tools
>> all the time. That you could/should learn at the place where it is
>> used/needed.
>
>The old OJT idea. But most places expect when they hire you you will
>hit the ground running.

Well, then they should adopt technologies that the people they
can hire know, but pout over sour grapes because the kids aren't
taught the antiquated stuff they want them to know.

This is what's going to kill COBOL, by the way: eventually we'll
hit an inflection point where the cost of maintenance is high
enough to justify a rewrite.

>Thus the reason for this latest craze for
>"certification". HR places a value on them. The government requires
>them. I really see little value in something you learned over the
>weekend in a boot camp and probably forgot by Teusday.

I suspect the reason so much "new" COBOL is written each year
is because, instead of modifying old and unmaintained code bases
they are instead copied and modified, leaving the original
intact. Much of this has to do with the design of COBOL, by
the way.

- Dan C.

On 11/6/2023 8:07 AM, bill wrote:
> On 11/6/2023 5:58 AM, Johnny Billquist wrote:
>> On 2023-11-05 16:58, bill wrote:
>>
>>> We have so many "colleges" teaching trade school courses (like diesel
>>> mechanics, HVAC welding and even motorcycle mechanics)I really wish
>>> trade schools would step up to the plate ad start teaching IT and in
>>> particular thing like COBOL, Fortran and PL/I.  They are not going away.
>>
>> Academia should not teach languages. If they do, they are clearly not
>> doing the right thing.
>
> I hear this all the time.  Believe it or not, in a way, it is a debate
> that has been going on for centuries.  Should college teach trades or
> just liberal arts and leave the trades to others?  Like it or not, the
> majority of college students are there with a belief that they will
> learn something that will enhance their future earnings and not just to
> expand their minds.
>
> As for teaching languages. Every program I have ever seen taught
> languages.
>
>> They should teach methods, principles, concepts, ideas.
>
> They teach that, too, but without detailed knowledge of a language
> it really doesn't do much for the student.

They need a language to write code to see the principles
applied.

To really understand what is general and what is specific
for the language they need more than one language.

I would expect a CS degree to give knowledge of about 3-5
languages.

Most learn Python and Java today. But other languages are
seen: OCAML, Haskell, C#, C++, C, PHP, JavaScript etc..

After that they should be able to learn new languages.

The time to learn a language depends on the complexity
of the language. It will be a lot faster to learn Cobol
than Ada95.

>> The language is just a tool. You need to learn and use different tools
>> all the time. That you could/should learn at the place where it is
>> used/needed.
>
> The old OJT idea.  But most places expect when they hire you you will
> hit the ground running.  Thus the reason for this latest craze for
> "certification".  HR places a value on them.  The government requires
> them.  I really see little value in something you learned over the
> weekend in a boot camp and probably forgot by Teusday.
>
>>               And if you have all the teachings from academia, that
>> should be an easy thing.
>
> Not necessarily easy, but doable.  But, how many hiring managers are
> going to be willing to wait for you to learn something they expected
> you to learn in college before you can provide any value to the company?

If you hire someone with work experience then you may decide to
go for someone with experience in the languages and frameworks to
be used. After all you may need someone to teach those without that
experience if such skills are not present already in the org.

But if you hire someone right out of college, then it is really
sub-optimal to hire based on their skills in the specific
languages and frameworks. The level is not that high anyway and
it will only take a few months for those without those skills
to catch up. So it is really about hiring those that are
generally good.

Arne

Den 2023-11-07 kl. 01:18, skrev Arne Vajhøj:
> On 11/6/2023 6:31 AM, Johnny Billquist wrote:
>> On 2023-11-04 22:44, Arne Vajhøj wrote:
>>> On 11/4/2023 7:11 AM, Johnny Billquist wrote:
>>>> So how would memory mapped I/O be any faster? You basically cannot be
>>>> any faster than one DMA transfer. In fact, with memory mapped I/O, you
>>>> might be also hitting the page fault handling, and a reading in of a
>>>> full page, which might be more than you needed, causing some overhead
>>>> as well.
>>>
>>> Fewer layers to go through. More freedom to read ahead.
>>
>> Read ahead is something that the system can easily do both for normal I/O
>> and memory mapped I/O. It's just a question of speculative reads assuming
>> some pattern by the program. Most commonly that you are reading files
>> sequentially from start to finish.
>
> $QIO(w) and $IO_PERFORM(W) could.
>
> But at least for $QIO(W) then I would be very surprised if it did. It is
> from before VIOC/XFC so originally it did not have anywhere to
> store read ahead data. VMS engineering could have changed
> behavior when VIOC/XFC was introduced. But I doubt it.
>
> $IO_PERFORM(W) maybe. But I think the spirit is still
> "do exactly what the developer asked for and nothing more".
>
> Arne
>

And, of course, the "application" might do the prefetch.
Rdb has something called "detectable asyncronious pre-fetch"
that kiks in if Rdb detects that the user application fetch
database pages that are in sequence. It's on by default but
can be switched of if you do not think there is a benefit.
Such as if your I/Os are generally small and very random.

On 2023-11-06 14:07, bill wrote:
> On 11/6/2023 5:58 AM, Johnny Billquist wrote:
>> On 2023-11-05 16:58, bill wrote:
>>
>>> We have so many "colleges" teaching trade school courses (like diesel
>>> mechanics, HVAC welding and even motorcycle mechanics)I really wish
>>> trade schools would step up to the plate ad start teaching IT and in
>>> particular thing like COBOL, Fortran and PL/I.  They are not going away.
>>
>> Academia should not teach languages. If they do, they are clearly not
>> doing the right thing.
>
> I hear this all the time.  Believe it or not, in a way, it is a debate
> that has been going on for centuries.  Should college teach trades or
> just liberal arts and leave the trades to others?  Like it or not, the
> majority of college students are there with a belief that they will
> learn something that will enhance their future earnings and not just to
> expand their minds.
>
> As for teaching languages. Every program I have ever seen taught
> languages.

Yeah. And the first language we were exposed to in the CS program at my
university was Lisp. Not because the industry asked for it, but because
it was/is a good tool to learn a bunch of important concepts. A few
years later they switched to Scheme, and then Eiffel. Not because of any
industry need, but because of teaching needs.

And that's how it should be. Of course you'll learn some language or
other. But they are *tools* for a purpose. If you think that academia
should be repsonsible for churning out robots then I can see why you
have such a big issue with academia.

>> They should teach methods, principles, concepts, ideas.
>
> They teach that, too, but without detailed knowledge of a language
> it really doesn't do much for the student.

But the choice of language is very different if you pick it because it's
the best tool for teaching concepts, or if you pick it because
"industry" wants it.

I think I've made it clear enough which I think academia should go for...

>> The language is just a tool. You need to learn and use different tools
>> all the time. That you could/should learn at the place where it is
>> used/needed.
>
> The old OJT idea.  But most places expect when they hire you you will
> hit the ground running.  Thus the reason for this latest craze for
> "certification".  HR places a value on them.  The government requires
> them.  I really see little value in something you learned over the
> weekend in a boot camp and probably forgot by Teusday.

SOmething is broken in your corner. I can't tell what, and I certainly
can't fix it. But from where I stand, academia is not the problem, and
not the ones doing anything wrong, and is not the thing that should be
"fixed".

>>               And if you have all the teachings from academia, that
>> should be an easy thing.
>
> Not necessarily easy, but doable.  But, how many hiring managers are
> going to be willing to wait for you to learn something they expected
> you to learn in college before you can provide any value to the company?

How about people actually learn things on their own? If they have a good
toolbox, which academia should have provided to them, they can pick up
whatever they are then interested in, and in turn that will lead them to
applying for positions where they can make use of the skills they then
have, if they can't find a place that is willing to hire a new graduate.

But a new graduate have larger issues than not knowing the language. The
first few years they always needs a lot of hand holding and pointing
directions. Language skills is the least of the problems, really.

And having some language skills or not is not the biggest question when
hiring a new graduate. At least it has not been for me, when hiring. It
becomes mostly a question of assessing ability to understand problems,
being able to figure something out, being able to work in a group and
interact with others, and having a sound mind.

I have no idea about your experience or environment, but to me you are
describing something very foreign that I can't understand how it would
ever work. And then claiming that academia is the problem makes it just
sound even stranger.

Johnny

On 2023-11-07 01:54, Arne Vajhøj wrote:
> On 11/6/2023 8:07 AM, bill wrote:
>> On 11/6/2023 5:58 AM, Johnny Billquist wrote:
>>> On 2023-11-05 16:58, bill wrote:
>>>
>>>> We have so many "colleges" teaching trade school courses (like diesel
>>>> mechanics, HVAC welding and even motorcycle mechanics)I really wish
>>>> trade schools would step up to the plate ad start teaching IT and in
>>>> particular thing like COBOL, Fortran and PL/I.  They are not going
>>>> away.
>>>
>>> Academia should not teach languages. If they do, they are clearly not
>>> doing the right thing.
>>
>> I hear this all the time.  Believe it or not, in a way, it is a debate
>> that has been going on for centuries.  Should college teach trades or
>> just liberal arts and leave the trades to others?  Like it or not, the
>> majority of college students are there with a belief that they will
>> learn something that will enhance their future earnings and not just to
>> expand their minds.
>>
>> As for teaching languages. Every program I have ever seen taught
>> languages.
>>
>>> They should teach methods, principles, concepts, ideas.
>>
>> They teach that, too, but without detailed knowledge of a language
>> it really doesn't do much for the student.
>
> They need a language to write code to see the principles
> applied.

Indeed.

> To really understand what is general and what is specific
> for the language they need more than one language.
>
> I would expect a CS degree to give knowledge of about 3-5
> languages.
>
> Most learn Python and Java today. But other languages are
> seen: OCAML, Haskell, C#, C++, C, PHP, JavaScript etc..
>
> After that they should be able to learn new languages.

Hmm. Let's see... When I was at University, we did (more or less in order):
Lisp
C Prolog
Assmbler (68000)
SQL
Lexx
Yacc
Ada
C++

And then we also did some industry automation system, and some imaginary
CPUs to write our own microcode as well as some hypothetical
architecture for which we wrote compilers (actually, it was close to a
PDP-10, for fun).

I might have forgotten some language(s) as well. But I think it's clear
it's way more than 3-5 languages.

But the point was never about "this is what you will work with in the
real world", but now we're going to write a compiler for some imagined
language, targetting some imaginary CPU. For that you will need Lexx and
Yacc. So now you'll have to learn those tools/languages.

And to make things more clear:
Lisp was used for a course in program methology. C for data structures
and algorithms. Later also for operating systems. Prolog - logic
processing and natural language processing. Assembler - computer
architecture. SQL - Databases. Ada - structured programming and large
software systems (Ada have some really interesting and nice properties
for that). C++ for object oriented cruft.

> The time to learn a language depends on the complexity
> of the language. It will be a lot faster to learn Cobol
> than Ada95.

Sure. But this is really a minor thing anyway.

>>> The language is just a tool. You need to learn and use different
>>> tools all the time. That you could/should learn at the place where it
>>> is used/needed.
>>
>> The old OJT idea.  But most places expect when they hire you you will
>> hit the ground running.  Thus the reason for this latest craze for
>> "certification".  HR places a value on them.  The government requires
>> them.  I really see little value in something you learned over the
>> weekend in a boot camp and probably forgot by Teusday.
>>
>>>               And if you have all the teachings from academia, that
>>> should be an easy thing.
>>
>> Not necessarily easy, but doable.  But, how many hiring managers are
>> going to be willing to wait for you to learn something they expected
>> you to learn in college before you can provide any value to the company?
>
> If you hire someone with work experience then you may decide to
> go for someone with experience in the languages and frameworks to
> be used. After all you may need someone to teach those without that
> experience if such skills are not present already in the org.

Indeed. If you hire someone experienced, then the question of language
specific knowledge is interesting. But then we are talking about someone
who have already extensive previous experience with the language. It's
not related to academia.

> But if you hire someone right out of college, then it is really
> sub-optimal to hire based on their skills in the specific
> languages and frameworks. The level is not that high anyway and
> it will only take a few months for those without those skills
> to catch up. So it is really about hiring those that are
> generally good.

Yeah. These guys don't know much about anything anyway, and language
skills are pretty pointless. They will not be a good indicator if they
are a good hire or not at all. There are other things that you need to
look for to tell if it is a good hire or not.

Johnny

On 2023-11-07 01:18, Arne Vajhøj wrote:
> On 11/6/2023 6:31 AM, Johnny Billquist wrote:
>> On 2023-11-04 22:44, Arne Vajhøj wrote:
>>> On 11/4/2023 7:11 AM, Johnny Billquist wrote:
>>>> So how would memory mapped I/O be any faster? You basically cannot
>>>> be any faster than one DMA transfer. In fact, with memory mapped
>>>> I/O, you might be also hitting the page fault handling, and a
>>>> reading in of a full page, which might be more than you needed,
>>>> causing some overhead as well.
>>>
>>> Fewer layers to go through. More freedom to read ahead.
>>
>> Read ahead is something that the system can easily do both for normal
>> I/O and memory mapped I/O. It's just a question of speculative reads
>> assuming some pattern by the program. Most commonly that you are
>> reading files sequentially from start to finish.
>
> $QIO(w) and $IO_PERFORM(W) could.
>
> But at least for $QIO(W) then I would be very surprised if it did. It is
> from before VIOC/XFC so originally it did not have anywhere to
> store read ahead data. VMS engineering could have changed
> behavior when VIOC/XFC was introduced. But I doubt it.
>
> $IO_PERFORM(W) maybe. But I think the spirit is still
> "do exactly what the developer asked for and nothing more".

I'd say memory mapped I/O falls in the exact same category here.

Johnny

On 2023-11-04 15:14, Dan Cross wrote:
> In article <ui59v2$rn5$3@news.misty.com>,
> Johnny Billquist <bqt@softjar.se> wrote:
>> On 2023-11-03 22:45, Dan Cross wrote:
>>> In article <ff7f2845-84cc-4c59-a007-1b388c82543fn@googlegroups.com>,
>>> Jake Hamby (Solid State Jake) <jake.hamby@gmail.com> wrote:
>>>> I've become a little obsessed with the question of how well
>>>> OpenVMS performs relative to Linux inside a VM, under different
>>>> conditions. My current obsession is the libuv library which
>>>> provides a popular async I/O abstraction layer implemented for
>>>> all the different flavors of UNIX that have async I/O, as well
>>>> as for Windows. What might a VMS version look like? How many
>>>> cores could it scale up to without too much synchronization
>>>> overhead?
>>>>
>>>> Alternatively, for existing VMS apps, how might they be sped up
>>>> on non-VAX hardware? Based on the mailbox copy driver loop in
>>>> the VMS port of Perl that I spent some time piecing together,
>>>> I've noticed a few patterns that can't possibly perform well on
>>>> any hardware newer than Alpha, and maybe not on Alpha either.
>>>>
>>>> My first concern is the use of VAX interlocked queue
>>>> instructions, because they're system calls on Itanium and x86
>>>> (and PALcode on Alpha). They can't possibly run as fast as VMS
>>>> programmers may be assuming based on what made sense on a VAX
>>>> 30 years ago. The current hotness is to use lock-free data
>>>> structures as much as possible.
>>>
>>> I don't know that that's the current hotness so much as trying
>>> to structure problems so that the vast majority of access are
>>> local to a core, so that either locks are unnecessary, or
>>> taking a lock is just an uncontended write to an owned cache
>>> line.
>>>
>>> Consider the problem of a per-core queue with work-stealing;
>>> the specifics of what's in the queue don't matter so much as the
>>> overall structure of the problem: items in the queue might
>>> represent IO requests, or they may represent runnable threads,
>>> or whatever. Anyway, a core will usually process things on
>>> its own local queue, but if it runs out of things to do, it
>>> may choose some victim and steal some work from it.
>>> Conventionally, this will involve everyone taking locks on
>>> these queues, but that's ok: we expect that work stealing is
>>> pretty rare, and that usually a core is simply locking its own
>>> queue, which will be an uncontended write on a cacheline it
>>> already owns. In any event, _most of the time_ the overhead
>>> of using a "lock" will be minimal (assuming relatively simply
>>> spinlocks or MCS locks or something like that).
>>
>> I generally agree with everything you posted. But I think it's
>> unfortunate that you bring in cache lines into this, as it's somewhat
>> incorrect to talk about "owning" a cache line, and cache lines are not
>> really that relevant at all in this context.
>
> This is an odd thing to say. It's quite common to talk about
> core ownership of cache lines in both MESI and MOESI, which are
> the cache coherency protocols in use on modern x86 systems from
> both Intel and AMD, respectively. Indeed, the "O" in "MOESI" is
> for "owned" (https://en.wikipedia.org/wiki/MOESI_protocol).

I think we're talking past each other.

> Cache lines and their ownership are massively relevant in the
> context of mutual exclusion, atomic operations, and working on
> shared memory generally.

You are then talking about it in the sense that if a CPU writes to
memory, and it's in the cache, then yes, you get ownership properties
then/there. Primarily because you are not immediately writing back to
main memory, and other CPUs are allowed to then hold read-only copies. I
sortof covered that exact topic further down here, about either
invalidate or update the caches of other CPUs. And it also holds that
other CPUs cannot freely write or update those memory cells when another
CPU holds a dirty cache about it.

But before the CPU writes the data, it never owns the cache line. So it
don't make sense to say "uncontended write on a cacheline it already
owns". Ownership only happens when you do the write. And once you've
written, you own it.

I'm probably trying to make it too simple when I write about it. Trying
to avoid going overly-technical has its risks...

>> But that's all there is to it. Now, it is indeed very costly if you have
>> many CPUs trying to spin-lock on the same data, because each one will be
>> hitting the same memory, causing a big pressure on that memory address.
>
> More accurately, you'll be pushing the associated cache line
> through a huge number of state transitions as different cores
> vye for exclusive ownership of the line in order to write the
> lock value. Beyond a handful of cores, the generated cache
> coherency traffic begins to dominate, and overall throughput
> _decreases_.
>
> This is of paramount importance when we start talking about
> things like synchronization primitives, which are based on
> atomic updates to shared memory; cache-inefficient algorithms
> simply do not scale beyond a handful of actors, and why things
> like MCS locks or CHM locks are used on many-core machines.
> See, for example:
> https://people.csail.mit.edu/nickolai/papers/boyd-wickizer-locks.pdf

Yes. This is the scaling problem with spinlocks. Cache coherency starts
becoming costly. Having algorithms or structures that allow locking to
not be localized to one address is an important piece to help alleviate it.

>> (And then we had the PDP-11/74, which had to implement cache coherency
>> between CPUs without any hardware support...)
>
> Sounds rough. :-/

It is. CPU was modified to actually always step around the cache for one
instruction (ASRB - used for spin locks), and then you manually turn on
and off cache bypass on a per-page basis, or in general of the CPU,
depending on what is being done, in order to not get into issues of
cache inconsistency.

Johnny

On Monday, November 6, 2023 at 4:24:47 PM UTC-8, Dan Cross wrote:
> >
> >IBM also has a variety of badges and certificates you can get by taking courses in their catalog on COBOL, LinuxONE, REXX, z/OS system
> >administration, and more. Many of those courses give you Web remote access to a time-limited Windows VM on an intranet that connects to their
> >student account they create for you automatically when you sign up for the course, with a pre-configured Windows 3270 emulator. The Z Xplore
> >courses also give you a student account but show you how to connect from your own PC/Linux/Mac. I'm still working through the Z Xplore
> >courses.
>
> Sounds very time limited. :-(

The LinuxONE free accounts are definitely time limited. They get deleted after 120 days and then you can just request a new one. The Z Xplore accounts seem to last forever, or at least mine hasn't been deleted after well over a year. There's a page you can go to reset the password. It's also the same account across all the challenges (except for the LinuxONE ones).

The other IBM z/OS badge programs are different, and I think those accounts (the ones accessed through a temporary provisioned Windows cloud desktop) do get reset/deleted fairly quickly.

On Monday, November 6, 2023 at 5:51:47 AM UTC-8, bill wrote:
> >
> > I have never seen another legal way of gaining z/OS experience for free
> > (although a _really_ old MVS version is legally available for free).
> >
> z/OS is not the only mainframe. I have UNISYS OS2200 running on this
> PC just like the VMS Student Edition.

You downloaded the free UNISYS OS2200 system? I looked at that, and the Burroughs MCP emulation that UNISYS also sells, and decided those systems were far too old and weird for me to spend time on. VAX and System/360 may be old, but at least they used 8-bit bytes and two's-complement integer arithmetic.

There was going to be a hobbyist version of the ZD&T test environment, which is IBM's official emulator solution for z/OS development. It runs on 64-bit Linux. I paid the $160 or whatever the fee was, and then I got some emails from their sales people and then nothing. Apparently they had some change of plans and they switched from USB license sticks to cloud license keys and I may have gotten lost in the shuffle. Their website mentions the "Learners Edition" but it's been scrubbed from the main sales page. I'll have to send some emails to see what happened.

It was my interest in getting access to the "Learners Edition" (IBM only wanted to sell it to people with either professional experience with z/OS or who demonstrated mastery by doing the Z Xplore program and the z/OS Mainframe Practitioner badge, which is difficult!) that led me to learn as much about it as I did. Data sets are annoying!

A third option is paying to set up your own z/OS system in the IBM Cloud. They'll give you a free usage credit of $700 or $1000 or something like that, and then you can use that to mess around. Of course you have to delete everything so you don't get billed.

I believe the IRS has some OS 2200 systems somewhere (the vast majority of their work has been done using IBM and HLASM, not COBOL, which is why it's been so difficult for them to modernize it), because their coding standards page mentions it as one of the assembly languages they use, specifically:

UNISYS ClearPath OS2200 Meta-Assembler (MASM) Programming Reference Manual Level 6R3J
UNISYS ClearPath OS2200 Executive Control Language (ECL) and FURPUR Reference Manual

https://www.irs.gov/irm/part2/irm_02-005-003

Regards,
Jake Hamby

On Sunday, November 5, 2023 at 6:07:52 PM UTC-8, Arne Vajhøj wrote:
> > App servers like Node.js either have only a
> > single event loop, or one per core, so VMS's limit of 48 (64 - 8
> > reserved) should be more than adequate.
> First I am not sure how you get 64 - 8 to be 48. My calculator
> claims 56.
>
> :-)

I'm not sure how I got that, either! The reserved event flags are 24-31. Somehow I rounded 8 up to 16. When you call lib$get_ef(), the event flags are reserved from 63 downwards. That's probably a good way to flush out any obvious bugs related to bit shifting and the flags being split into two 32-bit clusters. You can only wait on a set of event flags in one or the other cluster, not both. The design I have in mind for libuv would have an event flag per event loop. 56 should be more than enough, and each event loop would only be waiting on a single flag (its own).

I made a bigger mistake in my earlier post, which I'm surprised no one seems to have caught. I wrongly stated that the RMS file block services allowed you to read/write starting from any position, but in fact you have to read/write starting from a 512-byte block boundary, the same as for $QIO and $IO_PERFORM. You can read less than a full block, so that's how the ends of files are handled (it's not like CP/M where everything gets rounded up and you lose the real file size or anything like that).

With that additional info, I now believe the C RTL is going through RMS primarily in order to handle all the different VMS record formats it has to support, and not because it offers support for a file seek position or R/W from arbitrary file offsets, because it doesn't support either. The same for the other VMS high-level languages. So for a high-performance async I/O implementation, like a libuv port, it'd make the most sense to open files in "UFO" mode and then call $IO_PERFORM and $QIO exclusively. That's especially true since sockets, pipes (mailboxes), and ttys (pseudoterminal or serial port) will have to work that way.

Regards,
Jake Hamby

On 2023-11-07, Johnny Billquist <bqt@softjar.se> wrote:
>
> And having some language skills or not is not the biggest question when
> hiring a new graduate. At least it has not been for me, when hiring. It
> becomes mostly a question of assessing ability to understand problems,
> being able to figure something out, being able to work in a group and
> interact with others, and having a sound mind.
>
> I have no idea about your experience or environment, but to me you are
> describing something very foreign that I can't understand how it would
> ever work. And then claiming that academia is the problem makes it just
> sound even stranger.
>

If you want to learn the current fashion for a job, that's what trade
schools are for (if you need them).

If you want to learn how to think and how to understand various abstract
concepts, that's what university is for.

A university will probably teach you the languages that are the current
fashion at the time of taking the courses, but their main purpose is to
give you a rounded and deep conceptual education.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 11/7/2023 12:27 PM, Jake Hamby (Solid State Jake) wrote:
> I made a bigger mistake in my earlier post, which I'm surprised no
> one seems to have caught. I wrongly stated that the RMS file block
> services allowed you to read/write starting from any position, but in
> fact you have to read/write starting from a 512-byte block boundary,
> the same as for $QIO and $IO_PERFORM. You can read less than a full
> block, so that's how the ends of files are handled (it's not like
> CP/M where everything gets rounded up and you lose the real file size
> or anything like that).

Depends on where you are measuring.

:-)

application---(API X)---RMS---(API Y)---VMS

API X starts at block boundaries for $READ and $WRITE but on
byte boundaries for $GET and $PUT.

API Y starts at block boundaries for $QIO(W).

So RMS $GET and $PUT translate from starting at byte boundary to
start at block boundary, which means a buffer.

As you note then $IO_PERFORM(W) start at block boundary just like $QIO(W).

When SSIO (I cannot remember the system service name right now)
becomes available it will allow start at byte boundaries.

> With that additional info, I now believe the C RTL is going through
> RMS primarily in order to handle all the different VMS record formats

In general that is what RMS provide.

The RM in RMS stands for record management, so it manage
records on top of the basic file blocks. That covers
understanding the record formats and locking.

I believe most language RTL use $GET and $PUT to let
RMS do all the record format handling.

C RTL is a little different I believe. fgets/fputs do records
but fread/fwrite do blocks. And then there is the difference between
ctx=rec and ctx=stm. I don't know when it use $GET/$PUT and when
it use $READ/$WRITE and does its own record handling.

Arne

On 11/6/23 6:18 PM, Arne Vajhøj wrote:
> On 11/6/2023 6:31 AM, Johnny Billquist wrote:

>> Read ahead is something that the system can easily do both for normal
>> I/O and memory mapped I/O. It's just a question of speculative reads
>> assuming some pattern by the program. Most commonly that you are
>> reading files sequentially from start to finish.
>
> $QIO(w) and $IO_PERFORM(W) could.
>
> But at least for $QIO(W) then I would be very surprised if it did. It is
> from before VIOC/XFC so originally it did not have anywhere to
> store read ahead data. VMS engineering could have changed
> behavior when VIOC/XFC was introduced. But I doubt it.

Are you saying that you think merely using $QIO bypasses XFC? If so,
how do you think SHOW MEM/CACHE can give you "Total QIOs"? And note
this from the performance management manual page 72:
"I/O service can be optimized at the application level by using RMS
global buffers to share caches among processes. This method offers the
benefit of satisfying an I/O request without issuing a QIO; whereas
system memory software cache (that is, XFC) is checked to satisfy QIOs."

https://docs.vmssoftware.com/docs/vsi-openvms-performance-management-manual.pdf

> $IO_PERFORM(W) maybe. But I think the spirit is still
> "do exactly what the developer asked for and nothing more".

Here it would make more sense that the buffered object you are required
to provide is the only "cache" you get. But I don't know the internals.

On 11/7/2023 5:44 PM, Craig A. Berry wrote:
> On 11/6/23 6:18 PM, Arne Vajhøj wrote:
>> On 11/6/2023 6:31 AM, Johnny Billquist wrote:
>
>>> Read ahead is something that the system can easily do both for normal
>>> I/O and memory mapped I/O. It's just a question of speculative reads
>>> assuming some pattern by the program. Most commonly that you are
>>> reading files sequentially from start to finish.
>>
>> $QIO(w) and $IO_PERFORM(W) could.
>>
>> But at least for $QIO(W) then I would be very surprised if it did. It is
>> from before VIOC/XFC so originally it did not have anywhere to
>> store read ahead data. VMS engineering could have changed
>> behavior when VIOC/XFC was introduced. But I doubt it.
>
> Are you saying that you think merely using $QIO bypasses XFC?  If so,
> how do you think SHOW MEM/CACHE can give you "Total QIOs"?  And note
> this from the performance management manual page 72:
> "I/O service can be optimized at the application level by using RMS
> global buffers to share caches among processes. This method offers the
> benefit of satisfying an I/O request without issuing a QIO; whereas
> system memory software cache (that is, XFC) is checked to satisfy QIOs."

I am sure that $QIO(W) hits XFC. Else there would not be
much point in XFC.

The question is whether $QIO(W) get more data read into XFC
than asked for by the user. It could but I doubt it.

Arne