In article <ui8dkv$20n6$1@dont-email.me>,
Craig A. Berry <craigberry@nospam.mac.com> wrote:
>
>On 11/4/23 4:41 PM, Dan Cross wrote:
>> In article <ui64is$3go9i$1@dont-email.me>,
>> Craig A. Berry <craigberry@nospam.mac.com> wrote:
>>> For disk I/O, yes, it's
>>> almost certain that using virtual memory primitives to synchronize
>>> integral pages between disk and memory will be faster than any other I/O
>>> method; that's why pretty much every database product on every platform
>>> does it.
>>
>> Everyone starts out thinking that, but most are wrong:
>> https://db.cs.cmu.edu/mmap-cidr2022/
>
>Interesting article. I don't buy its conclusions as an indication of
>what most databases do. Its main point seems to be that mapped memory
>makes transactions difficult because, "due to transparent paging, the OS
>can flush a dirty page to secondary storage at any time, irrespective of
>whether the writing transaction has committed." Well, yeah, you have to
>lock pages in memory, and apparently mlock() doesn't guarantee that it
>does what is says on the tin.

That's not what `mlock()` does; `mlock` makes sure that pages
associated with the locked region are physically present in RAM;
it says nothing about when those pages, if dirtied, are flushed
back to disk. Note that flushing the contents of a page back to
secondary storage doesn't automatically evict that page from
memory; it just ensures that the page contents on durable
storage match those in RAM.

Furthermore, lots of databases are too large to fit entirely
into physical memory, thus making `mlock` on their backing file
stores impractical.

Also, that was just one of the four problems they identified.
The others are unpredictable IO stalls, issues with error
handling, and performance issues due to outdated implementation
assumptions.

>They also seem to be surprised by the fact
>that paging to a remote storage system is slow. Well, duh. So there
>are complexities and difficulties arising from using mapped memory.

I think you are confusing what they refer to as "remote" in the
paper with something unrelated: they do not mean paging to a
remote _system_, as in a totally separate computer across a
network like an Ethernet, but rather are talking about problems
inherent in SMP systems and the need to synchronize with "remote
cores" sharing an address in the same physical system. Here, a
"remote core" is simply any CPU that is not the local CPU. This
is in a discussion of synchronization of page tables (and page
table contention) and the need to issue TLB shootdowns to other
cores that likely have cached translations for a page that is
being evicted from an address space; these are a known
performance bottleneck in managing virtual memory systems.
Multiple simultaneous modifications to a virtual address space
are known to be expensive and lead to performance bottlenecks.
For more details on both the problem space and an example
implementation that addresses some of the issues, see:
https://people.csail.mit.edu/nickolai/papers/clements-radixvm-2014-08-05.pdf

>But there are also complexities and difficulties with their recommended
>alternative of "traditional buffer pools" and libaio or io_uring. Stack
>Overflow is full of articles about problems, including broken
>implementations and security problems, with those APIs. I don't doubt
>they can be useful if the complexities and difficulties are managed
>correctly, but the same is true of mapped memory.

If one is going to implement a DBMS, I would hope one would have
enough experience and knowledge to avoid the sorts of problems
one would reach to Stack Overflow for help with. :-/

Regardless, I believe the authors' point is that, once one takes
into consideration the complexities they point out with respect
to mmap, it becomes much less attractive because one realizes
that all of the details one wanted to avoid (or rather, throw
over the wall to the OS) with respect to manual buffer management
are details one has to confront anyway. And other mechanisms
give you more precise control over these behaviors; combined
with _other_ services provided by the OS (io_uring, aio, etc)
you can get all of the benefits one naively expects out of mmap
without the downsides, at a similar level of complexity.

>Also noteworthy is that in the long list of databases whose
>implementations they considered, SQLite is the only major player they
>mention. SQLite has an unusual storage model (data typing imposed at run
>time, IIRC) and was never intended as a multi-user OLTP system, so its
>I/O choices aren't much of a guide to what databases in general do.
>
>If the authors of the article could show evidence that the research
>teams at Oracle, Microsoft, and IBM have come to the same conclusions
>they did and don't use mapped memory in their products anymore, that
>would be interesting, but they make only an oblique reference to SQL
>Server in a context that implies it does use mapped memory.

SQLite is public domain software, whereas most databases have
restrictive licenses that prohibit authors from mentioning them
by name. So no, they can't really show that directly.

>I suspect all of the major databases use any and every I/O mechanism
>available in different situations, chosen by a variety of engine
>choices, run-time heuristics, and configuration options. Mapped memory
>may not be the only game in town besides file I/O like it once was, but
>I'm just not buying that it's been entirely eclipsed.

See above.

- Dan C.

On Sunday, November 5, 2023 at 12:47:21 PM UTC-5, Arne Vajhøj wrote:
> On 11/5/2023 10:58 AM, bill wrote:
> > In 1980 I was a COBOL Programmer/Systems Analyst on IBM and Univac
> > Mainframes. Trade journals were already saying "COBOL is dead".
> > And yet, it went on. Two of the largest ISes in the country (probably
> > the world) were COBOL. Still are today and there is no plan or sign that
> > they will ever be replaced with another language. There was a third.
> > Contractor opted to not renew and the program died. Not because of any
> > flaws in COBOL but because academia refuses to teach it even as an
> > elective. System belonged to the contractor so stayed with them. New
> > system written from scratch in god only knows what language, Some
> > language du jour. The new system is slow, cumbersome, error prone and
> > lacks many of the features that the old system had.
> >
> > We have so many "colleges" teaching trade school courses (like diesel
> > mechanics, HVAC welding and even motorcycle mechanics)I really wish
> > trade schools would step up to the plate ad start teaching IT and in
> > particular thing like COBOL, Fortran and PL/I. They are not going away..
> There is not much point, because there will not be jobs
> for them.
>
> There are a lot of Cobol and PL/I code in production doing
> usually highly business critical stuff.
>
> But if cost and risk are too high to rewrite to C++
> or Java or C# or whatever, then the risk of
> rewriting it in Cobol is also too high.
>
> So instead the new functionality is put in
> secondary systems using newer technology and
> only changes that has to be done in the core
> are done there.
>
> All the less critical but developer time consuming
> code in UI and reporting are long gone.
>
> As a result the demand is small.
>
> It is not zero but despite the frequent "we have
> a problem" announcements, then it seems like they can
> keep and hire+train the people they need for
> maintenance.
>
> Arne

The demand these days is a little different as well.

I am currently working with 2 clients using PL/I code. They no longer have PL/I expertise onsite and don't know how to modify the existing code in one case, and don't know what the code does in the other case. Nice work for me, but it would be nice if there were resources for such tasks more readily available...

Dan

On 11/5/2023 1:03 PM, abrsvc wrote:
> On Sunday, November 5, 2023 at 12:47:21 PM UTC-5, Arne Vajhøj wrote:
>> As a result the demand is small.
>>
>> It is not zero but despite the frequent "we have a problem"
>> announcements, then it seems like they can keep and hire+train the
>> people they need for maintenance.

> The demand these days is a little different as well.
>
> I am currently working with 2 clients using PL/I code. They no
> longer have PL/I expertise onsite and don't know how to modify the
> existing code in one case, and don't know what the code does in the
> other case. Nice work for me, but it would be nice if there were
> resources for such tasks more readily available...

These cases exist.

And you are probably not the only consultant doing that
type of work.

But that does mean that it would make sense for colleges
to spew out hundreds or thousands of people with solid
PL/I skills.

Arne

On 11/5/2023 12:47 PM, Arne Vajhøj wrote:
> On 11/5/2023 10:58 AM, bill wrote:
>> In 1980 I was a COBOL Programmer/Systems Analyst on IBM and Univac
>> Mainframes.  Trade journals were already saying "COBOL is dead".
>> And yet, it went on.  Two of the largest ISes in the country (probably
>> the world) were COBOL. Still are today and there is no plan or sign that
>> they will ever be replaced with another language.  There was a third.
>> Contractor opted to not renew and the program died.  Not because of any
>> flaws in COBOL but because academia refuses to teach it even as an
>> elective.  System belonged to the contractor so stayed with them.  New
>> system written from scratch in god only knows what language, Some
>> language du jour.  The new system is slow, cumbersome, error prone and
>> lacks many of the features that the old system had.
>>
>> We have so many "colleges" teaching trade school courses (like diesel
>> mechanics, HVAC welding and even motorcycle mechanics)I really wish
>> trade schools would step up to the plate ad start teaching IT and in
>> particular thing like COBOL, Fortran and PL/I.  They are not going away.
>
> There is not much point, because there will not be jobs
> for them.

There are piles of jobs for them. All the current crop are retiring.
I just mentioned above a multi million dollar a year project that had
to be abandoned for lack of programmers? And why was that? They even
tried a strong internship program to bring in second and third year
undergrads they would teach COBOL. I put up the flyers all over the
CS Department where I worked. I talked with students about it, And
all this time one of the Professors went around telling the students
not to do it. And refusing to sign off on their internship program
to fulfill the internship requirement for the degree. I expect it was
treated the same at other CS Departments. Any student who had gone into
this program was guaranteed a very high paying job upon graduation.

>
> There are a lot of Cobol and PL/I code in production doing
> usually highly business critical stuff.
>
> But if cost and risk are too high to rewrite to C++
> or Java or C# or whatever, then the risk of
> rewriting it in Cobol is also too high.

If it is already written in COBOL why would you need to re-write
it in COBOL?

>
> So instead the new functionality is put in
> secondary systems using newer technology and
> only changes that has to be done in the core
> are done there.

Numerous surveys have shown that while new front ends in other
languages are being done so is a lot of new COBOL. I have less
experience and knowledge with current day Fortran but I do know
of a number of large engineering com0panies that still use it.
And not just to keep those VAX programs running. :-)

>
> All the less critical but developer time consuming
> code in UI and reporting are long gone.

UI's, yes. But then COBOL was never much n UI's anyway. That's
why web programming stuff was added to CICS ages ago. IBM certainly
knew that the web would replace all those 3270's. And, I have done
web front ends for COBOL. Time changes that doesn't mean the language
become useless. Not much done with ISAM nowadays. BUt DB's fit right
in.

>
> As a result the demand is small.
>
> It is not zero but despite the frequent "we have
> a problem" announcements, then it seems like they can
> keep and hire+train the people they need for
> maintenance.

By "we have a problem" I assume you are referencing the infamous New
Jersey State Employment Agency case. Too bad people stopped talking
about it before it turned out to not be a COBOL problem at all but a
web front end problem.

bill

In article <kqq3l0F6p35U1@mid.individual.net>,
bill <bill.gunshannon@gmail.com> wrote:
>On 11/5/2023 12:37 PM, John Dallman wrote:
>> In article <kqpsd9F6p36U1@mid.individual.net>, bill.gunshannon@gmail.com
>> (bill) wrote:
>>
>>> I really wish trade schools would step up to the plate and start
>>> teaching IT and in particular thing like COBOL, Fortran and PL/I.
>>> They are not going away.
>>
>> However, recruiting young people onto those courses, rather than ones in
>> mobile app creation, might be quite hard.
>
>Not every student studying computers is a budding computer scientist.
>Most schools with CS programs also have CIS programs and a lot of
>students go that route.
>
>> You'd also need relevant
>> compilers, operating systems and hardware.
>
>Just like VMS has a student program IBM and UNISYS, the two biggest
>mainframers left, offer programs for people to familiarize with their
>world. There are also Open Source COBOL and Fortran systems available.
>PL/i being the red headed step child. (As a side note, the recent
>edition of the CACM has an article that claims IBM intended PL/I to
>take over the computing world from the likes of COBOL, Fortran and
>ALGOL. Funny considering that they kept it hidden in their corporate
>bowels for so long.)
>
>>
>> I'm not saying this is a bad idea, but if it was easy, it might well
>> already be happening.
>
>The problem goes back to marketing. Academia has held it down for so
>long that people don't know about it.

You've said this before, but it's honestly kind of a silly thing
to say: Academia isn't "holding [COBOL] down" by not teaching
it, but rather, they've got a limited amount of time to
familiarize students with an ever-expanding field. The blunt
reality is that most new graduates will never program in COBOL,
regardless of whether insurers and financial services firms
continue to use it in highly vertical applications: this is not
due to some nefarious plot on the part of academics, it's due to
larger industry trends. There is thus little incentive to teach
it, and so it (correctly) falls very far down on the list of
priorities.

Moreover, it's a highly specialized, vertical skill: if someone
who has graduated a 4-year program in CS or CIS or MIS or
whatever can't pick it up in relatively short order, then that's
the real problem. Universities should be teaching techniques,
not technologies. Same with Fortran, frankly.

>It will take a sell but right now
>it is hard to find the right person to sell it. Maybe this whole loan
>debt thing will finally provide the push it needs.

That's your real problem. No one is selling your favorite
technologies to the younger generation, and the players in the
mainframe space make it sort of ridiculously hard to casually
gain exposure.

- Dan C.

On 11/5/2023 1:18 PM, Arne Vajhøj wrote:
> On 11/5/2023 1:03 PM, abrsvc wrote:
>> On Sunday, November 5, 2023 at 12:47:21 PM UTC-5, Arne Vajhøj wrote:
>>> As a result the demand is small.
>>>
>>> It is not zero but despite the frequent "we have a problem"
>>> announcements, then it seems like they can keep and hire+train the
>>> people they need for maintenance.
>
>> The demand these days is a little different as well.
>>
>> I am currently working with 2 clients using PL/I code.  They no
>> longer have PL/I expertise onsite and don't know how to modify the
>> existing code in one case, and don't know what the code does in the
>> other case.  Nice work for me, but it would be nice if there were
>> resources for such tasks more readily available...
>
> These cases exist.
>
> And you are probably not the only consultant doing that
> type of work.

True, but those numbers are going down, too.

>
> But that does mean that it would make sense for colleges
> to spew out hundreds or thousands of people with solid
> PL/I skills.

No one says hundreds of thousands. But if 1% of CS and CIS
students were offered a course in COBOL or Fortran or PL/I
it would provide a pool of about 2000 candidates a year. A
little better than 0.

bill

In article <kqq5kdF6p36U2@mid.individual.net>,
bill <bill.gunshannon@gmail.com> wrote:
>On 11/5/2023 1:18 PM, Arne Vajhøj wrote:
>> [snip]
>> But that does mean that it would make sense for colleges
>> to spew out hundreds or thousands of people with solid
>> PL/I skills.
>
>No one says hundreds of thousands. But if 1% of CS and CIS
>students were offered a course in COBOL or Fortran or PL/I
>it would provide a pool of about 2000 candidates a year. A
>little better than 0.

I wonder why you're so focused on confronting this in the
university system? What is so different about a single semester
course and a short-term job with OJT and an option to hire full
time at the end for new graduates?

If these organizations are so eager to hire programmers with
COBOL and Fortran experience, why don't they take charge of the
situation and provide the experience themselves?

- Dan C.

On 11/5/2023 1:35 PM, bill wrote:
> On 11/5/2023 1:18 PM, Arne Vajhøj wrote:
>> On 11/5/2023 1:03 PM, abrsvc wrote:
>>> On Sunday, November 5, 2023 at 12:47:21 PM UTC-5, Arne Vajhøj wrote:
>>>> As a result the demand is small.
>>>>
>>>> It is not zero but despite the frequent "we have a problem"
>>>> announcements, then it seems like they can keep and hire+train the
>>>> people they need for maintenance.
>>
>>> The demand these days is a little different as well.
>>>
>>> I am currently working with 2 clients using PL/I code.  They no
>>> longer have PL/I expertise onsite and don't know how to modify the
>>> existing code in one case, and don't know what the code does in the
>>> other case.  Nice work for me, but it would be nice if there were
>>> resources for such tasks more readily available...
>>
>> These cases exist.
>>
>> And you are probably not the only consultant doing that
>> type of work.
>
> True, but those numbers are going down, too.
>
>> But that does mean that it would make sense for colleges
>> to spew out hundreds or thousands of people with solid
>> PL/I skills.
>
> No one says hundreds of thousands.  But if 1% of CS and CIS
> students were offered a course in COBOL or Fortran or PL/I
> it would provide a pool of about 2000 candidates a year.  A
> little better than 0.

Maybe some should.

I expect it to have absolutely zero impact on the industry
(those using those languages do so because they have to - not
because they want to), but I believe in knowing classics.

Just like someone studying human languages should learn
latin and/or greek, then someone studying computer
languages should learn Fortran and/or Cobol (or maybe
PL/I or Ada or Pascal/Modula-2 there are plenty of
classical languages).

Arne

On 11/5/2023 1:23 PM, bill wrote:
> On 11/5/2023 12:47 PM, Arne Vajhøj wrote:
>> On 11/5/2023 10:58 AM, bill wrote:
>>> In 1980 I was a COBOL Programmer/Systems Analyst on IBM and Univac
>>> Mainframes.  Trade journals were already saying "COBOL is dead".
>>> And yet, it went on.  Two of the largest ISes in the country (probably
>>> the world) were COBOL. Still are today and there is no plan or sign that
>>> they will ever be replaced with another language.  There was a third.
>>> Contractor opted to not renew and the program died.  Not because of any
>>> flaws in COBOL but because academia refuses to teach it even as an
>>> elective.  System belonged to the contractor so stayed with them.  New
>>> system written from scratch in god only knows what language, Some
>>> language du jour.  The new system is slow, cumbersome, error prone and
>>> lacks many of the features that the old system had.
>>>
>>> We have so many "colleges" teaching trade school courses (like diesel
>>> mechanics, HVAC welding and even motorcycle mechanics)I really wish
>>> trade schools would step up to the plate ad start teaching IT and in
>>> particular thing like COBOL, Fortran and PL/I.  They are not going away.
>>
>> There is not much point, because there will not be jobs
>> for them.
>
> There are piles of jobs for them.

If you look at the job market then the re are not many jobs.

And the majority of those jobs that are there happen to be in India.

Observable fact.

>   All the current crop are retiring.

Supply is decreasing.

So no excess demand mean that demand is also decreasing.

> I just mentioned above a multi million dollar a year project that had
> to be abandoned for lack of programmers?  And why was that?  They even
> tried a strong internship program to bring in second and third year
> undergrads they would teach COBOL.  I put up the flyers all over the
> CS Department where I worked.  I talked with students about it,  And
> all this time one of the Professors went around telling the students
> not to do it.  And refusing to sign off on their internship program
> to fulfill the internship requirement for the degree.  I expect it was
> treated the same at other CS Departments.  Any student who had gone into
> this program was guaranteed a very high paying job upon graduation.

Consider how a market economy works.

If demand exceed supply then price goes up until demand and
supply are equal.

For a skill that take time to learn then that can cause
a crazy peak for a few years until supply catches up.

If a company really need Cobol skills they raise pay until they get
the Cobol developers they need.

If they don't really need Cobol skills then they pick
another language.

You can see what is happening.

>> There are a lot of Cobol and PL/I code in production doing
>> usually highly business critical stuff.
>>
>> But if cost and risk are too high to rewrite to C++
>> or Java or C# or whatever, then the risk of
>> rewriting it in Cobol is also too high.
>
> If it is already written in COBOL why would you need to re-write
> it in COBOL?

Things evolve.

Always a need for new features and a need to reduce cost.

The traditional banks are a bit pressed by the constraints
of their old systems not being present in the systems from the
new fintech companies.

>> So instead the new functionality is put in
>> secondary systems using newer technology and
>> only changes that has to be done in the core
>> are done there.
>
> Numerous surveys have shown that while new front ends in other
> languages are being done so is a lot of new COBOL.

Sure about that?

The last such number I have seen saying so is from the late 90's.

>> All the less critical but developer time consuming
>> code in UI and reporting are long gone.
>
> UI's, yes.  But then COBOL was never much n UI's anyway.  That's
> why web programming stuff was added to CICS ages ago.  IBM certainly
> knew that the web would replace all those 3270's.  And, I have done
> web front ends for COBOL.  Time changes that doesn't mean the language
> become useless.  Not much done with ISAM nowadays. BUt DB's fit right
> in.

Cobol is not cost effective for web development.

>> As a result the demand is small.
>>
>> It is not zero but despite the frequent "we have
>> a problem" announcements, then it seems like they can
>> keep and hire+train the people they need for
>> maintenance.
>
> By "we have a problem" I assume you are referencing the infamous New
> Jersey State Employment Agency case.

No. The "Cobol programmers are getting old and retiring so we will
have a problem" message that come out occasionally.

> Too bad people stopped talking
> about it before it turned out to not be a COBOL problem at all but a
> web front end problem.

That is not what happened.

What happened was that people submitted their request for
unemployment benefits successfully, it took for ever to process
them, people checked status on their requests via web and the
web frontend got overwhelmed by impatient users wanting to
check why they did not receive any money. The symptoms
appeared at the frontend, but was caused by the backend.

Arne

On Sunday, November 5, 2023 at 10:43:09 AM UTC-8, Dan Cross wrote:
> In article <kqq5kd...@mid.individual.net>,
> bill <bill.gu...@gmail.com> wrote:
> >On 11/5/2023 1:18 PM, Arne Vajhøj wrote:
> >> [snip]
> >> But that does mean that it would make sense for colleges
> >> to spew out hundreds or thousands of people with solid
> >> PL/I skills.
> >
> >No one says hundreds of thousands. But if 1% of CS and CIS
> >students were offered a course in COBOL or Fortran or PL/I
> >it would provide a pool of about 2000 candidates a year. A
> >little better than 0.
> I wonder why you're so focused on confronting this in the
> university system? What is so different about a single semester
> course and a short-term job with OJT and an option to hire full
> time at the end for new graduates?
>
> If these organizations are so eager to hire programmers with
> COBOL and Fortran experience, why don't they take charge of the
> situation and provide the experience themselves?
>
> - Dan C.

IBM has a very successful program for self-taught instruction for university students, originally called "Master the Mainframe", which they've opened up to the general public under the name "IBM Z Xplore". It's quite fun. I highly recommend it. The challenges walk you through how to use VS Code to submit COBOL, JCL, and HLASM, how to use ISPF on a 3270 terminal, how to use the UNIX shell via ssh, how to use their modern Python integration, and one challenge uses the LinuxONE community cloud account and walks you through setting up Apache there to serve some files.

IBM also has a variety of badges and certificates you can get by taking courses in their catalog on COBOL, LinuxONE, REXX, z/OS system administration, and more. Many of those courses give you Web remote access to a time-limited Windows VM on an intranet that connects to their student account they create for you automatically when you sign up for the course, with a pre-configured Windows 3270 emulator. The Z Xplore courses also give you a student account but show you how to connect from your own PC/Linux/Mac. I'm still working through the Z Xplore courses.

On Sunday, November 5, 2023 at 10:02:14 AM UTC-8, bill wrote:
> On 11/5/2023 12:37 PM, John Dallman wrote:
> > You'd also need relevant
> > compilers, operating systems and hardware.
> Just like VMS has a student program IBM and UNISYS, the two biggest
> mainframers left, offer programs for people to familiarize with their
> world. There are also Open Source COBOL and Fortran systems available.
> PL/i being the red headed step child. (As a side note, the recent
> edition of the CACM has an article that claims IBM intended PL/I to
> take over the computing world from the likes of COBOL, Fortran and
> ALGOL. Funny considering that they kept it hidden in their corporate
> bowels for so long.)

When I got interested in learning the state of z/OS last year, I looked at PL/I, among other mainframe tech. It has hundreds of keywords, like COBOL, but unlike COBOL you can give variables the same names as keywords, which seems like it'd cause a lot of trouble. The parser must have to be very good at recovery from syntax errors.

The original PL/I was developed in cooperation with the SHARE and GUIDE users groups. I've heard PL/I is still more popular in Europe, relative to COBOL, than elsewhere. It doesn't seem to have much to recommend it for business users, with all the focus being on new COBOL features.

The language that IBM *has* kept a confidential trade secret and which they write their z/OS compilers and large parts of their mainframe stack in is a PL/I derivative called PL/X.

PL/X is a descendent of PL/S, which has a brief Wikipedia entry: https://en..wikipedia.org/wiki/IBM_PL/S

Apparently if you're a really good customer, you can get access to PL/S, but I don't know if that includes PL/X. IBM is very protective of their IP and thwarting attempts to clone what they consider "their" tech, so having their flagship OS and compilers written in a language that you can't get a compiler for outside of IBM is likely a part of that thinking. So IBM engineers will talk about using PL/X but not exactly what's in it.

Regards,
Jake Hamby

On Saturday, November 4, 2023 at 2:44:30 PM UTC-7, Arne Vajhøj wrote:
> On 11/4/2023 7:11 AM, Johnny Billquist wrote:
> >
> > I'm not sure I have ever understood why people think memory mapped files
> > would be faster than a QIO under VMS.
> Very few layers.
>
> Large degree of freedom to the OS about how to read.
> > With memory mapped I/O, what you essentially get is that I/O transfers
> > go directly from/to disk to user memory with a single operation. There
> > are no intermediate buffers, no additional copying. Which is what you
> > pretty much always have otherwise on Unix systems.
> >
> > However, a QIO under VMS is already a direct communication between the
> > physical memory and the device with no intermediate buffers, additional
> > copying or whatever, unless I'm confused (and VMS changed compared to
> > RSX here...).
> XFC?
> > So how would memory mapped I/O be any faster? You basically cannot be
> > any faster than one DMA transfer. In fact, with memory mapped I/O, you
> > might be also hitting the page fault handling, and a reading in of a
> > full page, which might be more than you needed, causing some overhead as
> > well.
> Fewer layers to go through. More freedom to read ahead.
> > Also, what do $IO_PERFORM do, that could possibly make it faster than QIO?
> $QIO(W) is original. $IO_PERFORM(W) was added much later.
>
> $IO_PERFORM(W) is called fast path IO. The name and the fact
> that it was added later hint at it being faster.
>
> That name has always give me associations to a strategy of
> doing lots of checks upfront and then skip layers
> and checks when doing the actual reads/writes. But I
> have no idea if that is actually what it does.

I think you've summed up the open questions and tradeoffs with VMS I/O. Memory mapped I/O could be much faster under certain access patterns but slower for others. It's great for random access patterns that leverage the page cache. $IO_PERFORM(W) seems to primarily fix $QIO having too many arguments to hold in registers (beyond 6 get pushed to the stack) and eliminating a few spinlocks (as Stephen Hoffman quoted in his reply).

I pushed some work in progress to my GitHub repo for the VMS port of libuv that I started to work on. I'm not sure if it should be considered a bug in DECthreads that it behaves so impossibly poorly on the async ping test that writes a byte to a pipe to wake a child sleeping on a poll() and then sleeps on poll() to wait for the reply on a pipe going the other way. I'm getting no more than 60 round trips / sec, even after commenting out the sched_yield() call before it goes back to sleep. Being so unworkably slow using the POSIX APIs, I didn't bother to try to run the test cases.

My guess would be that VMS's user-mode thread scheduler doesn't realize immediately that one of the benchmark process's threads has signaled another one to wake up from a poll() by writing to a pipe shared between them, and it's going to sleep on a timer before eventually waking up. The poll() version of the benchmark uses almost no CPU because it's mostly sleeping. You can get instant inter-thread wakeup with local event flags, and as I mentioned at the start of this thread, I did see above 10K round-trips/sec with a hacked-up version of libuv's async benchmark to test the do-nothing wake-up case.

What makes libuv such an interesting test is that the naive port using the C RTL synchronous POSIX APIs is almost unusably bad, and in the best-case scenario has too many unneeded layers of emulation, but by using the Win32 and Linux io_uring implementations as a model, it's possible to get acceptable behavior using all native VMS APIs, with completion ASTs waking up the event loops using one local event flag per event loop. App servers like Node..js either have only a single event loop, or one per core, so VMS's limit of 48 (64 - 8 reserved) should be more than adequate.

So that gets back to RMS vs. $QIO(W)/$IO_PERFORM(W). Since nobody cares about anything other than Stream_LF sequential files in UNIX world, there's nothing advanced that we need or want from RMS as far as record handling, but we do want to perform reads/writes of any size starting from any offset. If you use $QIO(W), then, like $IO_PERFORM(W), you must access file data by virtual block offset (starting at block 1, not 0), and then you're effectively doing raw disk I/O, except with the OS mapping only the blocks in the file you have access to through your channel.

The reason C RTL and everyone else go through RMS is so they can do byte-level file I/O instead of block-level, and to take advantage of XFC caching. Everyone keeps talking about raw throughput and database servers and not the much more common use cases for file I/O.

For purposes of libuv, it's convenient that Windows has a flag you must set if you're going to mmap() a file, while VMS also has a flag you have to set to use $CRMPSC. The VMS flag you must set is FAB$V_UFO, for user file open, and if you set it then that means you also are forced to use $QIO(W) (or $IO_PERFORM(W)) to access the file contents because RMS essentially opens the file as "foreign" and only handles open, create, and close. So you can't use RMS to read/write a file and also mmap() it, or at least not from the same open channel.

Coincidentally, both VMS and Windows have a convenient "delete temp file on close" flag, which the VMS port of Perl also uses, but VMS has the additional quirk that you can make temp files that don't even get directory entries.

With VMS being closer to UNIX in some ways and closer to Windows in others, while being distinct from either, for an abstraction layer like libuv, which needs to provide a file descriptor-like API (except async), it makes sense to only use the RMS file access APIs, in async mode, because going the raw virtual block read/write route is going to be a lot of code, a lot of bugs, and it bypasses the XFC block cache, which is almost certainly not what users are going to want.

The last detail I want to mention is that if I want to support libuv's mapping of stdio file handles and fds to its own API, which is probably not used often, the ideal situation would be if I could get the underlying I/O channel for a C RTL UNIX file descriptor, and vice versa. But the only API remotely like this was added by customer request (I found the forum thread requesting it), and it returns the I/O channel for a FILE *, not an fd. That call is decc$get_channel(__FILE_ptr32 fp).

From the header files, I see that <socket.h> has decc$socket_fd(int __channel) to turn an open socket channel into an fd. That will definitely be useful, since every type of file needs different special handling in libuv anyway. Just below it is decc$get_sdc (int __descrip_no) to go the other direction. If there aren't equivalent APIs for files and mailboxes/pipes, then the fallback would be to have an internal two-way pipe between them like Perl does for spawning subprocesses.

Regards,
Jake Hamby

On 11/5/2023 4:50 PM, John Dallman wrote:
> In article <ui8nnq$995$1@reader2.panix.com>,
> cross@spitfire.i.gajendra.net (Dan Cross) wrote:
>
>> If these organizations are so eager to hire programmers with
>> COBOL and Fortran experience, why don't they take charge of the
>> situation and provide the experience themselves?
>
> Management short-terminism, I think.

Maybe, to some extent.

But I provided an example of academia doing all it could to make the
effort fail. And I say again, I really doubt my University is somehow
unique. If the professors at my University take action to quash COBOL
I have to believe others are doing it to. It has been a long time
since the academics decided rather than preparing their students for
the world after graduation they preferred to steer them in another
direction.

>
> Incidentally, the situation for Fortran is rather different from COBOL
> and PL/I. Academic computer scientists don't usually touch it, but
> physicists, computational chemists and the like still use it heavily. So
> there are still people coming onto the job market who know it.

Yes, but how well (and I don't mean syntax)? Who is teaching them?
If they are not getting this training from CS Departments are they
getting any of the fundamentals or just syntax?

Some of the worst business programs I ever had to work with were in
Fortran and written by engineering faculty who needed something to do
during the summer.

Note, I said some of the worst. The worst was COBOL written by
government contractors who obviously had no real COBOL experience
and definitely no supervision. But then, low bidder and all that.
(Think about it. Budget tracking with 6 figure sums and all the
intermediate results were unsigned!!)

bill

On 05/11/2023 19:11, Arne Vajhøj wrote:
> On 11/5/2023 1:23 PM, bill wrote:
>> On 11/5/2023 12:47 PM, Arne Vajhøj wrote:
>>> On 11/5/2023 10:58 AM, bill wrote:

<Big snip>
I thought that MicroFocus Cobol was getting a lot of interest a few
years back

--
Chris

On 11/5/2023 6:53 PM, Chris Townley wrote:
> I thought that MicroFocus Cobol was getting a lot of interest a few
> years back

A very large number stay on Cobol on mainframe.

Some number migrate to C++ or Java or whatever on Linux.

But some migrate to Cobol on Linux or Windows.

There are some companies that specializes in the latter.

Microfocus (now owned by OpenText) is the most well known. But
there are other like Heirloom.

That approach reduces migration cost and risk tremendously. And
still achieve massive hardware cost reduction.

But it does obviously not fix the software architecture. The
Cobol code does not change behavior just because it runs
on Linux or Windows.

I am considering it sort of a half solution.

But Microfocus is making good money on it. They have acquired
a ton of other software over the years so their revenue is a lot
more than Cobol, but it seems likely that they sell for 0.5-1.0 B$
Cobol software and services.

Arne

On 11/5/2023 6:14 PM, bill wrote:
> On 11/5/2023 4:50 PM, John Dallman wrote:
>> Incidentally, the situation for Fortran is rather different from COBOL
>> and PL/I. Academic computer scientists don't usually touch it, but
>> physicists, computational chemists and the like still use it heavily. So
>> there are still people coming onto the job market who know it.
>
> Yes, but how well (and I don't mean syntax)?  Who is teaching them?
> If they are not getting this training from CS Departments are they
> getting any of the fundamentals or just syntax?

Scientific computing is pretty different from business applications.

I don't think trying to apply best practices for business applications
to scientific computing makes sense.

The business application will need to be maintained 10-20-30-40-50
years with constantly changes.

Many scientific applications (not scientific libraries those are
different and more like business applications) only need one
successful run and then the code goes on the shelf for future
copy paste.

Very different contexts.

Fortran was very widely used back in the days in physics, astronomy,
chemistry, biology, medicine, economics etc..

Some probably still use Fortran.

But a lot of that stuff are done in Python today. Today
Python is the language for scientific computing.

Note that the interpreted Python is of course only
"orchestrating" the number crunching - the number crunching
itself are done in native code - Fortran, Fortran converted to C
(yes!) or C.

So the Fortran code still exist, but the scientists don't use
it and don't even see it.

> Some of the worst business programs I ever had to work with were in
> Fortran and written by engineering faculty who needed something to do
> during the summer.

Writing a business application in a language designed for
scientific computing by people with mostly scientific
computing experience has a very low chance of success.

Writing a nuclear explosion simulation in Cobol by
people with expertise in ISAM files and BCD would probably not
go well either.

Arne

On 11/5/2023 7:39 PM, Arne Vajhøj wrote:
> Fortran was very widely used back in the days in physics, astronomy,
> chemistry, biology, medicine, economics etc..
>
> Some probably still use Fortran.
>
> But a lot of that stuff are done in Python today. Today
> Python is the language for scientific computing.
>
> Note that the interpreted Python is of course only
> "orchestrating" the number crunching - the number crunching
> itself are done in native code - Fortran, Fortran converted to C
> (yes!) or C.

And in case someone wonders.

SciPy should contain Fortran while NumPy should contain Fortran
converted to C.

Arne

On 11/5/2023 7:39 PM, Arne Vajhøj wrote:
> On 11/5/2023 6:14 PM, bill wrote:
>> On 11/5/2023 4:50 PM, John Dallman wrote:
>>> Incidentally, the situation for Fortran is rather different from COBOL
>>> and PL/I. Academic computer scientists don't usually touch it, but
>>> physicists, computational chemists and the like still use it heavily. So
>>> there are still people coming onto the job market who know it.
>>
>> Yes, but how well (and I don't mean syntax)?  Who is teaching them?
>> If they are not getting this training from CS Departments are they
>> getting any of the fundamentals or just syntax?
>
> Scientific computing is pretty different from business applications.

I never said it wasn't.

>
> I don't think trying to apply best practices for business applications
> to scientific computing makes sense.
>
> The business application will need to be maintained 10-20-30-40-50
> years with constantly changes.
>
> Many scientific applications (not scientific libraries those are
> different and more like business applications) only need one
> successful run and then the code goes on the shelf for future
> copy paste.
>
> Very different contexts.
>
> Fortran was very widely used back in the days in physics, astronomy,
> chemistry, biology, medicine, economics etc..
>
> Some probably still use Fortran.

And I agreed with all of it. They were called Domain Specific
Languages. Interestingly enough the same article in CACM I
mentioned earlier that said IBM intended PL/I to replace all
the other languages had a comment about not understanding why
there we even had COBOL, Fortran and ALGOL.

>
> But a lot of that stuff are done in Python today. Today
> Python is the language for scientific computing.

Python is a general purpose language that everyone wants to use for
everything.

>
> Note that the interpreted Python is of course only
> "orchestrating" the number crunching - the number crunching
> itself are done in native code - Fortran, Fortran converted to C
> (yes!) or C.

Well, when you come right down to it it's all done in machine
language. :-)

>
> So the Fortran code still exist, but the scientists don't use
> it and don't even see it.
>

I have to admit I have never seen anything about Python generating or
being converted into Fortran.

>> Some of the worst business programs I ever had to work with were in
>> Fortran and written by engineering faculty who needed something to do
>> during the summer.
>
> Writing a business application in a language designed for
> scientific computing by people with mostly scientific
> computing experience has a very low chance of success.
>
> Writing a nuclear explosion simulation in Cobol by
> people with expertise in ISAM files and BCD would probably not
> go well either.

And that was my point. Use Domain Specific Languages and use people
who understand what they are doing to write the program. Not general
purpose languages written by people with no specific expertise at all.

bill

On 11/5/2023 6:53 PM, Chris Townley wrote:
> On 05/11/2023 19:11, Arne Vajhøj wrote:
>> On 11/5/2023 1:23 PM, bill wrote:
>>> On 11/5/2023 12:47 PM, Arne Vajhøj wrote:
>>>> On 11/5/2023 10:58 AM, bill wrote:
>
> <Big snip>
> I thought that MicroFocus Cobol was getting a lot of interest a few
> years back
>

Still does.

bill

On 11/5/2023 5:25 PM, Jake Hamby (Solid State Jake) wrote:
> I pushed some work in progress to my GitHub repo for the VMS port of
> libuv that I started to work on. I'm not sure if it should be
> considered a bug in DECthreads that it behaves so impossibly poorly
> on the async ping test that writes a byte to a pipe to wake a child
> sleeping on a poll() and then sleeps on poll() to wait for the reply
> on a pipe going the other way. I'm getting no more than 60 round
> trips / sec, even after commenting out the sched_yield() call before
> it goes back to sleep. Being so unworkably slow using the POSIX APIs,
> I didn't bother to try to run the test cases.
>
> My guess would be that VMS's user-mode thread scheduler doesn't
> realize immediately that one of the benchmark process's threads has
> signaled another one to wake up from a poll() by writing to a pipe
> shared between them, and it's going to sleep on a timer before
> eventually waking up. The poll() version of the benchmark uses almost
> no CPU because it's mostly sleeping. You can get instant inter-thread
> wakeup with local event flags, and as I mentioned at the start of
> this thread, I did see above 10K round-trips/sec with a hacked-up
> version of libuv's async benchmark to test the do-nothing wake-up
> case.

It is not that unsual that when trying to emulate OS A on OS B,
then performance suffers.

> What makes libuv such an interesting test is that the naive port
> using the C RTL synchronous POSIX APIs is almost unusably bad, and in
> the best-case scenario has too many unneeded layers of emulation, but
> by using the Win32 and Linux io_uring implementations as a model,
> it's possible to get acceptable behavior using all native VMS APIs,
> with completion ASTs waking up the event loops using one local event
> flag per event loop.

That optimal performance require native OS features follows from above.

> App servers like Node.js either have only a
> single event loop, or one per core, so VMS's limit of 48 (64 - 8
> reserved) should be more than adequate.

First I am not sure how you get 64 - 8 to be 48. My calculator
claims 56.

:-)

But secondly the number of threads in an app server vary a
lot between app servers.

If you go to the Java world (servlet containers like Tomcat
or full application servers like WildFly) then expect to see
a lot more threads. From like 100 for development server to
like 1000 for production server.

> So that gets back to RMS vs. $QIO(W)/$IO_PERFORM(W). Since nobody
> cares about anything other than Stream_LF sequential files in UNIX
> world, there's nothing advanced that we need or want from RMS as far
> as record handling,

Not so sure about that.

Application that only work with STREAM_LF files is a PITA. Unfortunately
not unheard off. But no need to produce any more.

> but we do want to perform reads/writes of any
> size starting from any offset. If you use $QIO(W), then, like
> $IO_PERFORM(W), you must access file data by virtual block offset
> (starting at block 1, not 0), and then you're effectively doing raw
> disk I/O, except with the OS mapping only the blocks in the file you
> have access to through your channel.
>
> The reason C RTL and everyone else go through RMS is so they can do
> byte-level file I/O instead of block-level, and to take advantage of
> XFC caching.

I don't think XFC require RMS.

> Everyone keeps talking about raw throughput and database
> servers and not the much more common use cases for file I/O.

On VMS today a lot of applications use either index-sequential files
or some custom access to sequential files.

But in the general IT industry today the performance usually depend
on some sort of database (RDBMS or NoSQL) not flat files.

(index-sequential files are really a NoSQL database, but code wide
it is together with the rest of RMS on VMS)

Arne

On 11/5/2023 8:41 PM, bill wrote:
> On 11/5/2023 7:39 PM, Arne Vajhøj wrote:
>> So the Fortran code still exist, but the scientists don't use
>> it and don't even see it.
>
>
> I have to admit I have never seen anything about Python generating or
> being converted into Fortran.

Neither.

But Python code may (and for scientific computing usually do) use
libraries written in Fortran.

Arne

On 11/5/2023 9:09 PM, Arne Vajhøj wrote:
> On 11/5/2023 8:41 PM, bill wrote:
>> On 11/5/2023 7:39 PM, Arne Vajhøj wrote:
>>> So the Fortran code still exist, but the scientists don't use
>>> it and don't even see it.
>>
>>
>> I have to admit I have never seen anything about Python generating or
>> being converted into Fortran.
>
> Neither.
>
> But Python code may (and for scientific computing usually do) use
> libraries written in Fortran.

Oh... If all your talking about is libraries there really isn't any
reason I can think of why any language can't call libraries written in
a different language. I know that I can certainly (and frequently do)
call C libraries from COBOL. And back in the good old days we used to
call the math functions from NAGLIB in lots of different languages.

But I have to ask the question... If Python is so good why are the
libraries written in Fortran?

bill

On 11/5/2023 9:18 PM, bill wrote:
> On 11/5/2023 9:09 PM, Arne Vajhøj wrote:
>> On 11/5/2023 8:41 PM, bill wrote:
>>> On 11/5/2023 7:39 PM, Arne Vajhøj wrote:
>>>> So the Fortran code still exist, but the scientists don't use
>>>> it and don't even see it.
>>>
>>> I have to admit I have never seen anything about Python generating or
>>> being converted into Fortran.
>>
>> Neither.
>>
>> But Python code may (and for scientific computing usually do) use
>> libraries written in Fortran.
>
> Oh...  If all your talking about is libraries there really isn't any
> reason I can think of why any language can't call libraries written in
> a different language.  I know that I can certainly (and frequently do)
> call C libraries from COBOL.  And back in the good old days we used to
> call the math functions from NAGLIB in lots of different languages.

It is quite common.

The interesting aspect here was not the use of a library
in another language, but that the "new" language in the domain
Python under the hood use the "old" language of the domain
Fortran.

> But I have to ask the question...  If Python is so good why are the
> libraries written in Fortran?

Different languages are good for different things.

Python may only require 1/10'th of the lines of code
that Fortran does to express where to load matrices
from and how to multiply and invert them, but Fortran
may be a factor 100 faster actually doing the matrix
multiplications and inversions, because it is compiled
unlike Python that (commonly) is interpreted.

So expressing what you want done in Python and
using Fortran libraries combine the best of both
languages.

Arne

On 2023-11-05 15:28, Arne Vajhøj wrote:
> On 11/3/2023 8:35 PM, Jake Hamby (Solid State Jake) wrote:
>> Something else just occurred to me. While file I/O performance is
>> important, as well as local IPC, I've also been keeping in mind
>> something else: simultaneous socket connections. One benefit of
>> everyone going to RESTful APIs is you're not keeping a long-lived TCP
>> socket open like you would with other types of protocols.
>
> HTTP does support keep-alive.
>
> And RESTful web services do use that. Especially for high volume
> server to server setups.

Indeed. Since RESTful APIs are rather inefficient... Especially if you
have to setup new connections all the time.

Johnny

On 2023-11-05 16:58, bill wrote:

> We have so many "colleges" teaching trade school courses (like diesel
> mechanics, HVAC welding and even motorcycle mechanics)I really wish
> trade schools would step up to the plate ad start teaching IT and in
> particular thing like COBOL, Fortran and PL/I.  They are not going away.

Academia should not teach languages. If they do, they are clearly not
doing the right thing.

They should teach methods, principles, concepts, ideas.

The language is just a tool. You need to learn and use different tools
all the time. That you could/should learn at the place where it is
used/needed. And if you have all the teachings from academia, that
should be an easy thing.

Johnny