On 10/7/2021 8:50 AM, Craig A. Berry wrote:
> On 10/6/21 11:10 PM, Dave Froble wrote:
>> the real issue is that SSIO was aimed (it seems) at PostgreSQL.
>
> And Apache, and Samba, and other things that have been explicitly
> mentioned as having needed app-specific workarounds due to the absence
> of shared stream I/O support. SSIO *is* the general-purpose solution
> that you seem to be lamenting the lack of.

Samba I totally get.

Multiple PC's writing to a file on a Samba share would create
some interesting scenarios.

But why does Apache need it?

It should read files to serve - and since it is serving VMS files
then I think it be as VMSish as possible so totally standard RMS.
And it should write sequential text files like access.log.

What am I missing?

Arne

On 10/7/21 8:40 AM, Arne Vajhøj wrote:
> On 10/7/2021 8:50 AM, Craig A. Berry wrote:
>> On 10/6/21 11:10 PM, Dave Froble wrote:
>>> the real issue is that SSIO was aimed (it seems) at PostgreSQL.
>>
>> And Apache, and Samba, and other things that have been explicitly
>> mentioned as having needed app-specific workarounds due to the absence
>> of shared stream I/O support. SSIO *is* the general-purpose solution
>> that you seem to be lamenting the lack of.
>
> Samba I totally get.
>
> Multiple PC's writing to a file on a Samba share would create
> some interesting scenarios.
>
> But why does Apache need it?
>
> It should read files to serve - and since it is serving VMS files
> then I think it be as VMSish as possible so totally standard RMS.
> And it should write sequential text files like access.log.
>
> What am I missing?

log files (and probably the fact that multiple worker processes can be
writing to the same logs). And I forgot to mention that Java needs it
too. See:

<http://de.openvms.org/TUD2012/opensource_and_unix_portability.pdf>

Page 16 says:

• Java (CIFS too) uses a work-around
− Does open+read/write+close for every read/write!
− Restores current file offset after each close+open
− Significant performance issue
• Oracle problem with log and trace files
− Single writer with multiple readers
• Apache’s use of log files sub-optimal
− V1.3 places restriction
− V2.0 uses a work-around

On 10/7/2021 9:51 AM, Craig A. Berry wrote:
> On 10/7/21 8:40 AM, Arne Vajhøj wrote:
>> On 10/7/2021 8:50 AM, Craig A. Berry wrote:
>>> On 10/6/21 11:10 PM, Dave Froble wrote:
>>>> the real issue is that SSIO was aimed (it seems) at PostgreSQL.
>>>
>>> And Apache, and Samba, and other things that have been explicitly
>>> mentioned as having needed app-specific workarounds due to the absence
>>> of shared stream I/O support. SSIO *is* the general-purpose solution
>>> that you seem to be lamenting the lack of.
>>
>> Samba I totally get.
>>
>> Multiple PC's writing to a file on a Samba share would create
>> some interesting scenarios.
>>
>> But why does Apache need it?
>>
>> It should read files to serve - and since it is serving VMS files
>> then I think it be as VMSish as possible so totally standard RMS.
>> And it should write sequential text files like access.log.
>>
>> What am I missing?
>
> log files (and probably the fact that multiple worker processes can be
> writing to the same logs).

I still don't get it.

I thought SSIO was about shared access to byte streams.

Writing to log files should be fine using good old record based
writes (somewhere down the call stack SYS$PUT).

>   And I forgot to mention that Java needs it
> too.  See:
>
> <http://de.openvms.org/TUD2012/opensource_and_unix_portability.pdf>
>
> Page 16 says:
>
> • Java (CIFS too) uses a work-around
>   − Does open+read/write+close for every read/write!
>   − Restores current file offset after each close+open
>   − Significant performance issue

In this context does "Java" mean "Tomcat"?

Arne

On 10/7/2021 9:34 AM, Arne Vajhøj wrote:
>         fseek(fp, ix, SEEK_SET);

> var.txt size = 14 bytes
> var.txt sequential read (text): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43 8=43
> 9=0A
> var.txt sequential read (binary): 1=41 2=42 3=42 4=43 5=43 6=43
> var.txt direct read (text): 1=41 2=-1 3=02 4=-1 5=42 6=-1 7=-1 8=-1 9=43
> 10=-1 11=-1 12=-1 13=FF 14=-1
> var.txt direct read (binary): 1=41 2=-1 3=02 4=-1 5=42 6=-1 7=-1 8=-1
> 9=43 10=-1 11=-1 12=-1 13=FF 14=-1
> $ mcr sys$disk:[]process stmlf.txt
> stmlf.txt size = 9 bytes
> stmlf.txt sequential read (text): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43
> 8=43 9=0A
> stmlf.txt sequential read (binary): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43
> 8=43 9=0A
> stmlf.txt direct read (text): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43 8=43 9=0A
> stmlf.txt direct read (binary): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43 8=43
> 9=0A

In all fairness then I believe there are some documentation
somewhere that states that fseek is only supported to
beginning of a record. I cannot find it right now,
but I believe I once saw it somewhere.

Arne

On 10/7/2021 3:54 AM, Lawrence D’Oliveiro wrote:
> On Thursday, October 7, 2021 at 5:12:58 PM UTC+13, Dave Froble
> wrote:
>> Frankly, (and yes, I'm biased), I find records reasonable, and a
>> stream of bytes baffling and confusing. Guess it's what one is used
>> to.
>
> Trouble is, there are many binary file formats that do not map easily
> to a simple sequence of records (of whatever delimitation). Consider
> the IFF family of file formats, for example: these are built out of
> chunks, and certain chunk types can contain other chunks.
>
> For another example, consider file formats like TIFF and TTF, where
> there is a directory that identifies the location and size of the
> various major pieces. Oh, and PDF comes under this as well.

The whole record things is mostly for text files and RMS based
database style usage.

Even on VMS then true binary files are usually FIX 512 (or in rare
cases UDF) with the structure handled entirely by the application.

Attempts to do otherwise often end up with 32K problems.

> And then there are text-based format families, like XML, JSON, YAML,
> TOML ...

Different. Both on *nix and VMS that is a separate structure
on top of the basic file format.

Arne

On 2021-10-07 01:25:57 +0000, Greg Tinkler said:

> My point is SSIO seems to be focused on just PostgreSQL, whereas an RMS
> solution is much much easier to program, uses well tested code, and is
> already cluster ready putting the team ahead of the game and not
> building issues for the future.

....

Fix the existing RMS data corruption in 32-bit RMS and/or in the C
library, and get PostgreSQL available on OpenVMS soonest. I expect this
is the priority for VSI.

Everything else is aspirational.

Integrate stream file access support at the XQP and allow C and C++ and
other non-punched-card-style app designs and stream- and OO-focused
languages to optionally bypass RMS entirely.

Better integrate and document the existing range-locking support
available within DLM.

And in aggregate, stop trying to make the current 32-bit RMS NoSQL
database more complex than it already is, and re-architect such that
32-bit RMS NoSQL database becomes just another available database, and
preferably while providing room for 64-bit RMS rather than trying
another OpenVMS Alpha V7.0-style 32-/64-bit or FAB/RAB/RAB64/NAM/NAML
design, and make 32- or (hypothetical) 64-bit RMS not the sole
persistent-storage "funnel" for structured file access for apps running
on OpenVMS, short of those few using XQP or LOG_IO or PHY_IO. Existing
RMS apps are already headed for "fun" as part of the upcoming 64-bit
LBN work for VSI and for apps, and a whole lot of those apps just won't
make it past messes similar to apps still tied to ODS-2 naming. I'd
wager that most existing apps don't yet fully support ODS-5 naming,
UTF-8 and all, too. Similar app messes with latent 32-bit RMS
dependencies.

--
Pure Personal Opinion | HoffmanLabs LLC

On Thursday, October 7, 2021 at 3:54:53 AM UTC-4, Lawrence D’Oliveiro wrote:
> Trouble is, there are many binary file formats that do not map easily to a simple sequence of records (of whatever delimitation). Consider the IFF family of file formats, for example: these are built out of chunks, and certain chunk types can contain other chunks.

Whatever happened to Compound Document Architecture (CDA)? It always struck me as an effort (now abandoned) toward an object oriented file structure.

On 10/6/2021 10:00 PM, Lawrence D’Oliveiro wrote:
> Given that nowadays an SQL-based RDBMS like SQLite can offer full
> support for transactions, joins and subqueries (missing only more
> multi-user-type features like locking and replication), and yet still
> be resource-light enough to fit in your mobile phone, I would say the
> time for application developers to be grubbing about in ISAM files is
> past.

There are still cases where it make sense. RMS index-sequential files
are really a NoSQL Key Value Store in modern terminology and
they are still used and new ones even being developed (like
RocksDB).

But the default should change.

"use index-sequential file unless good reason to use relational database"

=>

"use relational database unless good reason to use
index-sequential file"

Arne

On 2021-10-07 07:54:51 +0000, Lawrence D’Oliveiro said:

> On Thursday, October 7, 2021 at 5:12:58 PM UTC+13, Dave Froble wrote:
>> Frankly, (and yes, I'm biased), I find records reasonable, and a stream
>> of bytes baffling and confusing. Guess it's what one is used to.
>
> Trouble is, there are many binary file formats that do not map easily
> to a simple sequence of records (of whatever delimitation). Consider
> the IFF family of file formats, for example: these are built out of
> chunks, and certain chunk types can contain other chunks.
>
> For another example, consider file formats like TIFF and TTF, where
> there is a directory that identifies the location and size of the
> various major pieces. Oh, and PDF comes under this as well.
>
> And then there are text-based format families, like XML, JSON, YAML, TOML ...

There are many examples. It's far easier to map a whole executable
image into virtual memory or to use file system calls to load the whole
image into virtual memory, too. (This is an app design I never would
have considered on a VAX, too.)

For a number of apps and designs, I find RMS problematic for its
fondness for records in the lower parts of its position within the I/O
stack "funnel", and problematic again at somewhat higher levels of the
I/O stack "funnel" with what little RMS can do with those database
records it wants to enforce; its lack of marshaling and unmarshaling
for apps needing those services, among other sorts of designs, and with
all the usual "fun" with making changes to the contents and formats of
RMS records within apps.

Trying to make all apps fit within one NoSQL database really isn't all
that great of a solution. Getting PostgreSQL, SQLite, and other
databases better integrated is helpful. Longer-term and as I'd
mentioned in another reply, demoting 32-bit RMS to "just another local
database" status, too.

And to be absolutely clear here: if an app developer needs a NoSQL
database and as many apps can, having 32-bit RMS is entirely useful. At
least until the app developer needs to make changes or additions to the
record structures, when 32-bit RMS starts showing its age. A problem
related to how we now have roughly two-dozen files necessary within a
cluster configuration.

--
Pure Personal Opinion | HoffmanLabs LLC

On 10/7/21 9:42 AM, Arne Vajhøj wrote:
> On 10/7/2021 9:34 AM, Arne Vajhøj wrote:
>>          fseek(fp, ix, SEEK_SET);
>
>> var.txt size = 14 bytes
>> var.txt sequential read (text): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43
>> 8=43 9=0A
>> var.txt sequential read (binary): 1=41 2=42 3=42 4=43 5=43 6=43
>> var.txt direct read (text): 1=41 2=-1 3=02 4=-1 5=42 6=-1 7=-1 8=-1
>> 9=43 10=-1 11=-1 12=-1 13=FF 14=-1
>> var.txt direct read (binary): 1=41 2=-1 3=02 4=-1 5=42 6=-1 7=-1 8=-1
>> 9=43 10=-1 11=-1 12=-1 13=FF 14=-1
>> $ mcr sys$disk:[]process stmlf.txt
>> stmlf.txt size = 9 bytes
>> stmlf.txt sequential read (text): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43
>> 8=43 9=0A
>> stmlf.txt sequential read (binary): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43
>> 8=43 9=0A
>> stmlf.txt direct read (text): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43 8=43
>> 9=0A
>> stmlf.txt direct read (binary): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43
>> 8=43 9=0A
>
> In all fairness then I believe there are some documentation
> somewhere that states that fseek is only supported to
> beginning of a record. I cannot find it right now,
> but I believe I once saw it somewhere.

Is this what you're looking for?

$ help crtl fseek description

CRTL

fseek

Description

The fseek function can position a fixed-length record-access
file with no carriage control or a stream-access file on any
byte offset, but can position all other files only on record
boundaries.

The available Standard I/O functions position a variable-length
or VFC record file at its first byte, at the end-of-file, or on
a record boundary. Therefore, the arguments given to fseek must
specify any of the following:

o The beginning or end of the file

o A 0 offset from the current position (an arbitrary record
boundary)

o The position returned by a previous, valid ftell call

On 2021-10-07 15:37:04 +0000, David Jones said:

> On Thursday, October 7, 2021 at 3:54:53 AM UTC-4, Lawrence D’Oliveiro wrote:
>> Trouble is, there are many binary file formats that do not map easily
>> to a simple sequence of records (of whatever delimitation). Consider
>> the IFF family of file formats, for example: these are built out of
>> chunks, and certain chunk types can contain other chunks.
> Whatever happened to Compound Document Architecture (CDA)? It always
> struck me as an effort (now abandoned) toward an object oriented file
> structure.

DEC ceded the desktop app business.

The modern equivalent to CDA is PDF.

--
Pure Personal Opinion | HoffmanLabs LLC

On 10/7/21 9:01 AM, Arne Vajhøj wrote:
> On 10/7/2021 9:51 AM, Craig A. Berry wrote:
>> On 10/7/21 8:40 AM, Arne Vajhøj wrote:
>>> On 10/7/2021 8:50 AM, Craig A. Berry wrote:
>>>> On 10/6/21 11:10 PM, Dave Froble wrote:
>>>>> the real issue is that SSIO was aimed (it seems) at PostgreSQL.
>>>>
>>>> And Apache, and Samba, and other things that have been explicitly
>>>> mentioned as having needed app-specific workarounds due to the absence
>>>> of shared stream I/O support. SSIO *is* the general-purpose solution
>>>> that you seem to be lamenting the lack of.
>>>
>>> Samba I totally get.
>>>
>>> Multiple PC's writing to a file on a Samba share would create
>>> some interesting scenarios.
>>>
>>> But why does Apache need it?
>>>
>>> It should read files to serve - and since it is serving VMS files
>>> then I think it be as VMSish as possible so totally standard RMS.
>>> And it should write sequential text files like access.log.
>>>
>>> What am I missing?
>>
>> log files (and probably the fact that multiple worker processes can be
>> writing to the same logs).
>
> I still don't get it.
>
> I thought SSIO was about shared access to byte streams.
>
> Writing to log files should be fine using good old record based
> writes (somewhere down the call stack SYS$PUT).

Don't ask me, ask the authors of the document to which I linked. Or the
folks at VSI who inherited their work. I may be wrong and it's not
about log files, but suppose it is. If you start from the premise that
the log files are stream-oriented and you have multiple writers and
multiple readers at the same time, then that's pretty much the
definition of shared access to a byte stream. Doing it differently for a
platform that prefers records would be extra cost and extra maintenance.

>>                            And I forgot to mention that Java needs it
>> too.  See:
>>
>> <http://de.openvms.org/TUD2012/opensource_and_unix_portability.pdf>
>>
>> Page 16 says:
>>
>> • Java (CIFS too) uses a work-around
>>    − Does open+read/write+close for every read/write!
>>    − Restores current file offset after each close+open
>>    − Significant performance issue
>
> In this context does "Java" mean "Tomcat"?

You know as much as I do -- probably more ;-).

On 10/7/2021 8:50 AM, Craig A. Berry wrote:
>
> On 10/6/21 11:10 PM, Dave Froble wrote:
>
>> the real issue is that SSIO was aimed (it seems) at PostgreSQL.
>
> And Apache, and Samba, and other things that have been explicitly
> mentioned as having needed app-specific workarounds due to the absence
> of shared stream I/O support. SSIO *is* the general-purpose solution
> that you seem to be lamenting the lack of.
>

A while back we were discussing doing away with I/O to buffers, and
accessing the data in place. Slower access perhaps, but doing away with
the reading and writing to/from buffers. Haven't heard much about that
lately. I don't get out much.

Such type of activity would really benefit from having the capability of
locking just the required data, and, would need the capability of
reading and writing just the required data.

I'm aware of how useful something like SSIO would be. I'm just appalled
by the design and implementation. As mentioned, it seems aimed at just
a few current uses, and totally ignores how useful it would be for many
more future uses. This is rather consistent with the long time apathy
with which VMS has been treated. It's more a patch than an enhancement.
This is what I lament.

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

On 10/7/2021 10:01 AM, Arne Vajhøj wrote:

>
> I still don't get it.
>
> I thought SSIO was about shared access to byte streams.

That is a bit of tunnel vision.

Locking numeric ranges could be used for many other things. Such a
capability should be generic, not just for a single purpose.

That's the problem I see, the tunnel vision when approaching the issue,
rather than the vision to see just how useful the capability could be.

Craig's post points that out.

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

On 10/7/2021 12:01 PM, Craig A. Berry wrote:
> On 10/7/21 9:42 AM, Arne Vajhøj wrote:
>> On 10/7/2021 9:34 AM, Arne Vajhøj wrote:
>>>          fseek(fp, ix, SEEK_SET);
>>
>>> var.txt size = 14 bytes
>>> var.txt sequential read (text): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43
>>> 8=43 9=0A
>>> var.txt sequential read (binary): 1=41 2=42 3=42 4=43 5=43 6=43
>>> var.txt direct read (text): 1=41 2=-1 3=02 4=-1 5=42 6=-1 7=-1 8=-1
>>> 9=43 10=-1 11=-1 12=-1 13=FF 14=-1
>>> var.txt direct read (binary): 1=41 2=-1 3=02 4=-1 5=42 6=-1 7=-1 8=-1
>>> 9=43 10=-1 11=-1 12=-1 13=FF 14=-1
>>> $ mcr sys$disk:[]process stmlf.txt
>>> stmlf.txt size = 9 bytes
>>> stmlf.txt sequential read (text): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43
>>> 8=43 9=0A
>>> stmlf.txt sequential read (binary): 1=41 2=0A 3=42 4=42 5=0A 6=43
>>> 7=43 8=43 9=0A
>>> stmlf.txt direct read (text): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43 8=43
>>> 9=0A
>>> stmlf.txt direct read (binary): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43
>>> 8=43 9=0A
>>
>> In all fairness then I believe there are some documentation
>> somewhere that states that fseek is only supported to
>> beginning of a record. I cannot find it right now,
>> but I believe I once saw it somewhere.
>
> Is this what you're looking for?
>
> $ help crtl fseek description
>
> CRTL
>
>   fseek
>
>     Description
>
>          The fseek function can position a fixed-length record-access
>          file with no carriage control or a stream-access file on any
>          byte offset, but can position all other files only on record
>          boundaries.
>
>          The available Standard I/O functions position a variable-length
>          or VFC record file at its first byte, at the end-of-file, or on
>          a record boundary. Therefore, the arguments given to fseek must
>          specify any of the following:
>
>          o  The beginning or end of the file
>
>          o  A 0 offset from the current position (an arbitrary record
>             boundary)
>
>          o  The position returned by a previous, valid ftell call

YES.

And shame on me, because I only checked help crtl fseek arguments.

Arne

On 10/7/2021 8:54 AM, Greg Tinkler wrote:
>
>> Well the perceived issue is what happens when taking out locks, and at
>> some point there is a conflict. Say needing 127 blocks locked, and the
>> conflict is on the last block. That means 126 locks to be released, and
>> perhaps try again.
> Maybe, maybe not. It depends on the locking fan out factors for the differing levels. It is possible that only 1 lock is needed, may be more, the wort case would be 127. NB there is also BLAST to assist with managing the lock promotion/demotions.
>
>
>> As long as storage is block oriented, then regardless of the numeric
>> range of bytes, all blocks encompassing the byte range will need to be
>> read, including locking, and written. This usually will include data
>> outside the byte range.
>
> Yup, as is the case on Unix...let the drivers worry about how and why this is done, block/byte what ever the IO device needs.
>
>> Forget RMS, I/O would be at the QIO level.
>
> Why? Underneath RMS is QIO, what RMS gives us the the coordination of the buffers/buckets/clumps/block across the cluster to ensure not lost updates, as per the example used to justify SSIO.

Too limited and specific purpose. RMS might be able to make use of some
capabilities, but so might other applications.

RMS does some things well, and doesn't have some capabilities that it
perhaps should have. Data field definitions in records comes to mind.

>> RMS keyed files can have variable record lengths.
> True, VAR only not VFC or STM*, but fixed length key fields, with fixed offsets in the record
>
>> RMS relative files require fixed length records. (if I remember correctly)
> Yup, there are implicitly fixed length.
>
> ===
> Have been thinking about the byte range locking. As most of the use will be for locking ranges in a file it should be integrated with RMS, i.e. RMS should have an API to allow this as it already does the locking to the buffer/bucket/clump/block. Just need another 1 or 2 layers of lock tree and you have it. And it all be cluster wide, and it will be compatible with other users of RMS.

Short sighted thinking. Numeric range locking might be useful in many
applications.

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

On 10/7/2021 12:27 PM, Dave Froble wrote:
> On 10/7/2021 10:01 AM, Arne Vajhøj wrote:
>> I still don't get it.
>>
>> I thought SSIO was about shared access to byte streams.
>
> That is a bit of tunnel vision.

Not really. More like the definition.

<quote>
SSIO
====
Shared Stream IO feature provides POSIX compliant read/write to byte
stream files.
Hence SSIO feature, the data consistency is guaranteed when mutiple
processes are performing a Read/Write to non overlapping byte boundaries
with the same block boundary.
</quote>

> Locking numeric ranges could be used for many other things.  Such a
> capability should be generic, not just for a single purpose.

I agree that range locking is a useful feature for many other purposes
than SSIO.

> That's the problem I see, the tunnel vision when approaching the issue,
> rather than the vision to see just how useful the capability could be.
>
> Craig's post points that out.

It listed some project that could benefit from SSIO besides
PostgreSQL.

And I just don't understand some of the examples since they
sound traditional record oriented to me.

Arne

On 10/7/2021 8:59 AM, Simon Clubley wrote:
> On 2021-10-07, Greg Tinkler <tinklerg@gmail.com> wrote:
>> On Thursday, 7 October 2021 at 11:26:38 pm UTC+11, Simon Clubley wrote:
>>
>>> How do you find byte 12,335,456 in a variable length RMS sequential file
>>> without reading from the start of the file ?
>>>
>>> That's why there are restrictions on RMS supported file formats in an
>>> application in some cases.
>>
>> The same way it is done on Unix, calculate the block offset, go get it, and extract the byte. no difference and nothing to do with the underlying format.
>>
>
> You don't know the block offset without scanning the file when it comes
> to some RMS file formats.
>
> IOW, data byte 12,335,456 will not be the same thing as file byte 12,335,456
> unless you restrict yourself to record formats that do not have embedded
> record metadata.
>
> Simon.
>

I'm guessing Unix files don't have metadata and such. So the comparison
is not valid.

For a non-RMS file, yes, the location can be calculated. But not so for
an RMs file with record characteristics included in the records.

Since Unix doesn't have RMS files, perhaps that confused Greg.

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

On 2021-10-07 16:18:28 +0000, Dave Froble said:

> A while back we were discussing doing away with I/O to buffers, and
> accessing the data in place. Slower access perhaps, but doing away
> with the reading and writing to/from buffers. Haven't heard much about
> that lately. I don't get out much.

Ayup. Nonvolatile byte-addressable storage hardware is available now,
and is in use in various applications.

Compatible memory hardware will be rather more available for OpenVMS
x86-64, for folks interested in investigating this for their apps.

Carving out a hunk of persistent storage will be interesting topic for
app developers on OpenVMS, though I can think of a couple of ways to
try.

Here's an HPE overview from a few years ago on the topic:
https://www.pdl.cmu.edu/SDI/2016/slides/keeton-2016-10-19-memory-driven-computing.pdf

I see some B-Tree work for this area in a newer paper, and a number of
other discussions.

> Such type of activity would really benefit from having the capability
> of locking just the required data, and, would need the capability of
> reading and writing just the required data.

Locking access to the contents of a global section, or locking access
to hardware-backed storage for external devices, is the same issue.

Whether DLM overhead is too high for that to be workable is another
discussion that the app developers will want to ponder.

> I'm aware of how useful something like SSIO would be. I'm just
> appalled by the design and implementation. As mentioned, it seems
> aimed at just a few current uses, and totally ignores how useful it
> would be for many more future uses. This is rather consistent with the
> long time apathy with which VMS has been treated. It's more a patch
> than an enhancement. This is what I lament.

Alas, there's no other outcome when upward-compatibility is an
overarching goal for the platform.

--
Pure Personal Opinion | HoffmanLabs LLC

On 10/7/2021 9:34 AM, Arne Vajhøj wrote:

> I suspect that the variable length file output below will
> surprise a few *nix developers.
>

Why do you post C code examples that confuse me and give me a headache?

:-)

Then again, Basic code examples might confuse Unix developers ...

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

On 10/7/2021 1:00 PM, Dave Froble wrote:
> On 10/7/2021 9:34 AM, Arne Vajhøj wrote:
>> I suspect that the variable length file output below will
>> surprise a few *nix developers.
>
> Why do you post C code examples that confuse me and give me a headache?
>
> :-)
>
> Then again, Basic code examples might confuse Unix developers ...

Sorry about the headache.

But the topic was identical code on *nix and VMS trying to
access a random position in a file.

C is available on both *nix and VMS so it was rather
obvious.

VMS Basic is not available on *nix.

I don't think there is quite the same options
in VMS Basic as in C for this, but I expect all the
options available in VMS Basic to produce a natural
expected result.

Arne

On 10/7/2021 11:34 AM, Stephen Hoffman wrote:
> On 2021-10-07 01:25:57 +0000, Greg Tinkler said:
>
>> My point is SSIO seems to be focused on just PostgreSQL, whereas an
>> RMS solution is much much easier to program, uses well tested code,
>> and is already cluster ready putting the team ahead of the game and
>> not building issues for the future.
>
> ...
>
> Fix the existing RMS data corruption in 32-bit RMS and/or in the C
> library, and get PostgreSQL available on OpenVMS soonest. I expect this
> is the priority for VSI.

Most likely.

> Everything else is aspirational.
>
> Integrate stream file access support at the XQP and allow C and C++ and
> other non-punched-card-style app designs and stream- and OO-focused
> languages to optionally bypass RMS entirely.

I don't use C, so I don't know much about it. But isn't this capability
already available? Even RMS has the BLOCK I/O capability, at least from
Basic.

As far as I know, QIO doesn't know a thing about RMS. Well, the
directory structure does know RMS, and to an extent is RMS.

> Better integrate and document the existing range-locking support
> available within DLM.

Yes, for sure. And if needed, make it much better.

> And in aggregate, stop trying to make the current 32-bit RMS NoSQL
> database more complex than it already is, and re-architect such that
> 32-bit RMS NoSQL database becomes just another available database, and
> preferably while providing room for 64-bit RMS rather than trying
> another OpenVMS Alpha V7.0-style 32-/64-bit or FAB/RAB/RAB64/NAM/NAML
> design, and make 32- or (hypothetical) 64-bit RMS not the sole
> persistent-storage "funnel" for structured file access for apps running
> on OpenVMS, short of those few using XQP or LOG_IO or PHY_IO. Existing
> RMS apps are already headed for "fun" as part of the upcoming 64-bit LBN
> work for VSI and for apps, and a whole lot of those apps just won't make
> it past messes similar to apps still tied to ODS-2 naming. I'd wager
> that most existing apps don't yet fully support ODS-5 naming, UTF-8 and
> all, too. Similar app messes with latent 32-bit RMS dependencies.

Oh, no, Steve. That is much too logical and reasonable. Can't have
that. We must insure that things stay totally screwed up.

Don't know how far work had progressed on alternate file systems. Might
or might not help to make RMS "just another capability". But, doing
what you suggest would go a long way toward making VMS more useful in
the future.

I've got the suspicion that VMS clusters, while good, create some of the
problems in attempting to add new capabilities to VMS. Need I mention
"MOUNT"? Better segregation might help to add new and different
capabilities. Not sure how easy that might be.

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

On 10/7/2021 1:16 PM, Dave Froble wrote:
> On 10/7/2021 11:34 AM, Stephen Hoffman wrote:
>> Integrate stream file access support at the XQP and allow C and C++ and
>> other non-punched-card-style app designs and stream- and OO-focused
>> languages to optionally bypass RMS entirely.
>
> I don't use C, so I don't know much about it.  But isn't this capability
> already available?  Even RMS has the BLOCK I/O capability, at least from
> Basic.

C/C++ and most newer languages have a "stream view" of files while
RMS has a "record view" of files.

If they used different file systems everything would be fine.

If all text files are STMLF then it works and the "stream view"
and the "record view" produces consistent results.

But trying to mix on variable length or VFC files becomes
a minefield.

I know you don't like C, but try look at the example I posted.
Some of the outputs are very weird.

Arne

On 10/7/2021 12:12 PM, Craig A. Berry wrote:
> On 10/7/21 9:01 AM, Arne Vajhøj wrote:
>> I still don't get it.

> Don't ask me, ask the authors of the document to which I linked. Or the
> folks at VSI who inherited their work.

I know - I should not shoot the messenger. Sorry.

Arne

On 10/7/2021 11:50 AM, Arne Vajhøj wrote:
> On 10/6/2021 10:00 PM, Lawrence D’Oliveiro wrote:
>> Given that nowadays an SQL-based RDBMS like SQLite can offer full
>> support for transactions, joins and subqueries (missing only more
>> multi-user-type features like locking and replication), and yet still
>> be resource-light enough to fit in your mobile phone, I would say the
>> time for application developers to be grubbing about in ISAM files is
>> past.
>
> There are still cases where it make sense. RMS index-sequential files
> are really a NoSQL Key Value Store in modern terminology and
> they are still used and new ones even being developed (like
> RocksDB).
>
> But the default should change.
>
> "use index-sequential file unless good reason to use relational database"
>
> =>
>
> "use relational database unless good reason to use
> index-sequential file"
>
> Arne
>
>

I'd suggest there should not be a "default". Rather, make good
thoughtful decisions. Have valid reasons for any decisions or choices.

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486