I notice that SSIO (beta) in included in an up coming V9.1 field test. So I read up on the issues it is trying to solve.

One concerning thing was to have CRTL (via SSIO) access directly to XFC. From an architectural point of view this is wrong at so many levels, but if that is what needs to happen then open it up so RMS and other code bases can use it.

The main reason stated was the need to do byte offset/count IO’s. Well lets solve that first, change RMS by adding SYS$READB and SYS$WRITEB. These would be useful to all code using RMS.
SYS$READB read from byte offset for count, return latest data from that byte range.
SYS$WRITEB write from byte offset for count, update latest copy of underlying blocks.

SYS$WRITEB needs to use latest copy of data, and could use the new SSIO interface to XFC but RMS has it's own methods for this.
It may seem like a big ask getting all the latest blocks, but if you think about it it only needs to re-read the last and first block if it does not already have the latest copy. Also no need if the offset starts at the beginning of a block, and it fills the last block.

By having these as part of RMS we want to ensure the blocks/buffers are coordinated so any other user of RMS will see the changes, and we get their changes.

This seems to be at the core of the CRTL issue, it does NOT use RMS, nor does it synchronize its blocks/buffers, leading to the lost update problem.

So with this ‘simple’ addition the CRTL should be altered to us RMS for all file IO.

An extra that could be added, if the file is RFM=fixed, and the C code uses it that way with the same record length then use the SYS$GET/SYS$PUT so it will play nicely with an RMS access to those files.

Anyway just my 2 cent worth.

gt down under

On 2021-10-06 02:06:29 +0000, Greg Tinkler said:

> I notice that SSIO (beta) in included in an up coming V9.1 field test.
> So I read up on the issues it is trying to solve.
>
> One concerning thing was to have CRTL (via SSIO) access directly to
> XFC. From an architectural point of view this is wrong at so many
> levels,...

Off the top, some of the various existing stuff that breaks layering on
OpenVMS includes HBVS volume shadowing, MOUNT, and byte-range locking.

IP as a layered product is broken layering.

The C select() call is a fine mess of mis-layering.

The XQP design is mis-layering.

There are other examples.

There are examples of breaking layering to advantage, such as ZFS
else-platform.

All discussions of layering and esthetics aside, I presume the primary
purpose of the SSIO project is to permit porting PostgreSQL to OpenVMS,
posthaste.

--
Pure Personal Opinion | HoffmanLabs LLC

On Wednesday, 6 October 2021 at 2:09:17 pm UTC+11, Stephen Hoffman wrote:

> Off the top, some of the various existing stuff that breaks layering on
> OpenVMS includes HBVS volume shadowing, MOUNT, and byte-range locking.
>
> IP as a layered product is broken layering.
>
> The C select() call is a fine mess of mis-layering.
>
> The XQP design is mis-layering.
>
> There are other examples.
>
> There are examples of breaking layering to advantage, such as ZFS
> else-platform.
>
> All discussions of layering and esthetics aside, I presume the primary
> purpose of the SSIO project is to permit porting PostgreSQL to OpenVMS,
> posthaste.

Yup, exactly, hence get CRTL to use RMS which does work.

Re byte range locking, why not just use locking granularity (aka Rdb) to do the job. Very efficient and has worked for decades, and no need to change VMS DLM. Sure it may be nice to have an API that does this for us, but hey we are programmers.

gt

On 10/5/21 9:06 PM, Greg Tinkler wrote:

> An extra that could be added, if the file is RFM=fixed, and the C
> code uses it that way with the same record length then use the
> SYS$GET/SYS$PUT so it will play nicely with an RMS access to those files.

I don't know the degree to which the current plan corresponds to the
original plan from a decade or so ago, but back then only stream files
were going to be supported by SSIO, which makes sense since the whole
point is locking byte ranges.

On 10/5/2021 10:06 PM, Greg Tinkler wrote:
> I notice that SSIO (beta) in included in an up coming V9.1 field
> test. So I read up on the issues it is trying to solve.
>
> One concerning thing was to have CRTL (via SSIO) access directly to
> XFC. From an architectural point of view this is wrong at so many
> levels, but if that is what needs to happen then open it up so RMS
> and other code bases can use it.
>
> The main reason stated was the need to do byte offset/count IO’s.
> Well lets solve that first, change RMS by adding SYS$READB and
> SYS$WRITEB. These would be useful to all code using RMS. SYS$READB
> read from byte offset for count, return latest data from that byte
> range. SYS$WRITEB write from byte offset for count, update latest
> copy of underlying blocks.

> By having these as part of RMS we want to ensure the blocks/buffers
> are coordinated so any other user of RMS will see the changes, and we
> get their changes.
>
> This seems to be at the core of the CRTL issue, it does NOT use RMS,
> nor does it synchronize its blocks/buffers, leading to the lost
> update problem.
>
> So with this ‘simple’ addition the CRTL should be altered to us RMS
> for all file IO.
>
> An extra that could be added, if the file is RFM=fixed, and the C
> code uses it that way with the same record length then use the
> SYS$GET/SYS$PUT so it will play nicely with an RMS access to those
> files.

To be honest then I think the safest way to implement this is
to put lots of restrictions on when it is doable.

Examples:
* No cluster support (announcement already states that!)
* Only FIX 512, STMLF and UDF are supported
* no mixing with traditional RMS calls

Some applications coming over from *nix most known PostgreSQL needs
this. But trying to cover all types of cases would be a lot of
work.

Arne

On Wednesday, October 6, 2021 at 8:40:13 AM UTC-4, Craig A. Berry wrote:
> On 10/5/21 9:06 PM, Greg Tinkler wrote:
>
> > An extra that could be added, if the file is RFM=fixed, and the C
> > code uses it that way with the same record length then use the
> > SYS$GET/SYS$PUT so it will play nicely with an RMS access to those files.
> I don't know the degree to which the current plan corresponds to the
> original plan from a decade or so ago, but back then only stream files
> were going to be supported by SSIO, which makes sense since the whole
> point is locking byte ranges.

Open source software ports often comes with the restriction that it only works
with stream-LF files. Maybe they should add flag to directory files that if set
only allows it to contain stream-LF or directory files.

I keep a stmlf.fdl file in my login directory to use for copying (i.e. convert/fdl=...)
text files to NFS shares.

On 10/6/2021 8:40 AM, Craig A. Berry wrote:
>
> On 10/5/21 9:06 PM, Greg Tinkler wrote:
>
>> An extra that could be added, if the file is RFM=fixed, and the C
>> code uses it that way with the same record length then use the
>> SYS$GET/SYS$PUT so it will play nicely with an RMS access to those files.
>
> I don't know the degree to which the current plan corresponds to the
> original plan from a decade or so ago, but back then only stream files
> were going to be supported by SSIO, which makes sense since the whole
> point is locking byte ranges.

It has been my impression that for quite some time at HP, work on
specific requests tended to be very specific to that request, and failed
to consider capabilities as general to VMS.

The approach to SSIO appears to be an example of this. Basically, do
the least required to achieve the specific result. In the case of SSIO
the result appears to be rather useless, at least so far.

For some years I've advocated a more general enhancement to the VMS DLM,
specifically, numeric range locking. Such would address a basic issue
I've had with the VMS DLM for a rather long time.

I've a database product, a rather old product. At the time it was
implemented it was rather useful. But there was a locking issue. The
DLM locks resource names. The database would support I/O transfers of 1
to 127 disk blocks. How would one lock 127 contiguous disk blocks? The
blunt force method could be taking out 127 locks, not an optimum
solution. Having numeric range locking back in 1984 would have been
quite useful.

I've also suggested in the past that a simple enhancement to the DLM,
specifically the addition of a "type of lock" with the capability of
adding logic for specific "types" would solve the locking part of SSIO
and do so as a part of VMS, not as part of the CRTL.

As for byte range I/O, I'm not sure what is and isn't possible with disk
drives. It has been my impression that only whole block transfers are
possible. Perhaps I've been wrong. Perhaps SSDs have more flexibility.

Not really an issue for me anymore.

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

On 10/6/2021 9:45 AM, Dave Froble wrote:
> On 10/6/2021 8:40 AM, Craig A. Berry wrote:
>> On 10/5/21 9:06 PM, Greg Tinkler wrote:
>>> An extra that could be added, if the file is RFM=fixed, and the C
>>> code  uses it that way with the same record length then use the
>>> SYS$GET/SYS$PUT so it will play nicely with an RMS access to those
>>> files.
>>
>> I don't know the degree to which the current plan corresponds to the
>> original plan from a decade or so ago, but back then only stream files
>> were going to be supported by SSIO, which makes sense since the whole
>> point is locking byte ranges.
>
> It has been my impression that for quite some time at HP, work on
> specific requests tended to be very specific to that request, and failed
> to consider capabilities as general to VMS.
>
> The approach to SSIO appears to be an example of this.  Basically, do
> the least required to achieve the specific result.  In the case of SSIO
> the result appears to be rather useless, at least so far.

General is better than specific.

When not considering resources.

My impression is that VSI engineering resources are very limited - and
several orders of magnitudes smaller than DEC 40 years ago.

So when they have the choice of solving something 80% for 200 hours of
effort or 100% for 1000 hours of effort then ...

> For some years I've advocated a more general enhancement to the VMS DLM,
> specifically, numeric range locking.  Such would address a basic issue
> I've had with the VMS DLM for a rather long time.

> I've also suggested in the past that a simple enhancement to the DLM,
> specifically the addition of a "type of lock" with the capability of
> adding logic for specific "types" would solve the locking part of SSIO
> and do so as a part of VMS, not as part of the CRTL.

That would make sense to me.

But I do not count.

> As for byte range I/O, I'm not sure what is and isn't possible with disk
> drives.  It has been my impression that only whole block transfers are
> possible.  Perhaps I've been wrong.  Perhaps SSDs have more flexibility.

No matter what the disk can do then the VMS file system is still
block oriented and I believe the system services take block offsets
not byte offsets.

Arne

On 2021-10-06 03:32:50 +0000, Greg Tinkler said:

> Yup, exactly, hence get CRTL to use RMS which does work.

For this case, RMS really doesn't work at all well. Says why right
there in the name, too. Record management, not stream management.

C and IP have both been tussling with mismatched assumptions within the
OpenVMS file system since the instantiation of C on OpenVMS, too.

Lately, I've been tussling with the record-oriented assumptions within
OpenVMS. Records just never got as far along as objects. And RMS
records are an unmitigated joy around upgrades and mixed-version
clusters.

The various stream-format files are one of the ensuing compromises here.

> Re byte range locking, why not just use locking granularity (aka Rdb)
> to do the job. Very efficient and has worked for decades, and no need
> to change VMS DLM.

The use of Oracle Rdb isn't viable as a dependency for many folks, and
lock granularity doesn't work at all well for arbitrary and overlapping
locking ranges.

> Sure it may be nice to have an API that does this for us, but hey we
> are programmers.

I don't want us each writing and debugging and maintaining
range-locking code for what is part of the C standard library, but you
do you.

As much as I'd like a general range-locking solution here in DLM, and
with adding (better?) stream I/O support into RMS, and as much as I'd
like to see OO API support added, and IP integration, and app and app
security integration with sandboxes, packaging, and package management,
and a whole pile of other badly-needed work, I'd infer that the folks
at VSI really want PostgreSQL as an available database option soonest.

There's a very long history of "can-kicking" here and a whole lot of
that is almost inherent and inevitable with the upward-compatibility
goals for the platform, and with resulting miasma far less visible to
those of us that have used OpenVMS for the past decade or three or
more, but is front and center with any new developer looking at the
APIs, and with any wholly new 64-bit app work.

--
Pure Personal Opinion | HoffmanLabs LLC

In article <4e4ac776-f43f-4a84-8846-7fd0b6d949een@googlegroups.com>,
osuvman50@gmail.com (David Jones) wrote:

> Open source software ports often comes with the restriction that it
> only works with stream-LF files. Maybe they should add a flag to
> directory files that if set only allows it to contain stream-LF
> or directory files.

People used to UNIX or Windows generally find the other VMS file types
baffling and confusing. I got used to the idea, but never made use of
them, since my employers already had fewer customers on VMS than they did
UNIX when I joined, and the disparity only increased.

John

What a good conversation, some feedback.
>To be honest then I think the safest way to implement this is
>to put lots of restrictions on when it is doable.
>
>Examples:
>* No cluster support (announcement already states that!)
>* Only FIX 512, STMLF and UDF are supported
>* no mixing with traditional RMS calls

My point is SSIO seems to be focused on just PostgreSQL, whereas an RMS solution is much much easier to program, uses well tested code, and is already cluster ready putting the team ahead of the game and not building issues for the future.

>I've a database product, a rather old product. At the time it was
>implemented it was rather useful. But there was a locking issue. The
>DLM locks resource names. The database would support I/O transfers of 1
>to 127 disk blocks. How would one lock 127 contiguous disk blocks? The
>blunt force method could be taking out 127 locks, not an optimum
>solution. Having numeric range locking back in 1984 would have been
>quite useful.

Yup DLM uses resource names, but they can be hierarchical, like a B-Tree index. Also the resources need only exist when needed, removed it not. The the lock tree size depends on the lock contention.

This is why I made reference to Rdb, it uses this technique, and they are probably not the only ones. NB each level controls a range of resources and each level can have it’s own fan out factor. The depth and lowest level is always dependant on the applications requirements.

FYI I am pretty sure RMS uses RFA to lock a record, this is an implied range of 1 record.

>No matter what the disk can do then the VMS file system is still
>block oriented and I believe the system services take block offsets
>not byte offsets.
All disks are block based, even on Unix. With some SSD’s yes you can do byte transfers, but this should be left to the driver to optimise. Also with X86_64 it weill be virtualised so what the..

>For this case, RMS really doesn't work at all well. Says why right
>there in the name, too. Record management, not stream management.

Well yes and no. If you think about it most Unix text IO is record, ie LF terminated, and binary is fixed records not necessarily the same length in the file.

RMS for $GET and $PUT are record based, but $READ and $WRITE are block based, missing is $READB and $WRITEB, not just for CRTL but useful for various applications.

RMS ISAM with fixed length records is a pain, I have long argued ISAM should support variable length records, don’t care if they are VFC or STMLF, I would allow for both as VFC could allow for binary variable length records.

Likewise the keys on an ISAM file should be able to be variable based on a separator e.g “,” or <tab> or a combination.

>The use of Oracle Rdb isn't viable as a dependency for many folks, and
>lock granularity doesn't work at all well for arbitrary and overlapping
>locking ranges.
I think you will be a B-Tree style dynamic resource tree, similar to what Rdb uses, will work well. Any ‘byte range’ implementation will need some index to find interesting locks, DLM uses hash which is as efficient as you can get.

>> Sure it may be nice to have an API that does this for us, but hey we
>> are programmers.

>I don't want us each writing and debugging and maintaining
>range-locking code for what is part of the C standard library, but you
>do you.
NO, quite the opposite. I believe there is a POSIX standard for a locking API, and as VMS, sorry OpenVMS, wishes to maintain its POSIX stamp it should use these API’s using DLM underneath. NB DLM is also already cluster based, but you know that.

>People used to UNIX or Windows generally find the other VMS file types
>baffling and confusing.

I always wondered why the CRTL did not have some smarts to present a VFC records as STMLF and vise-versa, effectively hiding the internal record structures. This could be done via open using the VMS extension “rfm=STMLF” which should be the default unless it is a binary file “rfm=unf”. If the file is VFC then CRTL could to the translation. Wishful thinking.

gt down under

On Thursday, October 7, 2021 at 2:18:57 AM UTC+13, osuv...@gmail.com wrote:

> Open source software ports often comes with the restriction that it only works
> with stream-LF files.

I would say that’s partially true. Typically there are options to treat files as “text” or “binary”. A “binary” file is just a stream of arbitrary 8-bit bytes, which are supposed to be read or written without any imposition of record boundaries, sector-size rounding or special treatment of any byte values. A “text” file is assumed to be broken up into lines. It is true that LF is the traditional Unix line delimiter. But enlightened toolkits like Python are capable of reading text files in “universal newline” mode, so for example if you copy a text file created on MS-DOS (line delimiter = CR+LF, because CP/M did it that way, for no rational reason) in binary mode onto a Linux system, your Python text-processing script running on the latter can cope with it without a hiccup.

On 10/6/2021 9:25 PM, Greg Tinkler wrote:
> What a good conversation, some feedback.
>> To be honest then I think the safest way to implement this is
>> to put lots of restrictions on when it is doable.
>>
>> Examples:
>> * No cluster support (announcement already states that!)
>> * Only FIX 512, STMLF and UDF are supported
>> * no mixing with traditional RMS calls
>
> My point is SSIO seems to be focused on just PostgreSQL, whereas an
> RMS solution is much much easier to program, uses well tested code,
> and is already cluster ready putting the team ahead of the game and
> not building issues for the future.
I very much doubt that a full RMS solution is much easier.

:-)

>> For this case, RMS really doesn't work at all well. Says why right
>> there in the name, too. Record management, not stream management.
>
> Well yes and no. If you think about it most Unix text IO is record,
> ie LF terminated, and binary is fixed records not necessarily the
> same length in the file.
>
> RMS for $GET and $PUT are record based, but $READ and $WRITE are
> block based, missing is $READB and $WRITEB, not just for CRTL but
> useful for various applications.
>
> RMS ISAM with fixed length records is a pain, I have long argued ISAM
> should support variable length records, don’t care if they are VFC or
> STMLF, I would allow for both as VFC could allow for binary variable
> length records.
????

Index-sequential files and RMS API supports variable length.

Not all language API's on top of RMS does.

>> The use of Oracle Rdb isn't viable as a dependency for many folks, and
>> lock granularity doesn't work at all well for arbitrary and overlapping
>> locking ranges.

> I think you will be a B-Tree style dynamic resource tree, similar to
> what Rdb uses, will work well. Any ‘byte range’ implementation will
> need some index to find interesting locks, DLM uses hash which is as
> efficient as you can get.
Hash is effective for finding exact matches but useless for finding
other matches aka "starting with". For those a tree is better.

Arne

On Thursday, October 7, 2021 at 8:04:43 AM UTC+13, John Dallman wrote:
> People used to UNIX or Windows generally find the other VMS file types
> baffling and confusing.

One question I never saw answered (because I never came across examples of files to check it) was whether in “VFC” files, the record count included the fixed header or not? And was that the same or different in the on-disk format versus the in-memory RMS structure with the “RSZ” (“RAB$W_RSZ”?) field?

By the way, I knew FORTRAN carriage control is now an anachronism, but I didn’t realize that it is now considered so obsolete, that compilers won’t support it any more.

On 10/6/2021 9:51 PM, Lawrence D’Oliveiro wrote:
> On Thursday, October 7, 2021 at 8:04:43 AM UTC+13, John Dallman wrote:
>> People used to UNIX or Windows generally find the other VMS file types
>> baffling and confusing.
>
> One question I never saw answered (because I never came across
> examples of files to check it) was whether in “VFC” files, the record
> count included the fixed header or not? And was that the same or
> different in the on-disk format versus the in-memory RMS structure
> with the “RSZ” (“RAB$W_RSZ”?) field?

Try it!

$ open/write z.z z.z
$ write z.z "ABC"
$ close z.z
$ dir/full z.z

Directory DISK2:[ARNE]

z.z;1 File ID: (5295,236,0)
....
Record format: VFC, 2 byte header, maximum 0 bytes, longest 3 bytes
....
$ dump z.z

Dump of file DISK2:[ARNE]z.z;1 on 6-OCT-2021 21:54:39.48
File ID (5295,236,0) End of file block 1 / Allocated 16

Virtual block number 1 (00000001), 512 (0200) bytes

00000000 00000000 00000000 00000000 00000000 0000FFFF 00434241
8D010005 ....ABC......................... 000000

Arne

On Thursday, October 7, 2021 at 2:25:59 PM UTC+13, tink...@gmail.com wrote:

> All disks are block based, even on Unix.

The difference being, on *nix systems, the responsibility for blocking and deblocking is left to the filesystem layer. So if a file is n bytes long, and n mod «sector size» ≠ 0, the application never sees what is in the padding bytes, if any.

Some filesystems even implement “tail packing”, which means the leftover bits of multiple files can share the same block, all transparently to the application, minimizing fragmentation.

By the way, Linus Torvalds did apparently use a VMS system at some point. (Must have been after his Sinclair QL days.) Guess what reason he gave, when asked why he hated it ...

> RMS ISAM with fixed length records is a pain, I have long argued ISAM should support
>variable length records ...

Given that nowadays an SQL-based RDBMS like SQLite can offer full support for transactions, joins and subqueries (missing only more multi-user-type features like locking and replication), and yet still be resource-light enough to fit in your mobile phone, I would say the time for application developers to be grubbing about in ISAM files is past.

On 10/6/2021 9:25 PM, Greg Tinkler wrote:
> What a good conversation, some feedback.
>> To be honest then I think the safest way to implement this is
>> to put lots of restrictions on when it is doable.
>>
>> Examples:
>> * No cluster support (announcement already states that!)
>> * Only FIX 512, STMLF and UDF are supported
>> * no mixing with traditional RMS calls
>
> My point is SSIO seems to be focused on just PostgreSQL, whereas an RMS solution is much much easier to program, uses well tested code, and is already cluster ready putting the team ahead of the game and not building issues for the future.

RMS is a bit too high level for what's being discussed.

But yeah, the real issue is that SSIO was aimed (it seems) at
PostgreSQL. In my opinion, that is poor software architecture and design.

>> I've a database product, a rather old product. At the time it was
>> implemented it was rather useful. But there was a locking issue. The
>> DLM locks resource names. The database would support I/O transfers of 1
>> to 127 disk blocks. How would one lock 127 contiguous disk blocks? The
>> blunt force method could be taking out 127 locks, not an optimum
>> solution. Having numeric range locking back in 1984 would have been
>> quite useful.
>
> Yup DLM uses resource names, but they can be hierarchical, like a B-Tree index. Also the resources need only exist when needed, removed it not. The the lock tree size depends on the lock contention.

Well the perceived issue is what happens when taking out locks, and at
some point there is a conflict. Say needing 127 blocks locked, and the
conflict is on the last block. That means 126 locks to be released, and
perhaps try again.

In reality, the large I/O buffer capability is rarely used, and then
it's usually with exclusive file access, which precludes the need for
block locks, just the file lock. For random access, single block
locking and I/O is good. Larger I/O buffers are usually used for
sequential access, both read only, and updating.

> This is why I made reference to Rdb, it uses this technique, and they are probably not the only ones. NB each level controls a range of resources and each level can have it’s own fan out factor. The depth and lowest level is always dependant on the applications requirements.
>
> FYI I am pretty sure RMS uses RFA to lock a record, this is an implied range of 1 record.

RMS has some interesting internals, basically below application usage.

Global buffers
Multiple buffers
Multi-block count

RMS can (I believe, it's been a long while) keep track of file usage,
and provide data from an RMS buffer to a user's buffer. No disk
activity required. Writes of course must go to disk. But even so, the
data can still be in the updated global buffers for use by multiple tasks.

>> No matter what the disk can do then the VMS file system is still
>> block oriented and I believe the system services take block offsets
>> not byte offsets.
> All disks are block based, even on Unix. With some SSD’s yes you can do byte transfers, but this should be left to the driver to optimise. Also with X86_64 it weill be virtualised so what the..

As long as storage is block oriented, then regardless of the numeric
range of bytes, all blocks encompassing the byte range will need to be
read, including locking, and written. This usually will include data
outside the byte range.

>> For this case, RMS really doesn't work at all well. Says why right
>> there in the name, too. Record management, not stream management.

Ayep. RMS is record based.

> Well yes and no. If you think about it most Unix text IO is record, ie LF terminated, and binary is fixed records not necessarily the same length in the file.
>
> RMS for $GET and $PUT are record based, but $READ and $WRITE are block based, missing is $READB and $WRITEB, not just for CRTL but useful for various applications.

Forget RMS, I/O would be at the QIO level.

> RMS ISAM with fixed length records is a pain, I have long argued ISAM should support variable length records, don’t care if they are VFC or STMLF, I would allow for both as VFC could allow for binary variable length records.

RMS keyed files can have variable record lengths.
RMS relative files require fixed length records. (if I remember correctly)
RMS sequential files can have variable record lengths.

> Likewise the keys on an ISAM file should be able to be variable based on a separator e.g “,” or <tab> or a combination.
>
>> The use of Oracle Rdb isn't viable as a dependency for many folks, and
>> lock granularity doesn't work at all well for arbitrary and overlapping
>> locking ranges.
> I think you will be a B-Tree style dynamic resource tree, similar to what Rdb uses, will work well. Any ‘byte range’ implementation will need some index to find interesting locks, DLM uses hash which is as efficient as you can get.
>
>>> Sure it may be nice to have an API that does this for us, but hey we
>>> are programmers.
>
>> I don't want us each writing and debugging and maintaining
>> range-locking code for what is part of the C standard library, but you
>> do you.
> NO, quite the opposite. I believe there is a POSIX standard for a locking API, and as VMS, sorry OpenVMS, wishes to maintain its POSIX stamp it should use these API’s using DLM underneath. NB DLM is also already cluster based, but you know that.
>
>> People used to UNIX or Windows generally find the other VMS file types
>> baffling and confusing.

That is because, without additional apps, Unix I/O is a stream of bytes.
There is no concept of records, such as that provided by RMS.

Frankly, (and yes, I'm biased), I find records reasonable, and a stream
of bytes baffling and confusing. Guess it's what one is used to.

> I always wondered why the CRTL did not have some smarts to present a VFC records as STMLF and vise-versa, effectively hiding the internal record structures. This could be done via open using the VMS extension “rfm=STMLF” which should be the default unless it is a binary file “rfm=unf”. If the file is VFC then CRTL could to the translation. Wishful thinking.

I would suggest the use of "VMS" in the above, rather than "CRTL". That
is unless one considers the CRTL VMS ...

> gt down under
>

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

On Thursday, October 7, 2021 at 5:12:58 PM UTC+13, Dave Froble wrote:
> Frankly, (and yes, I'm biased), I find records reasonable, and a stream
> of bytes baffling and confusing. Guess it's what one is used to.

Trouble is, there are many binary file formats that do not map easily to a simple sequence of records (of whatever delimitation). Consider the IFF family of file formats, for example: these are built out of chunks, and certain chunk types can contain other chunks.

For another example, consider file formats like TIFF and TTF, where there is a directory that identifies the location and size of the various major pieces. Oh, and PDF comes under this as well.

And then there are text-based format families, like XML, JSON, YAML, TOML ....

On 2021-10-06, John Dallman <jgd@cix.co.uk> wrote:
> In article <4e4ac776-f43f-4a84-8846-7fd0b6d949een@googlegroups.com>,
> osuvman50@gmail.com (David Jones) wrote:
>
>> Open source software ports often comes with the restriction that it
>> only works with stream-LF files. Maybe they should add a flag to
>> directory files that if set only allows it to contain stream-LF
>> or directory files.
>
> People used to UNIX or Windows generally find the other VMS file types
> baffling and confusing. I got used to the idea, but never made use of
> them, since my employers already had fewer customers on VMS than they did
> UNIX when I joined, and the disparity only increased.
>

That because asking Unix/Windows people to learn about VMS records and
file structures is like asking a VMS person to learn about how to work
with records and files on z/OS using traditional z/OS methods.

It is something so very, very, different from what they are used to.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 2021-10-06, Greg Tinkler <tinklerg@gmail.com> wrote:
>
>>For this case, RMS really doesn't work at all well. Says why right
>>there in the name, too. Record management, not stream management.
>
> Well yes and no. If you think about it most Unix text IO is record, ie LF terminated, and binary is fixed records not necessarily the same length in the file.
>

How do you find byte 12,335,456 in a variable length RMS sequential file
without reading from the start of the file ?

That's why there are restrictions on RMS supported file formats in an
application in some cases.

>
> I always wondered why the CRTL did not have some smarts to present a VFC
> records as STMLF and vise-versa, effectively hiding the internal record
> structures. This could be done via open using the VMS extension ?rfm=STMLF?
> which should be the default unless it is a binary file ?rfm=unf?. If the file
> is VFC then CRTL could to the translation. Wishful thinking.
>

This could not be the default. What if LF characters are part of the
existing data record itself ? You have just destroyed the meaning of
the file in that case.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On Thursday, 7 October 2021 at 11:26:38 pm UTC+11, Simon Clubley wrote:

> How do you find byte 12,335,456 in a variable length RMS sequential file
> without reading from the start of the file ?
>
> That's why there are restrictions on RMS supported file formats in an
> application in some cases.

The same way it is done on Unix, calculate the block offset, go get it, and extract the byte. no difference and nothing to do with the underlying format.

> > I always wondered why the CRTL did not have some smarts to present a VFC
> > records as STMLF and vise-versa, effectively hiding the internal record
> > structures. This could be done via open using the VMS extension ?rfm=STMLF?
> > which should be the default unless it is a binary file ?rfm=unf?. If the file
> > is VFC then CRTL could to the translation. Wishful thinking.
> >
> This could not be the default. What if LF characters are part of the
> existing data record itself ? You have just destroyed the meaning of
> the file in that case.

Please read what I wrote, if the file has been opened "b" then don't, otherwise we need to assume it is stmLF. Yup probably another logical to set the default but I'm pretty sure if you create a new file using CRTL with the defaults then it will be stmLF anyway.

gt

On 10/6/21 11:10 PM, Dave Froble wrote:

> the real issue is that SSIO was aimed (it seems) at
> PostgreSQL.

And Apache, and Samba, and other things that have been explicitly
mentioned as having needed app-specific workarounds due to the absence
of shared stream I/O support. SSIO *is* the general-purpose solution
that you seem to be lamenting the lack of.

> Well the perceived issue is what happens when taking out locks, and at
> some point there is a conflict. Say needing 127 blocks locked, and the
> conflict is on the last block. That means 126 locks to be released, and
> perhaps try again.
Maybe, maybe not. It depends on the locking fan out factors for the differing levels. It is possible that only 1 lock is needed, may be more, the wort case would be 127. NB there is also BLAST to assist with managing the lock promotion/demotions.

> As long as storage is block oriented, then regardless of the numeric
> range of bytes, all blocks encompassing the byte range will need to be
> read, including locking, and written. This usually will include data
> outside the byte range.

Yup, as is the case on Unix...let the drivers worry about how and why this is done, block/byte what ever the IO device needs.

> Forget RMS, I/O would be at the QIO level.

Why? Underneath RMS is QIO, what RMS gives us the the coordination of the buffers/buckets/clumps/block across the cluster to ensure not lost updates, as per the example used to justify SSIO.

> RMS keyed files can have variable record lengths.
True, VAR only not VFC or STM*, but fixed length key fields, with fixed offsets in the record

> RMS relative files require fixed length records. (if I remember correctly)
Yup, there are implicitly fixed length.

==Have been thinking about the byte range locking. As most of the use will be for locking ranges in a file it should be integrated with RMS, i.e. RMS should have an API to allow this as it already does the locking to the buffer/bucket/clump/block. Just need another 1 or 2 layers of lock tree and you have it. And it all be cluster wide, and it will be compatible with other users of RMS.

gt

On 2021-10-07, Greg Tinkler <tinklerg@gmail.com> wrote:
> On Thursday, 7 October 2021 at 11:26:38 pm UTC+11, Simon Clubley wrote:
>
>> How do you find byte 12,335,456 in a variable length RMS sequential file
>> without reading from the start of the file ?
>>
>> That's why there are restrictions on RMS supported file formats in an
>> application in some cases.
>
> The same way it is done on Unix, calculate the block offset, go get it, and extract the byte. no difference and nothing to do with the underlying format.
>

You don't know the block offset without scanning the file when it comes
to some RMS file formats.

IOW, data byte 12,335,456 will not be the same thing as file byte 12,335,456
unless you restrict yourself to record formats that do not have embedded
record metadata.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 10/7/2021 8:59 AM, Simon Clubley wrote:
> On 2021-10-07, Greg Tinkler <tinklerg@gmail.com> wrote:
>> On Thursday, 7 October 2021 at 11:26:38 pm UTC+11, Simon Clubley wrote:
>>> How do you find byte 12,335,456 in a variable length RMS sequential file
>>> without reading from the start of the file ?
>>>
>>> That's why there are restrictions on RMS supported file formats in an
>>> application in some cases.
>>
>> The same way it is done on Unix, calculate the block offset, go get it, and extract the byte. no difference and nothing to do with the underlying format.
>
> You don't know the block offset without scanning the file when it comes
> to some RMS file formats.
>
> IOW, data byte 12,335,456 will not be the same thing as file byte 12,335,456
> unless you restrict yourself to record formats that do not have embedded
> record metadata.

Yes.

And it does not get better when using standard C IO.

I suspect that the variable length file output below will
surprise a few *nix developers.

$ type var.txt
A BB
CCC
$ type stmlf.txt
A BB
CCC
$ type process.c
#include <stdio.h>
#include <sys/stat.h>

void sequential(const char *fnm, int mode)
{ FILE *fp;
int ix, c;
printf("%s sequential read (%s):", fnm, mode ? "binary" : "text");
fp = fopen(fnm, mode ? "rb" : "r");
ix = 0;
while((c = fgetc(fp)) >= 0)
{
ix++;
if(c >= 0)
{
printf(" %d=%02X", ix, c);
}
else
{
printf(" %d=-1", ix);
}
}
printf("\n");
fclose(fp);
}

void direct(const char *fnm, int mode, int siz)
{ FILE *fp;
int ix, c;
printf("%s direct read (%s):", fnm, mode ? "binary" : "text");
fp = fopen(fnm, mode ? "rb" : "r");
for(ix = 0; ix < siz; ix++)
{
fseek(fp, ix, SEEK_SET);
c = fgetc(fp);
if(c >= 0)
{
printf(" %d=%02X", ix + 1, c);
}
else
{
printf(" %d=-1", ix + 1);
}
}
printf("\n");
fclose(fp);
}

int main(int argc,char *argv[])
{ struct stat buf;
stat(argv[1], &buf);
printf("%s size = %d bytes\n", argv[1], (int)buf.st_size);
sequential(argv[1], 0);
sequential(argv[1], 1);
direct(argv[1], 0, (int)buf.st_size);
direct(argv[1], 1, (int)buf.st_size);
return 0;
} $ cc process
$ link process
$ mcr sys$disk:[]process var.txt
var.txt size = 14 bytes
var.txt sequential read (text): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43 8=43 9=0A
var.txt sequential read (binary): 1=41 2=42 3=42 4=43 5=43 6=43
var.txt direct read (text): 1=41 2=-1 3=02 4=-1 5=42 6=-1 7=-1 8=-1 9=43
10=-1 11=-1 12=-1 13=FF 14=-1
var.txt direct read (binary): 1=41 2=-1 3=02 4=-1 5=42 6=-1 7=-1 8=-1
9=43 10=-1 11=-1 12=-1 13=FF 14=-1
$ mcr sys$disk:[]process stmlf.txt
stmlf.txt size = 9 bytes
stmlf.txt sequential read (text): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43
8=43 9=0A
stmlf.txt sequential read (binary): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43
8=43 9=0A
stmlf.txt direct read (text): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43 8=43 9=0A
stmlf.txt direct read (binary): 1=41 2=0A 3=42 4=42 5=0A 6=43 7=43 8=43 9=0A

Arne