On 10/9/2021 4:55 PM, Stephen Hoffman wrote:
> On 2021-10-09 18:22:02 +0000, Arne Vajhj said:
>
>> On 10/9/2021 12:47 PM, Stephen Hoffman wrote:
>>>
>>> RMS is a pretty good database, for its time. Alas, its become rather
>>> more dated, with an API design that is complex and limiting, and in
>>> competitive terms RMS is badly feature-limited.
>>>
>>> If you need a key-value store and where the developer entirely owns
>>> the fields used within the punched cards, and where y'all can fit
>>> your files in 2 TiB (or bound volume sets, gag), RMS is still a fine
>>> choice.
>>
>> Hoff I think you are muddying the water here.
>>
>> This discussion has so far been about ORG:SEQ files.
>>
>> ORG:IDX files are a Key Value Store. But that is a totally different
>> topic.
>
>
> And here I was trying to explicitly not slag on RMS and its
> capabilities, as that'd solely serve provoke a torrent of folks quite
> reasonably pointing out that RMS is perfect for {app}.
>

Which it is, for those apps that need and use it's capabilities. Well,
maybe not "perfect". There is that lack of definition of data fields
that is so lacking in RMS. What I believe you call "marshaling".

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

On 10/9/2021 6:52 PM, Dave Froble wrote:
> On 10/9/2021 2:22 PM, Arne Vajhøj wrote:
>> On 10/9/2021 12:47 PM, Stephen Hoffman wrote:
>>> RMS is a pretty good database, for its time.  Alas, its become rather
>>> more dated, with an API design that is complex and limiting, and in
>>> competitive terms RMS is badly feature-limited.
>>>
>>> If you need a key-value store and where the developer entirely owns
>>> the fields used within the punched cards, and where y'all can fit your
>>> files in 2 TiB (or bound volume sets, gag), RMS is still a fine choice.
>>
>> Hoff I think you are muddying the water here.
>>
>> This discussion has so far been about ORG:SEQ files.
>>
>> ORG:IDX files are a Key Value Store. But that is a totally
>> different topic.
>
> No, it is not.  The OP declared that RMS should be used for that.
>
> You are correct that we're concerned about stream files, but claims
> about RMS have been part of the discussion.

RMS is very much in scope for the discussion.

But considering files a stream of bytes and the SSIO
feature are only relevant for ORG:SEQ files.

Arne

On 10/9/2021 6:41 PM, Dave Froble wrote:
> On 10/9/2021 2:18 PM, Arne Vajhøj wrote:
>> On 10/9/2021 1:54 PM, Dave Froble wrote:
>>> On 10/9/2021 6:19 AM, Greg Tinkler wrote:
>>>> every thing is clumps of data being buffered is some way, the API
>>>> that accesses that data from the higher levels may be stream based.
>>>> In this case it is CRTL's role to translate the clumps of data
>>>> into/from stream API.
>>>
>>> So, how does Pascal, Fortran, Cobol, Basic, and such do it?
>>
>> They do not treat files as streams of bytes - they treat files
>> as sequences of records.
>>
>> The underlying problem is that the two paradigms are pretty
>> incompatible. It is not easy for CRTL to translate a sequence
>> of records to a stream of bytes in a consistent and meaningful
>> manner.
>
> Which is why Steve's suggestion for ODS2/ODS5 becoming just another file
> system.
>
> Which is why Steve's suggestion for RMS to become just another database
> product.  Well, if ODS? wants to use it for directories, Ok.
>
> But even if another "application" handles other files, there is still
> the issue of today's disks being block based (Ok, punched card if you
> must) devices.
>
> Stream devices is alien enough to today's VMS that it would be much
> better served by dedicated tools designed for that format.  (And it sure
> isn't RMS!)
>
> Then there is the interesting question of what the next format to come
> along might be.

It is certainly a possibility for VMS to add a new file system
where files are just streams of bytes aka no concept of records
in file system.

And it is certainly possible to change language IO to access
such a file system directly not through RMS.

But if ODS-2/5 and RMS are still to be supported (and they have
to for existing applications) then I am not sure that it will
make things less complicated.

Arne

On 10/7/2021 9:34 AM, Arne Vajhøj wrote:> On 10/7/2021 8:59 AM, Simon
Clubley wrote:
>> On 2021-10-07, Greg Tinkler <tinklerg@gmail.com> wrote:
>>> On Thursday, 7 October 2021 at 11:26:38 pm UTC+11, Simon Clubley wrote:
>>>> How do you find byte 12,335,456 in a variable length RMS sequential
>>>> file
>>>> without reading from the start of the file ?
>>>>
>>>> That's why there are restrictions on RMS supported file formats in an
>>>> application in some cases.
>>>
>>> The same way it is done on Unix, calculate the block offset, go get
>>> it, and extract the byte. no difference and nothing to do with the
>>> underlying format.
>>
>> You don't know the block offset without scanning the file when it comes
>> to some RMS file formats.
>>
>> IOW, data byte 12,335,456 will not be the same thing as file byte
>> 12,335,456
>> unless you restrict yourself to record formats that do not have embedded
>> record metadata.
>
> Yes.
>
> And it does not get better when using standard C IO.
>
> I suspect that the variable length file output below will
> surprise a few *nix developers.
I decided to add to the examples. More file types and
more languages.

Fair warning: this will be a long post.

:-)

It will be 3 parts:
- my summary of results
- raw results from test
- source code

Arne

My summary of results
=====================

Traditional record oriented languages (Fortran, Pascal, Basic etc.):
- limited options basically one can read records sequentially and
file size and offsets from start of file are irrelevant
- always works and the reads return the records with the expected
content
- the record format does not matter and RMS does a perfect
encapsulation of actual record format

Languages with a stream oriented view on files (C, Python, Java etc.):
* rfm:stmlf
- works consistently with all sorts of access and compatible
with *nix code
* all other rfm (var, vfc, fix and stm)
- open as text file and sequential read only: works consistently
and compatible with *nix code, some of the bytes read (LF)
are not in the file and the number of bytes read does not match
the file size but it basically works
- open as binary or try to use seek to arbitrary positions in
the file or try to use C ctx=stm: not *nix compatible, and
some of the behavior may surprise even VMS people [note that
some of the code are deemed unsupported by the VMS documentation,
but not always easy to find, and even if found for C then
one may have to guess other languages using CRTL use
that C function]

Raw results from test
=====================

var.txt Fortran READ: 41 * 42 42 * 43 43 43 * (6 bytes)
var.txt Pascal ReadLn: 41 * 42 42 * 43 43 43 * (6 bytes)
var.txt Basic Input: 41 * 42 42 * 43 43 43 * (6 bytes)
var.txt C size = 14 bytes
var.txt C fgetc (text): 41 0A 42 42 0A 43 43 43 0A (9 bytes)
var.txt C fgetc (binary): 41 42 42 43 43 43 (6 bytes)
var.txt C fseek and fgetc (text): 41 -1 02 -1 42 -1 -1 -1 43 -1 -1 -1 FF
-1 (14 bytes)
var.txt C fseek and fgetc (binary): 41 -1 02 -1 42 -1 -1 -1 43 -1 -1 -1
FF -1 (14 bytes)
var.txt C fgets (text): 41 0A * 42 42 0A * 43 43 43 0A * (9 bytes)
var.txt C fgets (binary): 41 42 42 43 43 43 * (6 bytes)
var.txt C fgetc (text stream): 01 00 41 00 02 00 42 42 03 00 43 43 43 00
(14 bytes)
var.txt C fgetc (binary stream): 01 00 41 00 02 00 42 42 03 00 43 43 43
00 (14 bytes)
var.txt C fseek and fgetc (text stream): 41 -1 02 -1 42 -1 -1 -1 43 -1
-1 -1 FF -1 (14 bytes)
var.txt C fseek and fgetc (binary stream): 41 -1 02 -1 42 -1 -1 -1 43 -1
-1 -1 FF -1 (14 bytes)
var.txt C fgets (text stream): 01 * (1 bytes)
var.txt C fgets (binary stream): 01 * (1 bytes)
var.txt Python size = 14 bytes
var.txt Python read 1 (mode r): 41 0A 42 42 0A 43 43 43 0A (9 bytes)
var.txt Python read 1 (mode rb): 01 00 41 00 02 00 42 42 03 00 43 43 43
00 (14 bytes)
var.txt Python seek and read 1 (mode r): 41 -1 02 -1 42 -1 -1 -1 43 -1
-1 -1 FF -1 (14 bytes)
var.txt Python seek and read 1 (mode rb): 01 00 41 00 02 00 42 42 03 00
43 43 43 00 (14 bytes)
var.txt Python readline (mode r): 41 0A * 42 42 0A * 43 43 43 0A * (9 bytes)
var.txt Python readline (mode rb): 01 00 41 00 02 00 42 42 03 00 43 43
43 00 * (14 bytes)
var.txt Java size = 14 bytes
var.txt Java InputStream read: 41 0A 42 42 0A 43 43 43 0A (9 bytes)
var.txt Java RandomAccessFile seek and read: 41 -1 02 -1 42 -1 -1 -1 43
-1 -1 -1 FF -1 (14 bytes)
var.txt Java InputStreamReader read: 41 0A 42 42 0A 43 43 43 0A (9 chars)
var.txt Java BufferedReader readLine: 41 * 42 42 * 43 43 43 * (6 chars)
vfc.txt Fortran READ: 41 * 42 42 * 43 43 43 * (6 bytes)
vfc.txt Pascal ReadLn: 41 * 42 42 * 43 43 43 * (6 bytes)
vfc.txt Basic Input: 41 * 42 42 * 43 43 43 * (6 bytes)
vfc.txt C size = 20 bytes
vfc.txt C fgetc (text): 41 0A 42 42 0A 43 43 43 0A (9 bytes)
vfc.txt C fgetc (binary): 01 8D 41 01 8D 42 42 01 8D 43 43 43 (12 bytes)
vfc.txt C fseek and fgetc (text): 41 0D -1 -1 0A -1 0A 0D -1 -1 -1 -1 0A
0D -1 -1 -1 -1 00 -1 (20 bytes)
vfc.txt C fseek and fgetc (binary): 01 8D -1 -1 04 -1 01 8D -1 -1 -1 -1
01 8D -1 -1 -1 -1 FF -1 (20 bytes)
vfc.txt C fgets (text): 41 0A * 42 42 0A * 43 43 43 0A * (9 bytes)
vfc.txt C fgets (binary): 01 -1 41 01 -1 42 42 01 -1 43 43 43 * (12 bytes)
vfc.txt C fgetc (text stream): 03 00 01 8D 41 00 04 00 01 8D 42 42 05 00
01 8D 43 43 43 00 (20 bytes)
vfc.txt C fgetc (binary stream): 03 00 01 8D 41 00 04 00 01 8D 42 42 05
00 01 8D 43 43 43 00 (20 bytes)
vfc.txt C fseek and fgetc (text stream): 41 0D -1 -1 0A -1 0A 0D -1 -1
-1 -1 0A 0D -1 -1 -1 -1 00 -1 (20 bytes)
vfc.txt C fseek and fgetc (binary stream): 01 8D -1 -1 04 -1 01 8D -1 -1
-1 -1 01 8D -1 -1 -1 -1 FF -1 (20 bytes)
vfc.txt C fgets (text stream): 03 * (1 bytes)
vfc.txt C fgets (binary stream): 03 * (1 bytes)
vfc.txt Python size = 20 bytes
vfc.txt Python read 1 (mode r): 41 0A 42 42 0A 43 43 43 0A (9 bytes)
vfc.txt Python read 1 (mode rb): 03 00 01 8D 41 00 04 00 01 8D 42 42 05
00 01 8D 43 43 43 00 (20 bytes)
vfc.txt Python seek and read 1 (mode r): 41 0D -1 -1 0A -1 0A 0D -1 -1
-1 -1 0A 0D -1 -1 -1 -1 00 -1 (20 bytes)
vfc.txt Python seek and read 1 (mode rb): 03 00 01 8D 41 00 04 00 01 8D
42 42 05 00 01 8D 43 43 43 00 (20 bytes)
vfc.txt Python readline (mode r): 41 0A * 42 42 0A * 43 43 43 0A * (9 bytes)
vfc.txt Python readline (mode rb): 03 00 01 8D 41 00 04 00 01 8D 42 42
05 00 01 8D 43 43 43 00 * (20 bytes)
vfc.txt Java size = 20 bytes
vfc.txt Java InputStream read: 41 0A 42 42 0A 43 43 43 0A (9 bytes)
vfc.txt Java RandomAccessFile seek and read: 41 0D -1 -1 0A -1 0A 0D -1
-1 -1 -1 0A 0D -1 -1 -1 -1 00 -1 (20 bytes)
vfc.txt Java InputStreamReader read: 41 0A 42 42 0A 43 43 43 0A (9 chars)
vfc.txt Java BufferedReader readLine: 41 * 42 42 * 43 43 43 * (6 chars)
stmlf.txt Fortran READ: 41 * 42 42 * 43 43 43 * (6 bytes)
stmlf.txt Pascal ReadLn: 41 * 42 42 * 43 43 43 * (6 bytes)
stmlf.txt Basic Input: 41 * 42 42 * 43 43 43 * (6 bytes)
stmlf.txt C size = 9 bytes
stmlf.txt C fgetc (text): 41 0A 42 42 0A 43 43 43 0A (9 bytes)
stmlf.txt C fgetc (binary): 41 0A 42 42 0A 43 43 43 0A (9 bytes)
stmlf.txt C fseek and fgetc (text): 41 0A 42 42 0A 43 43 43 0A (9 bytes)
stmlf.txt C fseek and fgetc (binary): 41 0A 42 42 0A 43 43 43 0A (9 bytes)
stmlf.txt C fgets (text): 41 0A * 42 42 0A * 43 43 43 0A * (9 bytes)
stmlf.txt C fgets (binary): 41 0A * 42 42 0A * 43 43 43 0A * (9 bytes)
stmlf.txt C fgetc (text stream): 41 0A 42 42 0A 43 43 43 0A (9 bytes)
stmlf.txt C fgetc (binary stream): 41 0A 42 42 0A 43 43 43 0A (9 bytes)
stmlf.txt C fseek and fgetc (text stream): 41 0A 42 42 0A 43 43 43 0A (9
bytes)
stmlf.txt C fseek and fgetc (binary stream): 41 0A 42 42 0A 43 43 43 0A
(9 bytes)
stmlf.txt C fgets (text stream): 41 0A * 42 42 0A * 43 43 43 0A * (9 bytes)
stmlf.txt C fgets (binary stream): 41 0A * 42 42 0A * 43 43 43 0A * (9
bytes)
stmlf.txt Python size = 9 bytes
stmlf.txt Python read 1 (mode r): 41 0A 42 42 0A 43 43 43 0A (9 bytes)
stmlf.txt Python read 1 (mode rb): 41 0A 42 42 0A 43 43 43 0A (9 bytes)
stmlf.txt Python seek and read 1 (mode r): 41 0A 42 42 0A 43 43 43 0A (9
bytes)
stmlf.txt Python seek and read 1 (mode rb): 41 0A 42 42 0A 43 43 43 0A
(9 bytes)
stmlf.txt Python readline (mode r): 41 0A * 42 42 0A * 43 43 43 0A * (9
bytes)
stmlf.txt Python readline (mode rb): 41 0A * 42 42 0A * 43 43 43 0A * (9
bytes)
stmlf.txt Java size = 9 bytes
stmlf.txt Java InputStream read: 41 0A 42 42 0A 43 43 43 0A (9 bytes)
stmlf.txt Java RandomAccessFile seek and read: 41 0A 42 42 0A 43 43 43
0A (9 bytes)
stmlf.txt Java InputStreamReader read: 41 0A 42 42 0A 43 43 43 0A (9 chars)
stmlf.txt Java BufferedReader readLine: 41 * 42 42 * 43 43 43 * (6 chars)
stm.txt Fortran READ: 41 * 42 42 * 43 43 43 * (6 bytes)
stm.txt Pascal ReadLn: 41 * 42 42 * 43 43 43 * (6 bytes)
stm.txt Basic Input: 41 * 42 42 * 43 43 43 * (6 bytes)
stm.txt C size = 12 bytes
stm.txt C fgetc (text): 41 0A 42 42 0A 43 43 43 0A (9 bytes)
stm.txt C fgetc (binary): 41 42 42 43 43 43 (6 bytes)
stm.txt C fseek and fgetc (text): 41 0A 0A 42 42 0A 0A 43 43 43 0A 0A
(12 bytes)
stm.txt C fseek and fgetc (binary): 41 42 0A 42 42 43 0A 43 43 43 -1 0A
(12 bytes)
stm.txt C fgets (text): 41 0A * 42 42 0A * 43 43 43 0A * (9 bytes)
stm.txt C fgets (binary): 41 42 42 43 43 43 * (6 bytes)
stm.txt C fgetc (text stream): 41 0D 0A 42 42 0D 0A 43 43 43 0D 0A (12
bytes)
stm.txt C fgetc (binary stream): 41 0D 0A 42 42 0D 0A 43 43 43 0D 0A (12
bytes)
stm.txt C fseek and fgetc (text stream): 41 0A 0A 42 42 0A 0A 43 43 43
0A 0A (12 bytes)
stm.txt C fseek and fgetc (binary stream): 41 42 0A 42 42 43 0A 43 43 43
-1 0A (12 bytes)
stm.txt C fgets (text stream): 41 0D 0A * 42 42 0D 0A * 43 43 43 0D 0A *
(12 bytes)
stm.txt C fgets (binary stream): 41 0D 0A * 42 42 0D 0A * 43 43 43 0D 0A
* (12 bytes)
stm.txt Python size = 12 bytes
stm.txt Python read 1 (mode r): 41 0A 42 42 0A 43 43 43 0A (9 bytes)
stm.txt Python read 1 (mode rb): 41 0D 0A 42 42 0D 0A 43 43 43 0D 0A (12
bytes)
stm.txt Python seek and read 1 (mode r): 41 0A 0A 42 42 0A 0A 43 43 43
0A 0A (12 bytes)
stm.txt Python seek and read 1 (mode rb): 41 0D 0A 42 42 0D 0A 43 43 43
0D 0A (12 bytes)
stm.txt Python readline (mode r): 41 0A * 42 42 0A * 43 43 43 0A * (9 bytes)
stm.txt Python readline (mode rb): 41 0D 0A * 42 42 0D 0A * 43 43 43 0D
0A * (12 bytes)
stm.txt Java size = 12 bytes
stm.txt Java InputStream read: 41 0A 42 42 0A 43 43 43 0A (9 bytes)
stm.txt Java RandomAccessFile seek and read: 41 0A 0A 42 42 0A 0A 43 43
43 0A 0A (12 bytes)
stm.txt Java InputStreamReader read: 41 0A 42 42 0A 43 43 43 0A (9 chars)
stm.txt Java BufferedReader readLine: 41 * 42 42 * 43 43 43 * (6 chars)
fix2.txt Fortran READ: 41 41 * 42 42 * 43 43 * (6 bytes)
fix2.txt Pascal ReadLn: 41 41 * 42 42 * 43 43 * (6 bytes)
fix2.txt Basic Input: 41 41 * 42 42 * 43 43 * (6 bytes)
fix2.txt C size = 6 bytes
fix2.txt C fgetc (text): 41 41 0A 42 42 0A 43 43 0A (9 bytes)
fix2.txt C fgetc (binary): 41 41 42 42 43 43 (6 bytes)
fix2.txt C fseek and fgetc (text): 41 41 42 42 43 43 (6 bytes)
fix2.txt C fseek and fgetc (binary): 41 41 42 42 43 43 (6 bytes)
fix2.txt C fgets (text): 41 41 0A * 42 42 0A * 43 43 0A * (9 bytes)
fix2.txt C fgets (binary): 41 41 42 42 43 43 * (6 bytes)
fix2.txt C fgetc (text stream): 41 41 42 42 43 43 (6 bytes)
fix2.txt C fgetc (binary stream): 41 41 42 42 43 43 (6 bytes)
fix2.txt C fseek and fgetc (text stream): 41 41 42 42 43 43 (6 bytes)
fix2.txt C fseek and fgetc (binary stream): 41 41 42 42 43 43 (6 bytes)
fix2.txt C fgets (text stream): 41 41 42 42 43 43 * (6 bytes)
fix2.txt C fgets (binary stream): 41 41 42 42 43 43 * (6 bytes)
fix2.txt Python size = 6 bytes
fix2.txt Python read 1 (mode r): 41 41 0A 42 42 0A 43 43 0A (9 bytes)
fix2.txt Python read 1 (mode rb): 41 41 42 42 43 43 (6 bytes)
fix2.txt Python seek and read 1 (mode r): 41 41 42 42 43 43 (6 bytes)
fix2.txt Python seek and read 1 (mode rb): 41 41 42 42 43 43 (6 bytes)
fix2.txt Python readline (mode r): 41 41 0A * 42 42 0A * 43 43 0A * (9
bytes)
fix2.txt Python readline (mode rb): 41 41 42 42 43 43 * (6 bytes)
fix2.txt Java size = 6 bytes
fix2.txt Java InputStream read: 41 41 0A 42 42 0A 43 43 0A (9 bytes)
fix2.txt Java RandomAccessFile seek and read: 41 41 42 42 43 43 (6 bytes)
fix2.txt Java InputStreamReader read: 41 41 0A 42 42 0A 43 43 0A (9 chars)
fix2.txt Java BufferedReader readLine: 41 41 * 42 42 * 43 43 * (6 chars)
fix1.txt Fortran READ: 41 * 42 * 43 * (3 bytes)
fix1.txt Pascal ReadLn: 41 * 42 * 43 * (3 bytes)
fix1.txt Basic Input: 41 * 42 * 43 * (3 bytes)
fix1.txt C size = 6 bytes
fix1.txt C fgetc (text): 41 0A 42 0A 43 0A (6 bytes)
fix1.txt C fgetc (binary): 41 42 43 (3 bytes)
fix1.txt C fseek and fgetc (text): 41 00 42 00 43 00 (6 bytes)
fix1.txt C fseek and fgetc (binary): 41 00 42 00 43 00 (6 bytes)
fix1.txt C fgets (text): 41 0A * 42 0A * 43 0A * (6 bytes)
fix1.txt C fgets (binary): 41 42 43 * (3 bytes)
fix1.txt C fgetc (text stream): 41 00 42 00 43 00 (6 bytes)
fix1.txt C fgetc (binary stream): 41 00 42 00 43 00 (6 bytes)
fix1.txt C fseek and fgetc (text stream): 41 00 42 00 43 00 (6 bytes)
fix1.txt C fseek and fgetc (binary stream): 41 00 42 00 43 00 (6 bytes)
fix1.txt C fgets (text stream): 41 * (1 bytes)
fix1.txt C fgets (binary stream): 41 * (1 bytes)
fix1.txt Python size = 6 bytes
fix1.txt Python read 1 (mode r): 41 0A 42 0A 43 0A (6 bytes)
fix1.txt Python read 1 (mode rb): 41 00 42 00 43 00 (6 bytes)
fix1.txt Python seek and read 1 (mode r): 41 00 42 00 43 00 (6 bytes)
fix1.txt Python seek and read 1 (mode rb): 41 00 42 00 43 00 (6 bytes)
fix1.txt Python readline (mode r): 41 0A * 42 0A * 43 0A * (6 bytes)
fix1.txt Python readline (mode rb): 41 00 42 00 43 00 * (6 bytes)
fix1.txt Java size = 6 bytes
fix1.txt Java InputStream read: 41 0A 42 0A 43 0A (6 bytes)
fix1.txt Java RandomAccessFile seek and read: 41 00 42 00 43 00 (6 bytes)
fix1.txt Java InputStreamReader read: 41 0A 42 0A 43 0A (6 chars)
fix1.txt Java BufferedReader readLine: 41 * 42 * 43 * (3 chars)

On 10/6/2021 9:45 PM, Lawrence D’Oliveiro wrote:
> On Thursday, October 7, 2021 at 2:18:57 AM UTC+13, osuv...@gmail.com
> wrote:
>> Open source software ports often comes with the restriction that it
>> only works with stream-LF files.
>
> I would say that’s partially true. Typically there are options to
> treat files as “text” or “binary”. A “binary” file is just a stream
> of arbitrary 8-bit bytes, which are supposed to be read or written
> without any imposition of record boundaries, sector-size rounding or
> special treatment of any byte values. A “text” file is assumed to be
> broken up into lines. It is true that LF is the traditional Unix line
> delimiter. But enlightened toolkits like Python are capable of
> reading text files in “universal newline” mode, so for example if you
> copy a text file created on MS-DOS (line delimiter = CR+LF, because
> CP/M did it that way, for no rational reason) in binary mode onto a
> Linux system, your Python text-processing script running on the
> latter can cope with it without a hiccup.

Python on VMS doesn't really have much of a choice if it relies
on C IO - it will have to follow VMS C IO's conventions for how
to deal with different record formats.

(I have not checked the VMS Python source code, but I assume
that it uses C IO calls and do not have some VMS specific
code making direct RMS calls)

Arne

Such passion, love it.

As for stream vs not stream, that appears to be a religious type issue so we will need to disagree on that.

As for the original 2 issues from the paper 10 years ago.
1) updating a random range of bytes in a file, and loosing some updates
Using RMS correctly will fix this problem. It is straight forward to build a sample using DCL, of all things, to get and update 2 random fix length records on the same disk block by 2 separate process and the data not be lost. This does not loose either updates.
2) multiple processes writing to a ORG:SEQ files concurrently.
RMS supports this if you use SYS$PUT() and have the file marked as shared. NB you need to put a record otherwise you have a teared update, i.e. where part of the record is written then part of some-elses record then ... This is independent of VMS or *nix, it the same issue.

gt

On Sunday, October 10, 2021 at 11:44:03 AM UTC+13, Dave Froble wrote:

> But even if another "application" handles other files, there is still
> the issue of today's disks being block based ...

Those same block-based disks (and block-based SSDs, and what have you) work fine on POSIX/*nix/Linux OSes. They always have worked fine, and they will continue to work fine into the future.

The key point is that VMS shied away from a radical idea that Unix embraced: that the filesystem itself should abstract away the need for blocking and deblocking, and offer up a file as just a stream of n bytes, with no requirement on n being a multiple of any integer greater than 1.

On 2021-10-09 23:55:05 +0000, Greg Tinkler said:

> Such passion, love it.
>
> As for stream vs not stream, that appears to be a religious type issue
> so we will need to disagree on that.
>
> As for the original 2 issues from the paper 10 years ago.
> 1) updating a random range of bytes in a file, and loosing some updates
> Using RMS correctly will fix this problem. It is straight forward
> to build a sample using DCL, of all things, to get and update 2 random
> fix length records on the same disk block by 2 separate process and the
> data not be lost. This does not loose either updates.
> 2) multiple processes writing to a ORG:SEQ files concurrently.
> RMS supports this if you use SYS$PUT() and have the file marked as
> shared. NB you need to put a record otherwise you have a teared
> update, i.e. where part of the record is written then part of
> some-elses record then ... This is independent of VMS or *nix, it the
> same issue.

RMS is a database, and databases are quite common on OpenVMS and other
platforms.

Some databases will support recovery from crashes, replication across
multiple hosts, failover, clustering, and other factors.

For transactional databases, rollbacks are really handy; where you make
a series of updates and then discover (for whatever reason) you cannot
complete the sequence. This might be due to an app-detected issue, or
an app or system crash during the update sequence. If all of the
updates are made, the app then issues the commit. If all cannot be
made, the database rolls back to the initial state.

Without transactional processing (in the absence of RMS journaling and
lacking DECdtm integration), recovering from multi-part and multi-file
RMS updates get really gnarly when some part of the whole sequence
blows out.

Various databases found on OpenVMS will completely bypass RMS, too.

Some of the available databases for OpenVMS and other platforms have
some other interesting features, too. One very common database has a
portable database format across disparate operating systems, for
instance. You can copy the database file from (for instance) Linux to
OpenVMS, and can read and write the database file from within the
database client and its APIs; without requiring any sort of database
export/import or database reformatting.

Above is before we get to discussions of (for instance) graph databases
and object databases and other sorts, which aren't as common on
OpenVMS. PostgreSQL does have some capabilities here, too. As do other
databases.

--
Pure Personal Opinion | HoffmanLabs LLC

On 10/9/2021 7:55 PM, Greg Tinkler wrote:
> As for stream vs not stream, that appears to be a religious type
> issue so we will need to disagree on that.
>
> As for the original 2 issues from the paper 10 years ago. 1) updating
> a random range of bytes in a file, and loosing some updates Using RMS
> correctly will fix this problem. It is straight forward to build a
> sample using DCL, of all things, to get and update 2 random fix
> length records on the same disk block by 2 separate process and the
> data not be lost. This does not loose either updates.

RMS handle this fine for records but not for streams of bytes.

Arne

On Sunday, October 10, 2021 at 5:47:14 AM UTC+13, Stephen Hoffman wrote:
> Punched cards and punched-card-based assumptions are rather more
> pernicious within OpenVMS and clustering, and mailboxes, and various
> other areas, alas. For those of us steeped in OpenVMS, the effects of
> these assumptions can be invisible.

E.g. VMS mailboxes are record-based, whereas on *nix, pipes (both named and unnamed) are stream-based. PF_UNIX sockets can be either stream or datagram (record)-based. As for network connections, TCP is stream-based, but I think UDP is datagram-based. (AppleTalk? I could tell you stories about AppleTalk...)

I have heard tell of newbie TCP programmers assuming that messages they send in one operation will be received in their entirety in one operation ... wonder how long it takes before they discover that’s not true ...

> Back to RMS and SSIO, apps that don't expect punched-card semantics can
> and variously do perform their own coordination, so sharing the
> underlying files with apps that do expect punched cards is unnecessary,
> and counterproductive.

I don’t think it has been traditional on *nix systems to bother with much (if any) locking at all, at least not at the file I/O level. There has never been anything on such systems (that I have seen) like the VMS Lock Manager, distributed or otherwise. Any more elaborate locking protocol needed would be implemented through the subsystem’s own custom server processes.

On 10/9/2021 8:26 PM, Lawrence D’Oliveiro wrote:
> I don’t think it has been traditional on *nix systems to bother with
> much (if any) locking at all, at least not at the file I/O level.
> There has never been anything on such systems (that I have seen) like
> the VMS Lock Manager, distributed or otherwise. Any more elaborate
> locking protocol needed would be implemented through the subsystem’s
> own custom server processes.

try:

man dlm_lock

:-)

And it is used by GFS2.

Arne

On 2021-10-09 23:04:18 +0000, Arne Vajhj said:

> But if ODS-2/5 and RMS are still to be supported (and they have to for
> existing applications) then I am not sure that it will make things less
> complicated.

Though with metadata storage intended for best integration with RMS,
ODS-2 and ODS-5 are somewhere between independent and ambivalent about
the internal structures and organization of the files.

This is specifically why I refer to the XQP in various places in this
thread, as well as to file system virtual I/O, too.

A stream-based app connected to the XQP directly can be compatible with
OpenVMS tools including DELETE, COPY, etc., just as those same DCL
tools already work with ODS-3 and ODS-4, and with files associated with
non-RMS-based apps.

Now for a different file system, yes, you'll need an alternative to XQP
and the existing ACPs, and an overhauled or an alternative MOUNT, etc.
The addition of a FUSE API and related layering fits (roughly) here,
too.

--
Pure Personal Opinion | HoffmanLabs LLC

On 2021-10-10 00:26:46 +0000, Lawrence D’Oliveiro said:

> I don’t think it has been traditional on *nix systems to bother with
> much (if any) locking at all, at least not at the file I/O level. There
> has never been anything on such systems (that I have seen) like the VMS
> Lock Manager, distributed or otherwise. Any more elaborate locking
> protocol needed would be implemented through the subsystem’s own custom
> server processes.

The lock manager is not in the default path for many traditional Linux
file operations, certainly.

Various apps can use range locks or local locking or other
synchronization for that, but Linux does offer integrated lock
management support.

I don't recall which Linux version first saw lock manager integration,
but it was a while back. In RHEL clusters running GFS2 for instance,
that DLM is a fundamental part of file access.

In a macOS cluster, the equivalent is Xsan:
https://help.apple.com/serverapp/mac/5.0/#/apd6ec9abf60

There are other examples.

For many developers, the database and the database developers deal with
this coordination, and not the app developer. Just as RMS deals with
this for coordinating punched-card access on OpenVMS.

--
Pure Personal Opinion | HoffmanLabs LLC

On Sunday, October 10, 2021 at 1:44:55 PM UTC+13, Stephen Hoffman wrote:
> Various apps can use range locks or local locking or other
> synchronization for that, but Linux does offer integrated lock
> management support.

Nothing VMS-style. All I can find is this <https://manpages.debian.org/bullseye/manpages-dev/fcntl.2.en.html>, which describes about 3 different styles of locking, all on byte ranges, and all associated with files.

> I don't recall which Linux version first saw lock manager integration,
> but it was a while back. In RHEL clusters running GFS2 for instance,
> that DLM is a fundamental part of file access.

Might be a Red-Hat-specific feature, then.

> For many developers, the database and the database developers deal with
> this coordination, and not the app developer.

Precisely. And with a proper database, you can use transactions as a way to avoid locking in many cases.

On 10/10/21 01:12, Lawrence D’Oliveiro wrote:
> On Sunday, October 10, 2021 at 11:44:03 AM UTC+13, Dave Froble wrote:
>
>> But even if another "application" handles other files, there is still
>> the issue of today's disks being block based ...
>
> Those same block-based disks (and block-based SSDs, and what have you) work fine on POSIX/*nix/Linux OSes. They always have worked fine, and they will continue to work fine into the future.
>
> The key point is that VMS shied away from a radical idea that Unix embraced: that the filesystem itself should abstract away the need for blocking and deblocking, and offer up a file as just a stream of n bytes, with no requirement on n being a multiple of any integer greater than 1.

Yes, and at the lowest level, it's completely transparent to data and
it's format. Filesystems, structure and format should be layered in top
of that. Obvious really...

Chris

On 10/9/2021 10:13 PM, Lawrence D’Oliveiro wrote:
> On Sunday, October 10, 2021 at 1:44:55 PM UTC+13, Stephen Hoffman wrote:
>> Various apps can use range locks or local locking or other
>> synchronization for that, but Linux does offer integrated lock
>> management support.
>
> Nothing VMS-style. All I can find is this <https://manpages.debian.org/bullseye/manpages-dev/fcntl.2.en.html>, which describes about 3 different styles of locking, all on byte ranges, and all associated with files.
>
>> I don't recall which Linux version first saw lock manager integration,
>> but it was a while back. In RHEL clusters running GFS2 for instance,
>> that DLM is a fundamental part of file access.
>
> Might be a Red-Hat-specific feature, then.

The DLM and GFS2 was added to Linux kernel 2.6.19 in 2006.

And it sure does remind one of VMS.

#include <libdlm.h>
int dlm_lock(uint32_t mode,
struct dlm_lksb *lksb,
uint32_t flags,
const void *name,
unsigned int namelen,
uint32_t parent, /* unused */
void (*astaddr) (void *astarg),
void *astarg,
void (*bastaddr) (void *astarg),
void *range); /* unused */

vs

int sys$enqw (unsigned int efn, unsigned int lkmode, struct _lksb *lksb,
unsigned int flags, void *resnam, unsigned int parid, void
(*astadr)(__unknown_params), unsigned __int64 astprm, void
(*blkast)(__unknown_params), unsigned int acmode, unsigned int
rsdm_id,...);

Arne

On 2021-10-10 02:13:48 +0000, Lawrence D’Oliveiro said:

> On Sunday, October 10, 2021 at 1:44:55 PM UTC+13, Stephen Hoffman wrote:
>> Various apps can use range locks or local locking or other
>> synchronization for that, but Linux does offer integrated lock
>> management support.
>
> Nothing VMS-style.

https://linux.die.net/man/3/dlm_lock

All of which should look familiar to folks familiar with the OpenVMS
DLM and the associated system service calls including $enq.

>> I don't recall which Linux version first saw lock manager integration,
>> but it was a while back. In RHEL clusters running GFS2 for instance,
>> that DLM is a fundamental part of file access.
>
> Might be a Red-Hat-specific feature, then.

Nope. Bog-standard Linux. DLM has been part of the Linux distro for
~fifteen years, IIRC.

--
Pure Personal Opinion | HoffmanLabs LLC

On Sunday, 10 October 2021 at 22:54:09 UTC+1, Stephen Hoffman wrote:
> On 2021-10-10 02:13:48 +0000, Lawrence D’Oliveiro said:
>
> > On Sunday, October 10, 2021 at 1:44:55 PM UTC+13, Stephen Hoffman wrote:
> >> Various apps can use range locks or local locking or other
> >> synchronization for that, but Linux does offer integrated lock
> >> management support.
> >
> > Nothing VMS-style.
> https://linux.die.net/man/3/dlm_lock
>
> All of which should look familiar to folks familiar with the OpenVMS
> DLM and the associated system service calls including $enq.
> >> I don't recall which Linux version first saw lock manager integration,
> >> but it was a while back. In RHEL clusters running GFS2 for instance,
> >> that DLM is a fundamental part of file access.
> >
> > Might be a Red-Hat-specific feature, then.
> Nope. Bog-standard Linux. DLM has been part of the Linux distro for
> ~fifteen years, IIRC.
> --
> Pure Personal Opinion | HoffmanLabs LLC

DLM a standard part of Linux? But *which* DLM, *which* Linux?

Around fifteen years ago might tie in conveniently with the timing of when HPQ open-sourced important former Tru64 components like its Distributed Lock Manager (which itself was introduced into Tru64 V5, as part of the TruCluster Server), and AdvFS, and other stuff which helped make Tru64 a technologically leading UNIX.

I couldn't quickly find any definitive evidence of this, however. Which may well just be me on a Monday morning.

On 2021-10-08, Lawrence D?Oliveiro <lawrencedo99@gmail.com> wrote:
> On Saturday, October 9, 2021 at 4:37:26 AM UTC+13, Stephen Hoffman wrote:
>> Having looked at this back in the VAX era, the slow process creations
>> our apps were incurring were arising from inefficiencies within the DCL
>> spawn-related processing, and not from within the OpenVMS process
>> creation overhead.
>>
>> Once that was identified and the obvious work-around implemented,
>> spawns were pretty speedy even VAX-era.
>>
>> To Simon's comment, how DCL gets mapped into process address space is
>> just ugly, too. And hard to debug.
>
> But the whole reason why DCL maps into a process in this way, so that user-mode code can be repeatedly loaded, run and then wiped from the same process, was precisely to avoid multiple process creations. Now you are saying that the DCL mechanism itself contributes to the overhead of process creations!
>

Sort-of wiped. Turns out that's a rather important difference. :-)

BTW, spawn is when you actually do want a new subprocess for some
reason. Not the same thing as loading an image into the existing
DCL process.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 2021-10-09, Dave Froble <davef@tsoft-inc.com> wrote:
> On 10/9/2021 4:55 PM, Stephen Hoffman wrote:
>>
>> And here I was trying to explicitly not slag on RMS and its
>> capabilities, as that'd solely serve provoke a torrent of folks quite
>> reasonably pointing out that RMS is perfect for {app}.
>>

Actually to those people I would say that RMS is pretty much perfect - for
applications written in Macro-32 that require record-level access.

The RMS APIs perfectly match the huge level of work required in writing
a full application in Macro-32 (that would be far easily written in a
higher-level language) and perfectly matches Macro-32's utter inability
to provide any meaningful abstraction layers in Macro-32 source code
when compared to abstractions available in those same higher-level languages.

The RMS APIs are what you would have designed in the 1970s. They are not
what you would design in this century.

>
> Which it is, for those apps that need and use it's capabilities. Well,
> maybe not "perfect". There is that lack of definition of data fields
> that is so lacking in RMS. What I believe you call "marshaling".
>

I would not describe this as marshalling, as marshalling is the conversion
of data from one format to another.

In this case, you would want the fields to be directly accessible
from source code via a field-level API instead of a record-level API
(as RMS is).

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 2021-10-11 17:29:52 +0000, Simon Clubley said:

> On 2021-10-08, Lawrence D?Oliveiro <lawrencedo99@gmail.com> wrote:
>> On Saturday, October 9, 2021 at 4:37:26 AM UTC+13, Stephen Hoffman wrote:
>>> Having looked at this back in the VAX era, the slow process creations
>>> our apps were incurring were arising from inefficiencies within the DCL
>>> spawn-related processing, and not from within the OpenVMS process
>>> creation overhead.
>>>
>>> Once that was identified and the obvious work-around implemented,
>>> spawns were pretty speedy even VAX-era.
>>>
>>> To Simon's comment, how DCL gets mapped into process address space is
>>> just ugly, too. And hard to debug.
>>
>> But the whole reason why DCL maps into a process in this way, so that
>> user-mode code can be repeatedly loaded, run and then wiped from the
>> same process, was precisely to avoid multiple process creations.

Supervisor-mode interpreters and user-mode memory management rundown
does allow user context to be maintained across a series of
executables, yes.

There are obviously other ways for an operating system to maintain user
context across a series of executable image invocations, of course.

>> Now you are saying that the DCL mechanism itself contributes to the
>> overhead of process creations!

The subprocess creation performance issue that was effecting the local
VAX systems was found due to an internal design choice within DCL
itself.

Not an architectural-level choice.

That design choice could conceivably be altered, should spawn
performance be a performance-limiting factor for a sufficiently-large
customer.

Modern hardware is a whole lot faster than any VAX ever was, so—outside
designs that create and then run down massive numbers of processes—the
whole "process creation" discussion has largely faded.

> Sort-of wiped. Turns out that's a rather important difference. :-)
>
> BTW, spawn is when you actually do want a new subprocess for some
> reason. Not the same thing as loading an image into the existing DCL
> process.

Folks around here do seem to be learning both about OpenVMS and about
Linux in recent threads, too. 😈

--
Pure Personal Opinion | HoffmanLabs LLC

On 2021-10-11 18:25:19 +0000, Simon Clubley said:

> On 2021-10-09, Dave Froble <davef@tsoft-inc.com> wrote:
>> On 10/9/2021 4:55 PM, Stephen Hoffman wrote:
>>>
>>> And here I was trying to explicitly not slag on RMS and its
>>> capabilities, as that'd solely serve provoke a torrent of folks quite
>>> reasonably pointing out that RMS is perfect for {app}.
>>>
>
> Actually to those people I would say that RMS is pretty much perfect -
> for applications written in Macro-32 that require record-level access.
>
> The RMS APIs perfectly match the huge level of work required in writing
> a full application in Macro-32 (that would be far easily written in a
> higher-level language) and perfectly matches Macro-32's utter inability
> to provide any meaningful abstraction layers in Macro-32 source code
> when compared to abstractions available in those same higher-level
> languages.

I spend too much time poking around in those same structures from C or
C++ too, even given the keyword arguments present in the CRTL API.

Dynamically adapting to the key structures within an RMS indexed file,
for instance.

> The RMS APIs are what you would have designed in the 1970s. They are
> not what you would design in this century.

For what it does, RMS works. Works well, too. It's less than fun for
inexperienced or new users, or for work such as evolving existing
record structures and cross-app-version and rolling-upgrade
compatibility.

The provided abstractions and features are comparatively weak.

>> Which it is, for those apps that need and use it's capabilities. Well,
>> maybe not "perfect". There is that lack of definition of data fields
>> that is so lacking in RMS. What I believe you call "marshaling".
>
> I would not describe this as marshalling, as marshalling is the
> conversion of data from one format to another.
>
> In this case, you would want the fields to be directly accessible from
> source code via a field-level API instead of a record-level API (as RMS
> is).

Elsewhere, I'm working with stuff where you tell the run-time "go store
this" with wads of in-memory data (objects, etc), and it goes off and
stores it.

The calls serialize the data in various formats, and the app developer
doesn't need to know the details.

For larger wads of stored data or where, various databases will account
for the structures of the stored data and map the memory structures to
or from the persistent storage structures.

The apps don't need to change unrelated field-level references whenever
the underlying storage structures change, either due to changes to the
underlying storage, or changes to the app.

--
Pure Personal Opinion | HoffmanLabs LLC

[I previously posted something similar, but it never seemed to make it into Google Groups.]

On Monday, October 11, 2021 at 9:54:19 PM UTC+13, John Wallace wrote:
> DLM a standard part of Linux? But *which* DLM, *which* Linux?

Turns out it’s a standard package in Debian <https://packages.debian.org/bullseye/libdlm3>, and therefore in most Linux distros. I thought there had to be an accompanying daemon, which is mentioned here <https://packages.debian.org/bullseye/libdlmcontrol3>.

The API is about as faithful a copy of the original VMS one as it is possible to be <https://manpages.debian.org/bullseye/libdlm-dev/dlm_lock.3.en.html>: the lock modes match one for one, it uses the term “AST” for the completion notification, and even the timeout is measured in centiseconds (good old VMS ticks), which I don’t think are used in any other Linux API.

Note, however, that you can get a file descriptor <https://manpages.debian.org/bullseye/libdlm-dev/libdlm.3.en.html> for use in event loops, if you’d rather implement your completion notifications that way.

On Monday, October 11, 2021 at 1:30:36 AM UTC+13, chris wrote:
>
> On 10/10/21 01:12, Lawrence D’Oliveiro wrote:
>>
>> The key point is that VMS shied away from a radical idea that Unix embraced:
>> that the filesystem itself should abstract away the need for blocking and
>> deblocking, and offer up a file as just a stream of n bytes, with no requirement
>> on n being a multiple of any integer greater than 1.
>>
> Yes, and at the lowest level, it's completely transparent to data and
> it's format. Filesystems, structure and format should be layered in top
> of that. Obvious really...

Which brings us to a point I’ve made before: the Linux kernel already runs on every architecture that VMS users and developers might care about, now and into the future. It already has a range of drivers for common (and not-so-common) hardware, including that in enterprise use. It includes a robust, high-performance TCP/IP stack.

Wouldn’t it be easier to just keep the parts of VMS that users and developers need, and implement them as a compatibility layer on top of a Linux kernel? And just scrap the rest. Wouldn’t that save a lot of effort?

On 10/11/2021 4:54 AM, John Wallace wrote:
> On Sunday, 10 October 2021 at 22:54:09 UTC+1, Stephen Hoffman wrote:
>> On 2021-10-10 02:13:48 +0000, Lawrence D’Oliveiro said:
>>> On Sunday, October 10, 2021 at 1:44:55 PM UTC+13, Stephen Hoffman wrote:
>>>> I don't recall which Linux version first saw lock manager integration,
>>>> but it was a while back. In RHEL clusters running GFS2 for instance,
>>>> that DLM is a fundamental part of file access.
>>>
>>> Might be a Red-Hat-specific feature, then.
>> Nope. Bog-standard Linux. DLM has been part of the Linux distro for
>
> DLM a standard part of Linux? But *which* DLM, *which* Linux?

The DLM itself is in the kernel since 2.6.19 so *all* with
a kernel newer than 2006.

Arne