I'm talking about the section files in the Figure 3-1 of the April 2001 edition of the OpenVMS Performance Management manual. (OpenVMS Version 7.3)

In the "HP OpenVMS Programming Concepts Manual" it says:

13.5.6 Sections

A section is a disk file or a portion of a disk file containing data or instructions that can be brought into memory and made available to a process for manipulation and execution. A section can also be one or more consecutive page frames in physical memory or I/O space; such sections, which require you to specify page frame number (PFN) mapping, are discussed in Section 13.5.6.15.

And according to the figure, it can be read from or written to. How is this different from a database?

On 2022-08-30 19:16:23 +0000, alanfe...@gmail.com said:

> I'm talking about the section files in the Figure 3-1 of the April 2001
> edition of the OpenVMS Performance Management manual. (OpenVMS Version
> 7.3)
>
> In the "HP OpenVMS Programming Concepts Manual" it says:
>
> 13.5.6 Sections
>
> A section is a disk file or a portion of a disk file containing data or
> instructions that can be brought into memory and made available to a
> process for manipulation and execution. A section can also be one or
> more consecutive page frames in physical memory or I/O space; such
> sections, which require you to specify page frame number (PFN) mapping,
> are discussed in Section 13.5.6.15.
> And according to the figure, it can be read from or written to. How is
> this different from a database?

A section file...

....is a form of persistent storage available for in-memory data structures.

....can be used to persist a user-written or third-party in-memory
database. Or can persist a pile of shared data structures. Or other
such.

....lacks any database-like APIs and features; it's a means to map
(transfer) arbitrary user data from persistent storage into memory, and
from memory into persistent storage.

....sections are the underpinning of Fortran COMMON, the C/Unix mmap()
APIs, of parts of TPU customizations, and related mechanisms.

....biggest developer issues with using a system section tends to be
flushing memory at the appropriate times to reduce or avoid data loss
should an app or system crash or power outage arise, and ensuring
multiple processes are correctly synchronized; interlocking, flushing
data to persistent storage, etc.

For more info: OpenVMS Programming Concepts Volume I chapter 5, 6, and
7, and all of Part III, and Programming Concepts Volume II Chapter 10.

Current VSI OpenVMS documentation: https://docs.vmssoftware.com

If you do want an in-memory database, SQLite can do that, among other
options. https://www.sqlite.org/inmemorydb.html

--
Pure Personal Opinion | HoffmanLabs LLC

On 8/30/2022 3:16 PM, alanfe...@gmail.com wrote:
> I'm talking about the section files in the Figure 3-1 of the April
> 2001 edition of the OpenVMS Performance Management manual. (OpenVMS
> Version 7.3)
>
> In the "HP OpenVMS Programming Concepts Manual" it says:
>
> 13.5.6 Sections
>
> A section is a disk file or a portion of a disk file containing data
> or instructions that can be brought into memory and made available to
> a process for manipulation and execution. A section can also be one
> or more consecutive page frames in physical memory or I/O space; such
> sections, which require you to specify page frame number (PFN)
> mapping, are discussed in Section 13.5.6.15.
>
> And according to the figure, it can be read from or written to. How
> is this different from a database?

There are many concepts here.

And some of them have some tricky details.

But let me try and explain it at a very high level.

1) global section

A global section is a piece of physical memory mapped into
the virtual address space of multiple processes so it can be
used for communication between processes. Basically processes
read from and write to shared memory.

2) global section file

A global section file is a file used as backing
storage (persistence) for a global section, which means
that any changes survive program terminations
and even system reboot.

3) memory mapped file

A memory mapped file is a normal file being mapped
in memory and accessed as memory.

This is 3 different use cases, but they use the same
system services. Different arguments to the call
determine whether it is an in memory only thing
(technically mapped to the page file) or mapped to
an on disk file. And whether that on disk file is
created specifically to act as backing for memory
or is created with real data using normal IO is
up to the application.

It is very different from a database. The above
provide the application with an area in virtual memory
with N bytes that maps to N bytes in a file on disk.
The application just get a sequence of N bytes - any
structure on those bytes are entirely up to the
application code. A database whether relational or NoSQL
provide a lot of structure to the application.

A TPU$SECTION file is really compiled byte code from
TPU source code. Similar to Java class files. Whether
TPU access it as a memory mapped file or using normal
IO that I don't know (and I have been using TPU since
VMS 4.4 around 1986-87). The name obviously give some
associations.

Arne

On 8/30/2022 6:10 PM, Stephen Hoffman wrote:
> On 2022-08-30 19:16:23 +0000, alanfe...@gmail.com said:
>> I'm talking about the section files in the Figure 3-1 of the April
>> 2001 edition of the OpenVMS Performance Management manual. (OpenVMS
>> Version 7.3)
>>
>> In the "HP OpenVMS Programming Concepts Manual" it says:
>>
>> 13.5.6 Sections
>>
>> A section is a disk file or a portion of a disk file containing data
>> or instructions that can be brought into memory and made available to
>> a process for manipulation and execution. A section can also be one or
>> more consecutive page frames in physical memory or I/O space; such
>> sections, which require you to specify page frame number (PFN)
>> mapping, are discussed in Section 13.5.6.15.
>> And according to the figure, it can be read from or written to. How is
>> this different from a database?
>
> A section file...
>
> ...is a form of persistent storage available for in-memory data structures.
>
> ...can be used to persist a user-written or third-party in-memory
> database.  Or can persist a pile of shared data structures. Or other such.
>
> ...lacks any database-like APIs and features; it's a means to map
> (transfer) arbitrary user data from persistent storage into memory, and
> from memory into persistent storage.
>
> ...sections are the underpinning of Fortran COMMON, the C/Unix mmap()
> APIs, of parts of TPU customizations, and related mechanisms.

I agree with everything except the Fortran common block thing.

A Fortran common block is a psect, but not this kind of section
(per default - it can of course be mapped like any other memory).

Arne

On Tuesday, August 30, 2022 at 3:16:25 PM UTC-4, alanfe...@gmail.com wrote:
> I'm talking about the section files in the Figure 3-1 of the April 2001 edition of the OpenVMS Performance Management manual. (OpenVMS Version 7.3)
>
> In the "HP OpenVMS Programming Concepts Manual" it says:
>
> 13.5.6 Sections
>
> A section is a disk file or a portion of a disk file containing data or instructions that can be brought into memory and made available to a process for manipulation and execution. A section can also be one or more consecutive page frames in physical memory or I/O space; such sections, which require you to specify page frame number (PFN) mapping, are discussed in Section 13.5.6.15.
>
> And according to the figure, it can be read from or written to. How is this different from a database?

Thanks Hoff and Arne Vajhoj, and anyone else who chimes in later!

On 2022-08-31 00:07:29 +0000, Arne Vajhj said:

> On 8/30/2022 6:10 PM, Stephen Hoffman wrote:
>>
>> A section file...
>>
>> ...is a form of persistent storage available for in-memory data structures.
>>
>> ...can be used to persist a user-written or third-party in-memory
>> database.  Or can persist a pile of shared data structures. Or other
>> such.
>>
>> ...lacks any database-like APIs and features; it's a means to map
>> (transfer) arbitrary user data from persistent storage into memory, and
>> from memory into persistent storage.
>>
>> ...sections are the underpinning of Fortran COMMON, the C/Unix mmap()
>> APIs, of parts of TPU customizations, and related mechanisms.
>
> I agree with everything except the Fortran common block thing.
>
> A Fortran common block is a psect, but not this kind of section (per
> default - it can of course be mapped like any other memory).

Sure.

It's a section of virtual memory with backing (and sharable) storage,
with the salient difference being whether it uses static
mapping—Fortran, BASIC, C, etc., common storage involving the linker—or
use dynamic mapping—run-time calls to $crmpsc, etc.

PFN mapping also differs in details, too.

The whole thing is lacking an abstraction layer or two, but entirely
workable for those building their own tooling.

Persisting data with sections is Less Than Fun; a local or distributed
transaction log—RAIS—may or will become interesting, even with local
persistent storage, as flushing pages of virtual memory to persistent
storage can get tricky, and that's before we get to outage-related
details such as 8 KiB storage mapped to half KiB storage sectors, etc.
Persistent memory can't get here fast enough for these and related
tasks, not that Intel didn't just bail out of that whole Optane market,
seemingly now betting on storage access via CXL.

--
Pure Personal Opinion | HoffmanLabs LLC

On 9/1/2022 12:53 PM, Stephen Hoffman wrote:
> On 2022-08-31 00:07:29 +0000, Arne Vajhj said:
>
>> On 8/30/2022 6:10 PM, Stephen Hoffman wrote:
>>>
>>> A section file...
>>>
>>> ...is a form of persistent storage available for in-memory data
>>> structures.
>>>
>>> ...can be used to persist a user-written or third-party in-memory
>>> database.  Or can persist a pile of shared data structures. Or other
>>> such.
>>>
>>> ...lacks any database-like APIs and features; it's a means to map
>>> (transfer) arbitrary user data from persistent storage into memory,
>>> and from memory into persistent storage.
>>>
>>> ...sections are the underpinning of Fortran COMMON, the C/Unix mmap()
>>> APIs, of parts of TPU customizations, and related mechanisms.
>>
>> I agree with everything except the Fortran common block thing.
>>
>> A Fortran common block is a psect, but not this kind of section (per
>> default - it can of course be mapped like any other memory).
>
> Sure.
>
> It's a section of virtual memory with backing (and sharable) storage,
> with the salient difference being whether it uses static
> mapping—Fortran, BASIC, C, etc., common storage involving the linker—or
> use dynamic mapping—run-time calls to $crmpsc, etc.

????

A Fortran common block does not share data between processes
and does not map data to disk.

It is basically just a named bundle of global variables.

(VAX C and DEC C with /EXTERN=COMMON_BLOCK use the same
construct for global variables)

To share with other processes or map data to disk something
more is needed like mapping it via SYS$CRMPSC or SYS$MGLBSC.

Or a bit more old fashioned. Use the common block in
a block data and build that to a shareable installed
writeable image.

Arne

On Thursday, September 1, 2022 at 8:37:00 PM UTC-4, Arne Vajhøj wrote:
> On 9/1/2022 12:53 PM, Stephen Hoffman wrote:

> > It's a section of virtual memory with backing (and sharable) storage,
> > with the salient difference being whether it uses static
> > mapping—Fortran, BASIC, C, etc., common storage involving the linker—or
> > use dynamic mapping—run-time calls to $crmpsc, etc.
> ????
>
> A Fortran common block does not share data between processes
> and does not map data to disk.

Arne, please read more carefully (and more manuals)

Stephen wrote "involving the linker".
Once you use the linker to make a Psect common, shared it IS shared between processes (using the same linker instructions) and it _does_ map data back to the disk.
I suppose you could say the linker allows one to link processes, not just bake images.

https://www.digiater.nl/openvms/doc/alpha-v8.3/83final/4548/4548pro_006.html#itn_image_ch
https://www.digiater.nl/openvms/doc/alpha-v8.3/83final/4548/4548pro_003.html#itn_common_model_sec

Hein

On Thursday, September 1, 2022 at 8:37:00 PM UTC-4, Arne Vajhøj wrote:
> On 9/1/2022 12:53 PM, Stephen Hoffman wrote:

> > It's a section of virtual memory with backing (and sharable) storage,
> > with the salient difference being whether it uses static
> > mapping—Fortran, BASIC, C, etc., common storage involving the linker—or
> > use dynamic mapping—run-time calls to $crmpsc, etc.
> ????
>
> A Fortran common block does not share data between processes
> and does not map data to disk.

I posted a knee-jerk reaction after reading that first line, which I 'deleted' but may have made it out there.

It suggested to Arne to please read more carefully (and more manuals) as Stephen wrote "involving the linker".
A Fortran common block _can_ be used to share data between processes and _can_ map to disk.
It just requires a good bit of linker involvement, more than most remember how to do.

Reading on I see Arne does know all this:

Arne> Or a bit more old fashioned. Use the common block in a block data and build that to a shareable installed writeable image.

Cheers,
Hein

On 2022-09-02 02:36, Arne Vajhøj wrote:
> On 9/1/2022 12:53 PM, Stephen Hoffman wrote:
>> On 2022-08-31 00:07:29 +0000, Arne Vajhj said:
>>
>>> On 8/30/2022 6:10 PM, Stephen Hoffman wrote:
>>>>
>>>> A section file...
>>>>
>>>> ...is a form of persistent storage available for in-memory data
>>>> structures.
>>>>
>>>> ...can be used to persist a user-written or third-party in-memory
>>>> database.  Or can persist a pile of shared data structures. Or other
>>>> such.
>>>>
>>>> ...lacks any database-like APIs and features; it's a means to map
>>>> (transfer) arbitrary user data from persistent storage into memory,
>>>> and from memory into persistent storage.
>>>>
>>>> ...sections are the underpinning of Fortran COMMON, the C/Unix
>>>> mmap() APIs, of parts of TPU customizations, and related mechanisms.
>>>
>>> I agree with everything except the Fortran common block thing.
>>>
>>> A Fortran common block is a psect, but not this kind of section (per
>>> default - it can of course be mapped like any other memory).
>>
>> Sure.
>>
>> It's a section of virtual memory with backing (and sharable) storage,
>> with the salient difference being whether it uses static
>> mapping—Fortran, BASIC, C, etc., common storage involving the
>> linker—or use dynamic mapping—run-time calls to $crmpsc, etc.
>
> ????
>
> A Fortran common block does not share data between processes
> and does not map data to disk.
>
> It is basically just a named bundle of global variables.
>
> (VAX C and DEC C with /EXTERN=COMMON_BLOCK use the same
> construct for global variables)
>
> To share with other processes or map data to disk something
> more is needed like mapping it via SYS$CRMPSC or SYS$MGLBSC.
>
> Or a bit more old fashioned. Use the common block in
> a block data and build that to a shareable installed
> writeable image.

I just love how this is identical to RSX (again). :-D

Johnny

On 2022-09-02 14:38:27 +0000, Hein RMS van den Heuvel said:

> On Thursday, September 1, 2022 at 8:37:00 PM UTC-4, Arne Vajhøj wrote:
>> On 9/1/2022 12:53 PM, Stephen Hoffman wrote:
>
>>> It's a section of virtual memory with backing (and sharable) storage,
>>> with the salient difference being whether it uses static
>>> mapping—Fortran, BASIC, C, etc., common storage involving the linker—or
>>> use dynamic mapping—run-time calls to $crmpsc, etc.
>> ????
>>
>> A Fortran common block does not share data between processes and does
>> not map data to disk.
>
> I posted a knee-jerk reaction after reading that first line, which I
> 'deleted' but may have made it out there.
>
> It suggested to Arne to please read more carefully (and more manuals)
> as Stephen wrote "involving the linker".
> A Fortran common block _can_ be used to share data between processes
> and _can_ map to disk.

Which are the two most common reasons I've found a Fortran or BASIC
COMMON useful. Pun intended.

Other reasons can include the creation of a more perfectly-convoluted
build procedure, or to make OpenVMS upgrades more problematic, or to
add fodder for data synchronization and data race conditions, of
course. 😜

Slightly more seriously, I prefer to avoid using a COMMON, due to
position-dependence and app update issues encountered over the years.
But when working in Fortran or BASIC, some trade-offs can be necessary.

> It just requires a good bit of linker involvement, more than most
> remember how to do.

Or more linker involvement than most would prefer to remember.

--
Pure Personal Opinion | HoffmanLabs LLC

On 3/09/2022 8:05 am, Stephen Hoffman wrote:
> On 2022-09-02 14:38:27 +0000, Hein RMS van den Heuvel said:
>
>> On Thursday, September 1, 2022 at 8:37:00 PM UTC-4, Arne Vajhøj
>> wrote:
>>> On 9/1/2022 12:53 PM, Stephen Hoffman wrote:
>>
>>>> It's a section of virtual memory with backing (and sharable)
>>>> storage, with the salient difference being whether it uses
>>>> static mapping—Fortran, BASIC, C, etc., common storage
>>>> involving the linker—or use dynamic mapping—run-time calls to
>>>> $crmpsc, etc.
>>> ????
>>>
>>> A Fortran common block does not share data between processes and
>>> does not map data to disk.
>>
>> I posted a knee-jerk reaction after reading that first line, which
>> I 'deleted' but may have made it out there.
>>
>> It suggested to Arne to please read more carefully (and more
>> manuals) as Stephen wrote "involving the linker". A Fortran common
>> block _can_ be used to share data between processes and _can_ map
>> to disk.
>
> Which are the two most common reasons I've found a Fortran or BASIC
> COMMON useful. Pun intended.
>
> Other reasons can include the creation of a more perfectly-convoluted
> build procedure, or to make OpenVMS upgrades more problematic, or to
> add fodder for data synchronization and data race conditions, of
> course. 😜
>
> Slightly more seriously, I prefer to avoid using a COMMON, due to
> position-dependence and app update issues encountered over the years.
> But when working in Fortran or BASIC, some trade-offs can be
> necessary.
>
>> It just requires a good bit of linker involvement, more than most
>> remember how to do.
>
> Or more linker involvement than most would prefer to remember.
>
>
>
>

And COBOL EXTERNAL