On Saturday, May 14, 2022 at 4:32:20 PM UTC-7, Stephen Hoffman wrote:
>
> OpenVMS uses what amounts to segmented virtual addressing, and the
> OpenVMS virtual address space design ties directly back to VAX. This
> at least through OpenVMS I64, and haven't checked OpenVMS x86-64 here.
>
> VAX had P0 user space where ~all VAX user apps and user data resides,
> 00000000 to 3FFFFFFF, P1 from 40000000 to 7FFFFFFF for the control
> region and CLI and baggage, and the remainder for S0 system space, and
> later S1 for more system space with VAX Extended Virtual Addressing.
> I'll here ignore VAX Extended Physical Addressing.
>
> Each of these P0, P1, S0, and S2 spaces are 30 bits of virtual addressing..
>
> Alpha uses 64-bit virtual and always has, and preserves the existing
> VAX 32-bit design with P0 and P1 at the lowest addresses
> (00000000.00000000 to 00000000.7FFFFFFF), with S0 and S1 at the highest
> (FFFFFFFF.80000000 to FFFFFFFF.FFFFFFFF, and adds P2 and S2 into the
> great huge gap between those two ranges. OpenVMS Alpha V7.0 opened up
> user access for P2 and S2 addressing.

While reading Hunter Goatley's explanation of the VMS memory map from an old article on how to write privileged code (https://hunter.goatley.com/writing-vms-privileged-code/part-i-the-fundamentals-part-1/), I realized that the size of P0 must be the explanation for why I could only malloc() up to 950MB of memory in the memtester port I did, until I switched to using LIB$GET_VM_64() and 64-bit pointers.

I was thinking of a 32-bit UNIX memory map split 50/50 between user and kernel space. Now that I have some experience with VMS long and short pointers, the only obnoxious quirks that I've encountered so far are that main() wants to use short pointers for argv[]: you can use "/pointer_size=long=argv" to get around that, but then getopt() and friends only work with short argv pointers.

Regards,
Jake

On Monday, May 16, 2022 at 8:21:31 AM UTC-7, chris wrote:
> I can see why there is a need for 32 bit addressing for legacy reasons.
> If app code is 32 bit, it implies that VMS has some internal 32
> bit space reserved within the 64 bit total for that, probably
> via run time libraries which either output 64 bit addresses, or have
> some way of informing vms to put the code into a certain 64 bit space
> and starting address...

I think the key to understanding how VMS handles 64-bit space internally is to read about 64-bit descriptors and the new routines for manipulating them (Programming Concepts Manual, Vol I: Part IV. Appendixes: Macros and Examples of 64-Bit Programming). There's also a 64-bit "item_list_64b" (ileb_64) that you can pass to system calls in place of the 32-bit "ile3".

Between 64-bit descriptors and 64-bit item lists, OpenVMS was retrofitted to support 64-bit addressing by the mid-1990s, but it has taken many years for applications and the CRTL to catch up. Having a 32-bit long and 32-bit size_t is a big part of the portability problem, but for VMS-native programs, the 64-bit support is there.

As a side note, it's fortunate that VMS has only ever run on little-endian architectures, because it's easier to expand pointers from 32 bits to 64 bits on little-endian in a backward-compatible way, since the address of the pointer stays the same regardless of the size (any extra high-order bytes are ignored). Without this, the data structures VMS uses would probably have been even trickier to retrofit.

Jake

On Monday, May 16, 2022 at 9:28:22 AM UTC-7, Jake Hamby wrote:
>
> As a side note, it's fortunate that VMS has only ever run on little-endian architectures, because it's easier to expand pointers from 32 bits to 64 bits on little-endian in a backward-compatible way, since the address of the pointer stays the same regardless of the size (any extra high-order bytes are ignored). Without this, the data structures VMS uses would probably have been even trickier to retrofit.

I forgot to mention a clever quirk of the OpenVMS calling standard that surprised me when I discovered it: all architectures pass the argument count and argument types in a register (that points to memory on VAX) on function calls, so the callee knows how many arguments were passed and how. All 32-bit params are sign-extended to 64 bits (even unsigned ints).

Therefore, any library routines can be transparently extended to take 64-bit addresses as parameters, and there are also extended versions of some POSIX/C functions that take extra parameters, which the library can detect by checking the arg count. So the OS provides a form of function overloading for all supported languages, without using name mangling.

Passing the argument info to callees in %rax is one of the few changes VSI had to make to the x86-64 ELF ABI to support VMS.

Regards,
Jake

On Saturday, May 14, 2022 at 4:32:20 PM UTC-7, Stephen Hoffman wrote:
>
> OpenVMS is the only operating system I'm aware of that chose this
> hybrid addressing path; of both 32-bit and 64-bit APIs mixed within the
> same executable images. Most other platforms decided to allow 32-bit
> and 64-bit apps and processes to coexist in parallel. But not
> co-resident within an app at run-time. This transition usually with a
> decade or more to allow apps to be migrated to 64-bit APIs for those
> platforms that have decided to deprecate and remove the older and
> problematic 32-bit APIs. As an example, macOS 10.15 "Catalina" removed
> 32-bit app and API support, and completed the migration started back
> around 10.5 "Leopard"; a ~decade ago.

You're forgetting about IBM z/OS. It's the only OS I'm aware of that has chosen an even more painful hybrid addressing path than OpenVMS. :) They started with a 24-bit addressing mode, then in the 1990s expanded to 31 bits (not 32), with one bit reserved to indicate the addressing mode. When they launched 64-bit Z and z/OS in 2000, they added a 64-bit AMODE, for a total of three.

IBM mainframers talk about their address space in terms of a "line" (at 16MB) and a "bar" (from 2GB to 4GB). Even though z/OS has been 64-bit now for over 20 years, with each new release they're still announcing that some component's storage pool can now be allocated "above the bar" (in 64-bit space), or occasionally, for very old components, "above the line" (in 31-bit space).

For Linux on Z, this is all irrelevant, and s390x looks like any other 64-bit Linux. But for z/OS, which has to be binary compatible to run customer code dating as far back as the mid-1960s, they are very much aware of not just two, but three, different addressing modes that can be mixed within the same program. And it's big-endian, so the address of pointers will change depending on how wide they are. And z/OS has multiple calling standards for each addressing mode, with the older calling standards not even using a call stack but passing pointers to parameter blocks. Fun times.

Jake

On 2022-05-16 15:21:25 +0000, chris said:

> I can see why there is a need for 32 bit addressing for legacy reasons.
> If app code is 32 bit, it implies that VMS has some internal 32 bit
> space reserved within the 64 bit total for that, probably via run time
> libraries which either output 64 bit addresses, or have some way of
> informing vms to put the code into a certain 64 bit space and starting
> address...

Existing and unmodified 32-bit user apps—apps expecting P0/P1/S0/S1
virtual address references—can't receive 64-bit descriptors and 64-bit
P2 and S2 virtual addresses from other user APIs and from other
third-party APIs and from system APIs.

With data flowing across the APIs in the other direction, the designs
of some existing user and third-party system APIs can't be
transparently modified to accept 64-bit virtual addresses arriving from
the calling apps.

Which in aggregate means that these areas of user and third-party and
OpenVMS executable code and data structures either have to sometimes
"pretend" that the P2 and S2 virtual address ranges don't exist, or
need parallel 32- and 64-bit APIs, etc.

The traditional imperative APIs used within most of
OpenVMS—descriptors, PQLs, itemlists, etc—are somewhat less flexible
around these changes than would be message-passing APIs for instance,
but message-passing API designs are not commonly encountered within
OpenVMS.

For the pedants reading and pointing to message-passing
(object-oriented, OO) code on OpenVMS: my reference here is to OpenVMS
system and RTL and device driver and related APIs. This outside of C++,
Java, Python, and other such code, all off doing C++ or Java or Python
or other message-passing tooling. BASIC mostly does well here too,
though that compiler remains 32-bit. BASIC currently
non-message-passing and probably should be updated to provide
message-passing support, too. But I digress.

--
Pure Personal Opinion | HoffmanLabs LLC

On 2022-05-16 16:58:57 +0000, Jake Hamby said:

> On Saturday, May 14, 2022 at 4:32:20 PM UTC-7, Stephen Hoffman wrote:
>>
>> OpenVMS is the only operating system I'm aware of that chose this
>> hybrid addressing path; of both 32-bit and 64-bit APIs mixed within the
>> same executable images. Most other platforms decided to allow 32-bit
>> and 64-bit apps and processes to coexist in parallel. But not
>> co-resident within an app at run-time. This transition usually with a
>> decade or more to allow apps to be migrated to 64-bit APIs for those
>> platforms that have decided to deprecate and remove the older and
>> problematic 32-bit APIs. As an example, macOS 10.15 "Catalina" removed
>> 32-bit app and API support, and completed the migration started back
>> around 10.5 "Leopard"; a ~decade ago.
>
> You're forgetting about IBM z/OS.

I try. 😉

Though I do well recall a brace of
dapper-jumpsuit-over-dapper-three-piece-suit-wearing technicians with
their ever-dapper briefcases yelling at each other in the server room.

And escorting the customer expectations management representatives out
of the office, with a request that IBM let me know when the service
tech and the parts would arrive, and to not show up before then as we
didn't need somebody sitting in the lobby reading magazines and I
wasn't going to delegate a developer or an operator to watch over the
IBM rep due to visitor security policies.

Oh, and that a pocket comb worked as well as the official IBM mainframe
chassis-accessing tool. 🤫

That site was not the usual IBM customer.

In the previous millennium, DEC spent a whole lot of time and focus
chasing after IBM from below (e.g. VAX 9000), and missed the rest of
the market "sneaking up" on DEC.

As for upward compatibility, sure, IBM has long had a better story. IBM
was also front and center in most of computing into the 1990s, with z
and 36/38/i series and particularly then with x series and the 5150,
and that all until their x series business was sold off. Now? IBM? Not
so central. In various ways, much like OpenVMS. For the existing IBM
installed base, z series sure, and 36/38/i series, while the future of
i and p series hardware looks way too much like Alpha or Itanium for my
preferences. The last posting here from a confused MVS user was some
time ago, too.

TL;DR: Whether IBM did something similar to the current mixture of 32-
and 64-bit APIs and segmented addressing and such found on OpenVMS, I
don't know.

--
Pure Personal Opinion | HoffmanLabs LLC

On 2022-05-15, Stephen Hoffman <seaohveh@hoffmanlabs.invalid> wrote:
>
> Once y'all use a flat 64-bit address space and APIs, the existing and
> hybrid memory management compatibility-focused design starts to smell
> vaguely of TKB.
>

The VMS 64-bit design comes across as a technical debt situation where,
given the nature of VMS application code, the "easier" approach was taken
initially at the expense of long-term maintenance and additional coding
work to deal with that decision.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 5/16/2022 11:21 AM, chris wrote:
> On 05/16/22 00:23, Arne Vajhøj wrote:
>> On 5/15/2022 7:15 PM, chris wrote:
>>> I think that sort of info is what I was driving at. So used to flat
>>> address spaces these days, it's difficult to imagine anything else, but
>>> it does look like VMS has plenty of space to play with, even if
>>> segmented in some way...
>>
>> I don't like the term segmented as opposed to flat to
>> describe the VMS memory either.
>
> I can see why there is a need for 32 bit addressing for legacy reasons.
> If app code is 32 bit, it implies that VMS has some internal 32
> bit space reserved within the 64 bit total for that, probably
> via run time libraries which either output 64 bit addresses, or have
> some way of informing vms to put the code into a certain 64 bit space
> and starting address...

Everything is in the same 64 bit address space.

It is just that some things can be addressed with 32 bit pointers
and other things cannot.

Demo:

$ type ptrfun.c
#include <stdio.h>
#include <stdlib.h>

#pragma pointer_size save
#pragma pointer_size 32
typedef char *char_ptr32;
#pragma pointer_size 64
typedef char *char_ptr64;
#pragma pointer_size restore

void api32(char_ptr32 a32)
{ printf("32 bit API see: %016p\n", a32);
}

void api64(char_ptr64 a64)
{ printf("64 bit API see: %016llp\n", a64);
}

int main()
{ char b[1];
printf("stack address : %016p (%d)\n", &b, sizeof(&b));
api32(b);
api64(b);
char_ptr32 p32 = _malloc32(1);
printf("32 bit pointer: %016p (%d)\n", p32, sizeof(p32));
api32(p32);
api64(p32);
char_ptr64 p64 = _malloc64(1);
printf("64 bit pointer: %016llp (%d)\n", p64, sizeof(p64));
api32(p64);
api64(p64);
for(int i = 0; i < 16; i++) _malloc64(0x10000000); // force next
allocation up in memory
p64 = _malloc64(1);
printf("64 bit pointer: %016llp (%d)\n", p64, sizeof(p64));
api32(p64);
api64(p64);
return 0;
} $ cc/pointer=64 ptrfun

api32(p64);
..........^
%CC-W-MAYLOSEDATA2, In this statement, "p64" has a larger data size than
"short pointer to char". Assignment can result in data los
s.
at line number 33 in file DISK2:[ARNE]ptrfun.c;4

api32(p64);
..........^
%CC-W-MAYLOSEDATA2, In this statement, "p64" has a larger data size than
"short pointer to char". Assignment can result in data los
s.
at line number 38 in file DISK2:[ARNE]ptrfun.c;4
$ link ptrfun
%LINK-W-WRNERS, compilation warnings
in module PTRFUN file DISK2:[ARNE]ptrfun.OBJ;3
$ run ptrfun
stack address : 000000007AE27A48 (4)
32 bit API see: 000000007AE27A48
64 bit API see: 000000007AE27A48
32 bit pointer: 000000000004B828 (4)
32 bit API see: 000000000004B828
64 bit API see: 000000000004B828
64 bit pointer: 0000000080000010 (8)
32 bit API see: 0000000080000010
64 bit API see: 0000000080000010
64 bit pointer: 00000001A0000030 (8)
32 bit API see: 00000000A0000030
64 bit API see: 00000001A0000030

The compiler does what it can - it gives a warning - it
can't fix the problem.

Arne

On 2022-05-16 18:24:00 +0000, Simon Clubley said:

> On 2022-05-15, Stephen Hoffman <seaohveh@hoffmanlabs.invalid> wrote:
>>
>> Once y'all use a flat 64-bit address space and APIs, the existing and
>> hybrid memory management compatibility-focused design starts to smell
>> vaguely of TKB.
>>
>
> The VMS 64-bit design comes across as a technical debt situation where,
> given the nature of VMS application code, the "easier" approach was
> taken initially at the expense of long-term maintenance and additional
> coding work to deal with that decision.

Compatibility is always a trade-off, and the complexities inherent in
compatibility inevitably accrete.

(And you can't churn APIs and tools too quickly, as the lack of
compatibility means few or none will want to continue to develop for
and update for your platform.)

The existing hybrid segmented approach wasn't easy on OpenVMS
development, and hasn't been good for substantial app overhauls and new
app work, though the design has been vastly better for existing apps
and existing installations.

Parallel 32- and 64-bit environments would have been both more work in
some ways and easier in some ways, though would have required the
third-parties and apps and tools and APIs to migrate to 64-bit.

Leave the 32-bit apps and APIs and tools alone, save for maintenance.
Port the existing apps and APUs and tools to 64-bit as demand arises.

Following the parallel approach would likely also have deferred the
availability of 64-bit addressing features for those apps that then
required it, too.

The hybrid approach in some ways reminds me of porting code back to
VAX. That ends up being a slog, due to missing calls, missing 64-bit
support, and related. And the folks still on VAX can have less
inclination to move forward.

No good answer, here.

--
Pure Personal Opinion | HoffmanLabs LLC

In article <t5tq5l$5fi$1@gioia.aioe.org>, chris-nospam@tridac.net (chris)
wrote:

> I can see why there is a need for 32 bit addressing for legacy
> reasons. If app code is 32 bit, it implies that VMS has some
> internal 32-bit space reserved within the 64 bit total for that,
> probably via run time libraries which either output 64 bit
> addresses, or have some way of informing vms to put the code
> into a certain 64 bit space and starting address...

This is not done in the same way as more conventional 64-bit operating
systems run 32-code.

The VMS APIs were all originally defined as 32-bit, naturally, but they
were defined in terms of absolute sizes, not in terms of the sizes of
language types. To get a 64-bit environment working quickly, only some of
the APIs were given 64-bit implementations, and they're all in the same
namespaces as the 32-bit APIs, so they have different names.

Since the DEC Alpha, the original 64-bit platform for VMS, simply did not
/have/ a 32-bit mode, "32-bit code" stores the low half of pointers in
memory, and sign-extends them when loading them into registers to use for
accessing memory.

Software written for 64-bit VMS expects to use different APIs to allocate
and manage memory outside the 2GB regions at the top and bottom of the
64-bit address space, and has different types for 64-bit pointers.
Porting software for more conventional 64-bit architectures to VMS can
thus present problems.

This may seem weird. VMS was the last OS written in the expectation that
a lot of programming would be in assembler, and without any planning at
design time for 64-bit addressing. It's quite sophisticated assembler,
but the OS is defined that way, rather than in terms of a high-level
language.

John

On 05/16/22 18:22, Stephen Hoffman wrote:
> On 2022-05-16 15:21:25 +0000, chris said:
>
>> I can see why there is a need for 32 bit addressing for legacy
>> reasons. If app code is 32 bit, it implies that VMS has some internal
>> 32 bit space reserved within the 64 bit total for that, probably via
>> run time libraries which either output 64 bit addresses, or have some
>> way of informing vms to put the code into a certain 64 bit space and
>> starting address...
>
> Existing and unmodified 32-bit user apps—apps expecting P0/P1/S0/S1
> virtual address references—can't receive 64-bit descriptors and 64-bit
> P2 and S2 virtual addresses from other user APIs and from other
> third-party APIs and from system APIs.
>
> With data flowing across the APIs in the other direction, the designs of
> some existing user and third-party system APIs can't be transparently
> modified to accept 64-bit virtual addresses arriving from the calling apps.
>
> Which in aggregate means that these areas of user and third-party and
> OpenVMS executable code and data structures either have to sometimes
> "pretend" that the P2 and S2 virtual address ranges don't exist, or need
> parallel 32- and 64-bit APIs, etc.
>
> The traditional imperative APIs used within most of OpenVMS—descriptors,
> PQLs, itemlists, etc—are somewhat less flexible around these changes
> than would be message-passing APIs for instance, but message-passing API
> designs are not commonly encountered within OpenVMS.
>
> For the pedants reading and pointing to message-passing
> (object-oriented, OO) code on OpenVMS: my reference here is to OpenVMS
> system and RTL and device driver and related APIs. This outside of C++,
> Java, Python, and other such code, all off doing C++ or Java or Python
> or other message-passing tooling. BASIC mostly does well here too,
> though that compiler remains 32-bit. BASIC currently non-message-passing
> and probably should be updated to provide message-passing support, too.
> But I digress.
>
>
>

Still trying to disambiguate this in terms of what happens with 32 bit
code. You mention P0/P1/S0/S1 above, so are these segments and does
32 bit code reside in one of them ?.

As I said earlier, if you have a 32 bit app, but a 64 bit (or >32 bits),
address space, how does the system know where in memory to place that
code to run it ?. A reserved space that vms knows about, or what ?.

Vax split 4Gb into 2 segments, fwir, but looks like current vms is
doing something similar, so it can run 32 bit code...

Chris

On 2022-05-17, chris <chris-nospam@tridac.net> wrote:
>
> Still trying to disambiguate this in terms of what happens with 32 bit
> code. You mention P0/P1/S0/S1 above, so are these segments and does
> 32 bit code reside in one of them ?.
>
> As I said earlier, if you have a 32 bit app, but a 64 bit (or >32 bits),
> address space, how does the system know where in memory to place that
> code to run it ?. A reserved space that vms knows about, or what ?.
>

There is no such thing as a pure 64-bit program on VMS. Unfortunately.

On 64-bit VMS, all existing programs use 32-bit pointers and APIs with
32-bit pointers. If you are prepared to modify your source code, you can
also use 64-bit pointers within the same program and call APIs that also
use 64-bit pointers.

Also, not all parts of VMS have 64-bit APIs. Parts of RMS are one example.

This is utterly unlike anything you will be used to in Linux (for example),
where the size of the pointer is abstracted away from you as a C pointer
type. In VMS, due to the original assembly language nature of VMS, pointer
sizes are directly visible in the source code.

All VMS programs, including 32-bit ones, live in the same 64-bit address
space, and it is a feature of the program and the APIs it calls whether
it can access the full 64-bit address space or not.

On VMS, there is no such thing as simply recompiling your program with
-m32 or -m64 and then calling it job (mostly) done.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 2022-05-17 15:00:14 +0000, chris said:

>
> Still trying to disambiguate this in terms of what happens with 32 bit
> code. You mention P0/P1/S0/S1 above, so are these segments and does 32
> bit code reside in one of them ?.

P0/P1/S0/S1 are the address ranges available via 32-bit sign-extended pointers.

> As I said earlier, if you have a 32 bit app, but a 64 bit (or >32
> bits), address space, how does the system know where in memory to place
> that code to run it ?. A reserved space that vms knows about, or what ?.

Hardware uses 64-bit virtual addresses throughout. APIs and ABIs use a
mixture of 32- and 64-bit pointers. 32-bit pointers are sign-extended
to 64-bit.

> Vax split 4Gb into 2 segments, fwir, but looks like current vms is
> doing something similar, so it can run 32 bit code...

VAX virtual addressing was split into P0, P1, S0, and later, with the
arrival of VAX Extended Virtual Addressing support, S1.

VAX Extended Physical Addressing arrived around the same time, and
increased the available physical address range.

OpenVMS on Alpha and on Itanium has P0, P1, S0, and S1 within the
32-bit virtual address range, and user APIs and ABIs using 32-bit
pointers and 32-bit pointer structures.

OpenVMS Alpha V7.0 and later added P2 and S2 comprising the remainder
of 64-bit space when you remove 32-bit space, and that range was then
split in half for user and system access.

OpenVMS V7.0 and later updated some 32-bit APIs and ABIs, added some
64-bit APIs and APIs, and modified some user-accessible data structures
to support 64-bit pointers not the least of which are itemlists and
string descriptors. 32-bit string descriptors are eight bytes in size.
The 64-bit descriptors... are not.

As implemented on OpenVMS V7.0 and later and on OpenVMS I64 on Itanium,
the lowest 31 bits (P0, P1) and the highest 31 bits (S0, S1) are 32-bit
space, and the ginormous virtual address range between those two is P2
and S2.

It's the APIs and ABIs that make 64-bit more "interesting" on OpenVMS.
Most other platforms didn't try to mix pointer sizes within the same
app executables, and didn't need to have APIs and ABIs for each size,
and a bunch of switches and knobs to control it all.

For an intro, read the Guide to 64-bit addressing doc. The contents of
that doc were subsequently folded into other manuals in later versions,
and that doc was retired. But for the purposes of this discussion
however, that doc will introduce the hybrid addressing model used by
OpenVMS.

http://www0.mi.infn.it/~calcolo/OpenVMS/ssb71/6467/6467ptoc.htm

32-bit descriptor with 32-bit pointers (can reach all of P0, P1, S0,
and S1, but not P2 nor S2):

struct dsc$descriptor {
unsigned short dsc$w_length; /* specific to descriptor class;
typically a 16-bit (unsigned) length */
unsigned char dsc$b_dtype; /* data type code */
unsigned char dsc$b_class; /* descriptor class code */
char *dsc$a_pointer; /* address of first byte of data element */
};

64-bit descriptor with 64-bit pointers (can reach all of virtual
address space):

struct dsc64$descriptor {
unsigned short dsc64$w_mbo; /* must-be-one field overlays 16-bit
length field */
unsigned char dsc64$b_dtype; /* data type code */
unsigned char dsc64$b_class; /* descriptor class code */
long dsc64$l_mbmo; /* must-be-minus-one field overlays 32-bit pointer */
unsigned __int64 dsc64$q_length; /* quadword length */
char *dsc64$pq_pointer; /* address of first byte of data storage */
};

Some APIs and ABIs deal only with 32-bit descriptors, some with both.

Itemlists were similarly modified.

As were other user-referenced data structures, and APIs and ABIs.

--
Pure Personal Opinion | HoffmanLabs LLC

> Still trying to disambiguate this in terms of what happens with 32 bit
> code. You mention P0/P1/S0/S1 above, so are these segments and does
> 32 bit code reside in one of them ?.
>
> As I said earlier, if you have a 32 bit app, but a 64 bit (or >32 bits),
> address space, how does the system know where in memory to place that
> code to run it ?. A reserved space that vms knows about, or what ?.

There are no such thing as a 32 bit application on Alpha,
Itanium or x86-64 - applications are all 64 bit (in the
meaning of 64 bit application used on other platforms).

But that 64 bit application may use:
- all 32 bit pointers
- all 64 bit pointers
- mix of 32 bit and 64 bit pointers

32 bit pointers can only access some of the
virtual address space (P0 and P1
on VMS), but can be used with API's that
expect 32 bit pointers and API' that expect 64 bit
pointers.

64 bit pointers can access all virtual address
space (P0, P1 and P2 on VMS), but can only safely
be used with API's that expect 64 bit pointers.

I don't think where the code reside P0 or P2 is
important for accessing data - it may matter for
function pointers, self modifying code and various
other cases though.

Arne

On 2022-05-17 20:28:30 +0000, Arne Vajhj said:

> There are no such thing as a 32 bit application on Alpha, Itanium or
> x86-64 - applications are all 64 bit (in the meaning of 64 bit
> application used on other platforms).

Other than that the default compilation and default link on OpenVMS
Alpha and OpenVMS I64 produces apps using 32-bit sign-extended
pointers, and necessities using 32-bit ABIs and APIs, sure.

> ...
> I don't think where the code reside P0 or P2 is important for accessing
> data - it may matter for function pointers, self modifying code and
> various other cases though.

Other than that those sign-extended 32-bit pointers not being able to
access P2 data, and apps limited to 30 bits of code and data and
somewhat less than 30 bits of stack space, sure.

--
Pure Personal Opinion | HoffmanLabs LLC

On 5/17/2022 4:42 PM, Stephen Hoffman wrote:
> On 2022-05-17 20:28:30 +0000, Arne Vajhj said:
>> There are no such thing as a 32 bit application on Alpha, Itanium or
>> x86-64 - applications are all 64 bit (in the meaning of 64 bit
>> application used on other platforms).
>
> Other than that the default compilation and default link on OpenVMS
> Alpha and OpenVMS I64 produces apps using 32-bit sign-extended pointers,
> and necessities using 32-bit ABIs and APIs, sure.

It is not what is meant by 32 bit application on other
platforms.

>> I don't think where the code reside P0 or P2 is important for
>> accessing data - it may matter for function pointers, self modifying
>> code and various other cases though.
>
> Other than that those sign-extended 32-bit pointers not being able to
> access P2 data,

That does not relate to where the code is.

> and apps limited to 30 bits of code and data and
> somewhat less than 30 bits of stack space, sure.

I thought they had started moving code up to P2.

And data can be in P2 with some extra work.

Stack may actually be the biggest problem as I have
never heard about any intentions of introducing a
P3 for big stack. And a lot of stack space can
be needed in massive multi-threaded apps.
10000 threads with 1 MB stacks is 10 GB stack.
Sure that is a bit extreme, but it is not
crazy extreme just a bit extreme.

Arne

On 5/17/2022 1:26 PM, Simon Clubley wrote:
> On 2022-05-17, chris <chris-nospam@tridac.net> wrote:
>> Still trying to disambiguate this in terms of what happens with 32 bit
>> code. You mention P0/P1/S0/S1 above, so are these segments and does
>> 32 bit code reside in one of them ?.
>>
>> As I said earlier, if you have a 32 bit app, but a 64 bit (or >32 bits),
>> address space, how does the system know where in memory to place that
>> code to run it ?. A reserved space that vms knows about, or what ?.
>
> There is no such thing as a pure 64-bit program on VMS. Unfortunately.

With an unusual definition of "pure 64 bit" meaning all pointers being
64 bit.

> On 64-bit VMS, all existing programs use 32-bit pointers and APIs with
> 32-bit pointers. If you are prepared to modify your source code, you can
> also use 64-bit pointers within the same program and call APIs that also
> use 64-bit pointers.

You can switch to 64 bit pointers using a compile switch for
some languages (C, Fortran).

Whether source code changes are necessary for API reasons must
depend on the type of code. Just CRTL calls should be handled
automatically without any source code changes. With VMS calls
(SYS$, LIB$ etc.) changes will likely be needed.

> This is utterly unlike anything you will be used to in Linux (for example),
> where the size of the pointer is abstracted away from you as a C pointer
> type. In VMS, due to the original assembly language nature of VMS, pointer
> sizes are directly visible in the source code.

Some compatibility decisions was made 3 decades ago that has long
lasting impact.

That the usage of Macro-32 for applications was a major reason
is something you have claimed many times. But there are no
proofs that was more important than the other potential reasons,
so it is speculation.

> All VMS programs, including 32-bit ones, live in the same 64-bit address
> space, and it is a feature of the program and the APIs it calls whether
> it can access the full 64-bit address space or not.
>
> On VMS, there is no such thing as simply recompiling your program with
> -m32 or -m64 and then calling it job (mostly) done.

Given that there are are no 32 bit applications/CPU modes/processes
then there are obviously no m32 and m64.

There are /POINTER=64 and /POINTER=32 to change default pointer
size.

Which may or may not be sufficient to do a recompile and
start using P2 space for data.

Arne

On 2022-05-17 21:00:39 +0000, Arne Vajhj said:

> On 5/17/2022 4:42 PM, Stephen Hoffman wrote:
>> On 2022-05-17 20:28:30 +0000, Arne Vajhj said:
>>> There are no such thing as a 32 bit application on Alpha, Itanium or
>>> x86-64 - applications are all 64 bit (in the meaning of 64 bit
>>> application used on other platforms).
>>
>> Other than that the default compilation and default link on OpenVMS
>> Alpha and OpenVMS I64 produces apps using 32-bit sign-extended
>> pointers, and necessities using 32-bit ABIs and APIs, sure.
>
> It is not what is meant by 32 bit application on other platforms.

It's also not what is meant by a 64-bit application on other platforms.

>>> I don't think where the code reside P0 or P2 is important for accessing
>>> data - it may matter for function pointers, self modifying code and
>>> various other cases though.
>>
>> Other than that those sign-extended 32-bit pointers not being able to
>> access P2 data,
>
> That does not relate to where the code is.

Correct. It refers to both code and data.

>> and apps limited to 30 bits of code and data and somewhat less than 30
>> bits of stack space, sure.
>
> I thought they had started moving code up to P2.

Code which is inaccessible to 32-bit addresses, so don't try to pass a
function pointer to that code.

--
Pure Personal Opinion | HoffmanLabs LLC

On 5/17/22 4:11 PM, Arne Vajhøj wrote:

> You can switch to 64 bit pointers using a compile switch for
> some languages (C, Fortran).
>
> Whether source code changes are necessary for API reasons must
> depend on the type of code. Just CRTL calls should be handled
> automatically without any source code changes.

Unless your C program is using getopt, something from the exec family,
or anything else that deals with the argv and environ arrays, some (but
not all) of which can be mitigated by further specifying
/POINTER_SIZE=LONG=ARGV.

Or if you have various other CRTL routines that have no 64-bit variants.

Or you've got a LIB$INITIALIZE routine to set the DECC$ features at
start-up time; I may not be explaining this right but I think that
routine has to be referenced by a 32-bit pointer because that's where
the image activator will look for it.

On 5/17/2022 4:42 PM, Stephen Hoffman wrote:
> On 2022-05-17 20:28:30 +0000, Arne Vajhj said:
>
>> There are no such thing as a 32 bit application on Alpha, Itanium or x86-64 -
>> applications are all 64 bit (in the meaning of 64 bit application used on
>> other platforms).
>
> Other than that the default compilation and default link on OpenVMS Alpha and
> OpenVMS I64 produces apps using 32-bit sign-extended pointers, and necessities
> using 32-bit ABIs and APIs, sure.
>
>> ...
>> I don't think where the code reside P0 or P2 is important for accessing data -
>> it may matter for function pointers, self modifying code and various other
>> cases though.
>
> Other than that those sign-extended 32-bit pointers not being able to access P2
> data, and apps limited to 30 bits of code and data and somewhat less than 30
> bits of stack space, sure.

I'm just wondering how many VMS uses actually need the 64 bit addressing? At
least Basic, and perhaps other languages, were never modified to take advantage
of 64 bit addressing.

Sure, there can be some uses that have such a requirement. If it doesn't exist
(VAX) then VMS cannot be used in such cases.

But, for the most part, is this a serious issue, or, something to dispare over?

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

On 05/17/22 18:26, Simon Clubley wrote:
> On 2022-05-17, chris<chris-nospam@tridac.net> wrote:
>>
>> Still trying to disambiguate this in terms of what happens with 32 bit
>> code. You mention P0/P1/S0/S1 above, so are these segments and does
>> 32 bit code reside in one of them ?.
>>
>> As I said earlier, if you have a 32 bit app, but a 64 bit (or>32 bits),
>> address space, how does the system know where in memory to place that
>> code to run it ?. A reserved space that vms knows about, or what ?.
>>
>
> There is no such thing as a pure 64-bit program on VMS. Unfortunately.
>
> On 64-bit VMS, all existing programs use 32-bit pointers and APIs with
> 32-bit pointers. If you are prepared to modify your source code, you can
> also use 64-bit pointers within the same program and call APIs that also
> use 64-bit pointers.
>
> Also, not all parts of VMS have 64-bit APIs. Parts of RMS are one example.
>
> This is utterly unlike anything you will be used to in Linux (for example),
> where the size of the pointer is abstracted away from you as a C pointer
> type. In VMS, due to the original assembly language nature of VMS, pointer
> sizes are directly visible in the source code.
>
> All VMS programs, including 32-bit ones, live in the same 64-bit address
> space, and it is a feature of the program and the APIs it calls whether
> it can access the full 64-bit address space or not.
>
> On VMS, there is no such thing as simply recompiling your program with
> -m32 or -m64 and then calling it job (mostly) done.
>
> Simon.
>

Disappointing, but still doesn't amswer the question. Namely, if the
code is 32 bit, how does vms know where to put that in 64 bit space ?.

The question is relevant, since code has to be loaded at an address,
usually defined at link time, unless fully position independent code
is used, which can be quite inefficient. Perhaps the run time libraries
deal with the that, but not clear. Also, how do the P0, P1, S0, S1
segments relate to that ?...

Chris

On 5/17/2022 6:51 PM, Dave Froble wrote:
> I'm just wondering how many VMS uses actually need the 64 bit
> addressing?  At least Basic, and perhaps other languages, were never
> modified to take advantage of 64 bit addressing.
>
> Sure, there can be some uses that have such a requirement.  If it
> doesn't exist (VAX) then VMS cannot be used in such cases.
>
> But, for the most part, is this a serious issue, or, something to
> dispare over?

There is a reason why Alpha was made 64 bit and not 32 bit
those 3 decades ago.

There are definitely application types that need 64 bit
capability.

Databases are one case. And I believe that was the case
driving the Alpha decision all these years ago.

There are other memory hungry application types.

Some platform software: cache servers, Java EE application
servers, message queue servers etc..

Many custom scientific applications (Fortran, Python).

For custom business application (Cobol, Basic, Pascal,
Java or in some cases C) there must still be a lot of applications
that don't need it (as in "really don't need it" not just
"can live without it").

But application tend to grow over time so more applications
will want to use the full 64 bit space as times goes by.

And if/when VSI start courting ISV's not currently on VMS to
support VMS, then the vast majority will expect good 64 bit
support.

So I would call it a medium to high severity issue that
by will get a bit worse every year until fixed.

Not something VSI has to fix this year. But something VSI
should fix before 2030.

Arne

On 5/17/2022 7:26 PM, Arne Vajhøj wrote:
> On 5/17/2022 6:51 PM, Dave Froble wrote:
>> I'm just wondering how many VMS uses actually need the 64 bit addressing? At
>> least Basic, and perhaps other languages, were never modified to take
>> advantage of 64 bit addressing.
>>
>> Sure, there can be some uses that have such a requirement. If it doesn't
>> exist (VAX) then VMS cannot be used in such cases.
>>
>> But, for the most part, is this a serious issue, or, something to dispare over?
>
> There is a reason why Alpha was made 64 bit and not 32 bit
> those 3 decades ago.
>
> There are definitely application types that need 64 bit
> capability.

I just wonder what percentage of current VMS users need 64 bit addresses.

> Databases are one case. And I believe that was the case
> driving the Alpha decision all these years ago.
>
> There are other memory hungry application types.
>
> Some platform software: cache servers, Java EE application
> servers, message queue servers etc..
>
> Many custom scientific applications (Fortran, Python).
>
> For custom business application (Cobol, Basic, Pascal,
> Java or in some cases C) there must still be a lot of applications
> that don't need it (as in "really don't need it" not just
> "can live without it").
>
> But application tend to grow over time so more applications
> will want to use the full 64 bit space as times goes by.
>
> And if/when VSI start courting ISV's not currently on VMS to
> support VMS, then the vast majority will expect good 64 bit
> support.

Agreed, it's a "check box" item, better have it.

> So I would call it a medium to high severity issue that
> by will get a bit worse every year until fixed.
>
> Not something VSI has to fix this year. But something VSI
> should fix before 2030.
>
> Arne
>
>
>
>

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

On 5/17/22 23:31, Stephen Hoffman wrote:
> On 2022-05-17 21:00:39 +0000, Arne Vajhj said:
> ...
>> I thought they had started moving code up to P2.
>
> Code which is inaccessible to 32-bit addresses, so don't try to pass a
> function pointer to that code.

On x86 and IA64 function pointers are always 32 bit.

On x86, by default the linker puts code into P2; on request the linker
can move code into P0. On x86 the function pointer is a code address of
a single code instruction, which uses a 64 bit address to jump to the
"real" code. The linker creates this code instruction. On IA64, by
default the linker puts code into P0; on request, the linker can move
code into P2. On IA64 the function pointer is the address of a data
structure, a Function Descriptor (FD), which contains a 64 bit address
of the "real" code. The linker creates the FD.

On 5/18/22 00:05, Craig A. Berry wrote:
> ...
> Or you've got a LIB$INITIALIZE routine to set the DECC$ features at
> start-up time; I may not be explaining this right but I think that
> routine has to be referenced by a 32-bit pointer because that's where
> the image activator will look for it.

The image activator doesn't care about the pointer size. Only for
shareable images, the image activator calls the code in the
LIB$INITIALIZE module, which was (by request) linked into the shareable.
Its entry point was written by the linker and that's what the image
activator is looking for. The code in LIB$INITIALIZE expects an array of
32-bit pointers in the LIB$INITIALIZE PSECTs. They are function pointers
which are always 32 bit. This array is set up by the developer and
filled with the pointers. If the source module for setting this up was
compiled with 64 bit pointers, an array of pointers has 64 bit entries.
With the function pointers being 32 bits, the array from
LIB$INITIALIZE's point of view contains zero pointers. For the
LIB$INITIALIZE code a zero pointer is the end of the list. With 64-bit
pointers, that stops you from having more than one init routine called
"from the image activator". When the image activation is done, the main
image is started with SYS$IMGSTA, which calls the code in the
LIB$INITIALIZE module, ...

That also explains, why you can debug init code of a main image, but you
can't debug the init code of a shareable image (other than you run it as
a main image).

Some people may think SYS$IMGSTA (or even LIB$INITIALIZE) is part of the
image activator. From my point of view, it is not.

This all applies to the traditional VMS image initialization with the
LIB$INITIALIZE PSECTs. On x86 the ELF .init_array sections are
supported. They contain 64-bit pointers.

FWIW, the linker on/for x86 was enhanced to include the LIB$INITIALIZE
module from STARLET.OLB, whenever it encounters a LIB$INITIALIZE PSECT
or .init_array section.