I figured out why the C++ version of Perl 5 that I'm nearly ready to check into my experimental tree was escaping all of the VMS filespecs and adding "/" to the end.

Perl uses SYS$FILESCAN to try to parse filenames, and if it succeeds, and the result is the same length as the input, it knows it's a VMS filename. Otherwise, it assumes it's a UNIX-style name. That test was failing, and on further printf debugging, I found sys$filescan was returning no values for any of the fields queried.

I was curious if there was anything unusual about the pointers to the strings, and I discovered that the C++ compiler, but not the C compiler, is allocating strings in the area just above 0x80000000, which is outside the normal 32-bit address range due to sign extension. See the diagram for reference:

https://docs.vmssoftware.com/vsi-openvms-programming-concepts-manual-volume-i/#_32_VA_CHAPT

To reproduce the issue, I used the sys_filescan.c example from James F. Duff: https://www.eight-cubed.com/examples/framework.php?file=sys_filescan.c

It runs properly as is, with C and C++ compilers, but if you modify it slightly to initialize the DESCRIPTOR with a strdup() of the string constant, now it works when compiled with the C compiler, but crashes inside of sys_filescan when compiled with Clang, and indeed, a printf shows the pointer looks like 0000000080076020 with Clang, and a value in P0 space with VSI C.

Regards,
Jake Hamby

That seems like something that shouldn't be happening, but since my last issue was user error, maybe this one is as well?

Regards,
Jake Hamby

On 10/22/2023 6:32 AM, Jake Hamby (Solid State Jake) wrote:
> I figured out why the C++ version of Perl 5 that I'm nearly ready to
> check into my experimental tree was escaping all of the VMS filespecs
> and adding "/" to the end.
>
> Perl uses SYS$FILESCAN to try to parse filenames, and if it succeeds,
> and the result is the same length as the input, it knows it's a VMS
> filename. Otherwise, it assumes it's a UNIX-style name. That test was
> failing, and on further printf debugging, I found sys$filescan was
> returning no values for any of the fields queried.
>
> I was curious if there was anything unusual about the pointers to the
> strings, and I discovered that the C++ compiler, but not the C
> compiler, is allocating strings in the area just above 0x80000000,
> which is outside the normal 32-bit address range due to sign
> extension. See the diagram for reference:
>
> https://docs.vmssoftware.com/vsi-openvms-programming-concepts-manual-volume-i/#_32_VA_CHAPT
>
> To reproduce the issue, I used the sys_filescan.c example from James
> F. Duff:
> https://www.eight-cubed.com/examples/framework.php?file=sys_filescan.c
>
> It runs properly as is, with C and C++ compilers, but if you modify
> it slightly to initialize the DESCRIPTOR with a strdup() of the
> string constant, now it works when compiled with the C compiler, but
> crashes inside of sys_filescan when compiled with Clang, and indeed,
> a printf shows the pointer looks like 0000000080076020 with Clang,
> and a value in P0 space with VSI C.

If SYS$FILESCAN only accept 32 bit pointers then the data
need to be in P0 or P1 space. P2 space does not work.

The documentation for SYS$FILESCAN does not mention anything
about 64 bit - I guess that means it does not support
64 bit pointers.

Either use another function that supports 64 bit pointers
or copy to a temporary area in P0 space. You can use
_malloc32 to allocate there.

Arne

On Sunday, October 22, 2023 at 9:02:28 AM UTC-4, Arne Vajhøj wrote:
>
> The documentation for SYS$FILESCAN does not mention anything
> about 64 bit - I guess that means it does not support
> 64 bit pointers.
>
> Either use another function that supports 64 bit pointers
> or copy to a temporary area in P0 space. You can use
> _malloc32 to allocate there.
>
> Arne
One of the arguments to sys$filescan is a structure which gets filled in
by the service with the addresses of the name components within the
string you pass it. The address field in this structure is only 32-bits, so
it can't deal with P2 addresses.

I think the easiest way to fix it would be to add a flag bit to the fldflags
argument that makes the valuelst argument an alternate, 64-bit friendly,
structure. Valuelst is an array, so making the individual items variable
sized is awkward for the applications using $filescan.

On 2023-10-22 10:32:28 +0000, Jake Hamby (Solid State Jake) said:

> I was curious if there was anything unusual about the pointers to the
> strings, and I discovered that the C++ compiler, but not the C
> compiler, is allocating strings in the area just above 0x80000000,
> which is outside the normal 32-bit address range due to sign extension.
> See the diagram for reference...

I'd try switching the call to a 64-bit itemlist as a potential
workaround for this. But whether $filescan supports that, or just
32-bit shown in the doc?

Once again, not busting everything together—going to 64-bit, rather
than this current 32- and 64-bit API morass—seemingly busts everything
separately.

--
Pure Personal Opinion | HoffmanLabs LLC

On Sunday, October 22, 2023 at 12:32:30 PM UTC+2, Jake Hamby (Solid State Jake) wrote:
....
> It runs properly as is, with C and C++ compilers, but if you modify it slightly to initialize the DESCRIPTOR with a strdup() of the string constant, now it works when compiled with the C compiler, but crashes inside of sys_filescan when compiled with Clang, and indeed, a printf shows the pointer looks like 0000000080076020 with Clang, and a value in P0 space with VSI C.

You are probably aware that if you do a default CC the included string.h selects the 32-bit version of strdup() and if you do a default CXX the included string.h selects the 64-bit version of strdup(). (If you link with /MAP/CROSS you will see that different RTL routines are referenced.) So, strictly speaking, it is not the compiler itself, it is the 32-/64-bit mode it is in. You very likely can get the same crash if you compile CC/POINTER_SIZE=64 and you can avoid the crash with CXX/POINTER_SIZE=32 (or you can use _strdup32(), which is kind of documented, as its prototype can be found in string.h).

As far as I know, SYS$FILESCAN does not accept a 64-bit string descriptor.

On Sunday, October 22, 2023 at 12:33:14 PM UTC-7, h47...@gmail.com wrote:
> On Sunday, October 22, 2023 at 12:32:30 PM UTC+2, Jake Hamby (Solid State Jake) wrote:
> ...
> > It runs properly as is, with C and C++ compilers, but if you modify it slightly to initialize the DESCRIPTOR with a strdup() of the string constant, now it works when compiled with the C compiler, but crashes inside of sys_filescan when compiled with Clang, and indeed, a printf shows the pointer looks like 0000000080076020 with Clang, and a value in P0 space with VSI C..
> You are probably aware that if you do a default CC the included string.h selects the 32-bit version of strdup() and if you do a default CXX the included string.h selects the 64-bit version of strdup(). (If you link with /MAP/CROSS you will see that different RTL routines are referenced.) So, strictly speaking, it is not the compiler itself, it is the 32-/64-bit mode it is in. You very likely can get the same crash if you compile CC/POINTER_SIZE=64 and you can avoid the crash with CXX/POINTER_SIZE=32 (or you can use _strdup32(), which is kind of documented, as its prototype can be found in string.h).
>
> As far as I know, SYS$FILESCAN does not accept a 64-bit string descriptor..

It looks like SYS$FILESCAN doesn't accept 64-bit string descriptors or 64-bit item lists, so there's going to be a problem with the return values as well as the input parameter. I wonder how old that code is. Is it written in VAX MACRO, or perhaps BLISS?

Before I went to bed late last night, I thought that the simplest workaround would be to use _malloc32, as Arne suggested, so I added this to the end of my vms/vmsish.h header, which is included by perl.h:

/* Some system services aren't 64-bit pointer clean, like sys$filescan. */
#if __INITIAL_POINTER_SIZE || defined(__clang__)
#define malloc(size) _malloc32(size)
#define calloc(num, size) _calloc32(num, size)
#define realloc(ptr, size) _realloc32(ptr, size)
#endif

The docs say that __INITIAL_POINTER_SIZE is always defined (by default to 0) with VMS compilers, and this header is only included on VMS, so while it looks strange to me not to have an #ifdef __INITIAL_POINTER_SIZE test, I've copied the standard #ifdef check from the VMS header files themselves and if it gives a compiler error, then someone's using a new compiler and this code should probably be revisited.

Thankfully the VMS-specific code in Perl uses a wrapper macro that checks for failed system calls; using 32-bit malloc(), I've gotten a little bit further along, and now I have two new crashes with stack dumps to look at, along with the error codes. Notice the second one is complaining about an argument that isn't 32-bit sign-extended.

MCR Sys$Disk:[]miniperl.exe "-I[.lib]" make_ext.pl "MAKE=MMK" "--nonxs"
%SYSTEM-E-INVARG, invalid argument
%TRACE-E-TRACEBACK, symbolic stack dump follows
image module routine line rel PC abs PC
MINIPERL VMS safe_popen #4243 0000000080172057 0000000080172057
MINIPERL VMS do_spawn2 #11106 0000000080184FFA 0000000080184FFA
MINIPERL VMS Perl_do_aspawn #11031 0000000080184B87 0000000080184B87
MINIPERL PP_SYS Perl_pp_system #4658 000000008029AAC2 000000008029AAC2
MINIPERL RUN Perl_runops_standard #41 0000000000000028 00000000802B4398
MINIPERL PERLMINI S_run_body #2801 000000008011AEB2 000000008011AEB2
MINIPERL PERLMINI perl_run #2716 000000008011A9B8 000000008011A9B8
MINIPERL MINIPERLMAIN main #133 00000000000000DB 00000000800000DB
MINIPERL 0 0000000080000258 0000000080000258
MINIPERL 0 00000000802CDD64 00000000802CDD64
MINIPERL 0 00000000802CE4FD 00000000802CE4FD
PTHREAD$RTL 0 000000008004122C FFFF83000950922C
PTHREAD$RTL 0 0000000080002316 FFFF8300094CA316
0 FFFF8300081FC0A6 FFFF8300081FC0A6
DCL 0 000000008006778B 000000007ADEB78B
%TRACE-I-LINENUMBER, Leading '#' specifies a source file record number.
%TRACE-I-END, end of TRACE stack dump
%SYSTEM-F-ARG_GTR_32_BITS, argument is not 32-bit sign-extended value
%TRACE-F-TRACEBACK, symbolic stack dump follows
image module routine line rel PC abs PC
MINIPERL VMS Perl_cando_by_name_int #12004 0000000080180CF1 0000000080180CF1
MINIPERL VMS find_vmspipe #3800 0000000080194C87 0000000080194C87
MINIPERL VMS safe_popen #4252 0000000080172144 0000000080172144
MINIPERL VMS do_spawn2 #11106 0000000080184FFA 0000000080184FFA
MINIPERL VMS Perl_do_aspawn #11031 0000000080184B87 0000000080184B87
MINIPERL PP_SYS Perl_pp_system #4658 000000008029AAC2 000000008029AAC2
MINIPERL RUN Perl_runops_standard #41 0000000000000028 00000000802B4398
MINIPERL PERLMINI S_run_body #2801 000000008011AEB2 000000008011AEB2
MINIPERL PERLMINI perl_run #2716 000000008011A9B8 000000008011A9B8
MINIPERL MINIPERLMAIN main #133 00000000000000DB 00000000800000DB
MINIPERL 0 0000000080000258 0000000080000258
MINIPERL 0 00000000802CDD64 00000000802CDD64
MINIPERL 0 00000000802CE4FD 00000000802CE4FD
PTHREAD$RTL 0 000000008004122C FFFF83000950922C
PTHREAD$RTL 0 0000000080002316 FFFF8300094CA316
0 FFFF8300081FC0A6 FFFF8300081FC0A6
DCL 0 000000008006778B 000000007ADEB78B
%TRACE-I-LINENUMBER, Leading '#' specifies a source file record number.
%TRACE-I-END, end of TRACE stack dump
Fatal VMS error (status=0x00000FCA) at DKA0:[local.vms-perl5-ptr64-clang]vms.c;1
82, line 4243Fatal VMS error (status=0x000026BC) at DKA0:[local.vms-perl5-ptr64-
clang]vms.c;182, line 12004
%MMK-F-ERRUPD, error status %X100026BC occurred when updating target NONXSEXT

Regards,
Jake Hamby

On Sunday, October 22, 2023 at 12:52:42 PM UTC-7, Jake Hamby (Solid State Jake) wrote:
>
> Before I went to bed late last night, I thought that the simplest workaround would be to use _malloc32, as Arne suggested, so I added this to the end of my vms/vmsish.h header, which is included by perl.h:
>
> /* Some system services aren't 64-bit pointer clean, like sys$filescan. */
> #if __INITIAL_POINTER_SIZE || defined(__clang__)
> #define malloc(size) _malloc32(size)
> #define calloc(num, size) _calloc32(num, size)
> #define realloc(ptr, size) _realloc32(ptr, size)
> #endif
>
> The docs say that __INITIAL_POINTER_SIZE is always defined (by default to 0) with VMS compilers, and this header is only included on VMS, so while it looks strange to me not to have an #ifdef __INITIAL_POINTER_SIZE test, I've copied the standard #ifdef check from the VMS header files themselves and if it gives a compiler error, then someone's using a new compiler and this code should probably be revisited.
>
> Thankfully the VMS-specific code in Perl uses a wrapper macro that checks for failed system calls; using 32-bit malloc(), I've gotten a little bit further along, and now I have two new crashes with stack dumps to look at, along with the error codes. Notice the second one is complaining about an argument that isn't 32-bit sign-extended.

The first of the two crashes was fixed by wrapping the definition of struct exit_control_block with "#pragma __required_pointer_size __short" so that it matches what the OS expected.

The second crash is in sys$check_access(). I couldn't figure out what could possibly be going wrong on my end, since all the values I passed were 32-bit. Finally, I tried running the example at Eight-Cubed and it crashes too (with Clang, but not with VSI C).

$ cxx sys_check_access.c
DISK$X86DATA:[home.jhamby.Projects]sys_check_access.c;1:51:33: warning: conversion from string literal to 'char * __ptr32' is deprecated [-Wc++11-compat-deprecated-writable-strings]
static $DESCRIPTOR (filename_d, "SYS$SYSDEVICE:[000000]000000.DIR");
^
1 warning generated.
$ link sys_check_access
$ run sys_check_access
%SYSTEM-F-ARG_GTR_32_BITS, argument is not 32-bit sign-extended value
%TRACE-F-TRACEBACK, symbolic stack dump follows
image module routine line rel PC abs PC
Error: traceback pc= 0000022c was not found
sys_check_access SYS_CHECK_ACCESS 0 000000000000022C 000000008000022C
sys_check_access 0 0000000080000293 0000000080000293
PTHREAD$RTL 0 000000008004122C FFFF83000950922C
PTHREAD$RTL 0 0000000080002316 FFFF8300094CA316
0 FFFF8300081FC0A6 FFFF8300081FC0A6
DCL 0 000000008006778B 000000007ADEB78B
%TRACE-I-END, end of TRACE stack dump
$ cc sys_check_access.c
$ link sys_check_access
$ run sys_check_access
No access to SYSUAF - try running from a privileged account

Finally, I've found an actual OS bug, I think! The previous times have been my own misunderstanding or the system service not supporting 64-bit descriptors and item lists, but here I'm actually seeing a crash with a well-known sample program. BTW, it crashes on clang with "/pointer_size=32" as well.

Regards,
Jake Hamby

On 10/22/23 2:52 PM, Jake Hamby (Solid State Jake) wrote:

> Before I went to bed late last night, I thought that the simplest
> workaround would be to use _malloc32, as Arne suggested, so I added this
> to the end of my vms/vmsish.h header, which is included by perl.h:
>
> /* Some system services aren't 64-bit pointer clean, like
> sys$filescan. */ > #if __INITIAL_POINTER_SIZE || defined(__clang__)
> #define malloc(size) _malloc32(size)
> #define calloc(num, size) _calloc32(num, size)
> #define realloc(ptr, size) _realloc32(ptr, size)
> #endif

That's almost certainly much too big a hammer. That's going to make all
allocations within the Perl interpreter 32-bit, which likely won't play
nice with the 64-bit pointers you've requested.

On 10/22/2023 3:52 PM, Jake Hamby (Solid State Jake) wrote:
> so I added this to the end of my vms/vmsish.h header, which is included by perl.h:
>
> /* Some system services aren't 64-bit pointer clean, like sys$filescan. */
> #if __INITIAL_POINTER_SIZE || defined(__clang__)
> #define malloc(size) _malloc32(size)
> #define calloc(num, size) _calloc32(num, size)
> #define realloc(ptr, size) _realloc32(ptr, size)
> #endif

I am a little curious. What is the goal of this exercise?

If the goal is to make Perl work with 64 bit pointers
and keep all data in P0 space, then it will probably help.

If the goal is allow Perl to work on huge data in P2 space,
then doing all dynamic allocations in P0 space is not the
right solution. Instead you should generally allocated in
P2 space and only allocate in P0 space for those few
data items that has to be used by VMS SYS$ and LIB$
functions only working with such. That will cause some
copying between P0 and P2 space, but that is how the cards
was dealt.

Arne

On Monday, October 23, 2023 at 12:32:33 AM UTC+2, Jake Hamby (Solid State Jake) wrote:
> On Sunday, October 22, 2023 at 12:52:42 PM UTC-7, Jake Hamby (Solid State Jake) wrote:
....
> The second crash is in sys$check_access(). I couldn't figure out what could possibly be going wrong on my end, since all the values I passed were 32-bit. Finally, I tried running the example at Eight-Cubed and it crashes too (with Clang, but not with VSI C).
>
> $ cxx sys_check_access.c
> DISK$X86DATA:[home.jhamby.Projects]sys_check_access.c;1:51:33: warning: conversion from string literal to 'char * __ptr32' is deprecated [-Wc++11-compat-deprecated-writable-strings]
> static $DESCRIPTOR (filename_d, "SYS$SYSDEVICE:[000000]000000.DIR");
> ^
> 1 warning generated.
> $ link sys_check_access
> $ run sys_check_access
> %SYSTEM-F-ARG_GTR_32_BITS, argument is not 32-bit sign-extended value
> %TRACE-F-TRACEBACK, symbolic stack dump follows
....
> Finally, I've found an actual OS bug, I think! The previous times have been my own misunderstanding or the system service not supporting 64-bit descriptors and item lists, but here I'm actually seeing a crash with a well-known sample program. BTW, it crashes on clang with "/pointer_size=32" as well.

Looks like a CXX compiler bug, not an OS bug. In the debugger, if you step through the code you will notice that the arguments for sys$check_access() aren't right. (OK, you need to know the VMS/x86 calling standard, the x86_86 instructions and ...)

On 2023-10-22, Arne Vajhøj <arne@vajhoej.dk> wrote:
>
> If SYS$FILESCAN only accept 32 bit pointers then the data
> need to be in P0 or P1 space. P2 space does not work.
>
> The documentation for SYS$FILESCAN does not mention anything
> about 64 bit - I guess that means it does not support
> 64 bit pointers.
>
> Either use another function that supports 64 bit pointers
> or copy to a temporary area in P0 space. You can use
> _malloc32 to allocate there.
>

Even if it's a user error on the part of Jake, it is still really
a documentation error in the VMS documentation.

The documentation should not talk about 64-bit pointers as if they
are something special. The assumption should be that 64-bit pointers
are valid, and if a specific system service doesn't support them, the
following should be written in the documentation for the system service:

NOTE: this system service supports 32-bit addresses only.

The same comments apply to the RMS documentation as well.

This concept of parts of the user's address space being unavailable
to normal system services is utterly alien to people coming from other
operating systems (and does not paint VMS in a good light to put it
mildly).

As such, the normal expectations elsewhere should be the expectations in
the VMS documentation, and the documentation should draw _explicit_
attention to the cases when that is not the situation.

BTW, the _only_ time I can recall seeing address space addressing being
an issue on Linux is with some hardware devices, and hence the use of
bounce buffers in the device driver to isolate that issue from user space
and the rest of the kernel.

I have _never_ seen the issues discussed here in any Linux code I write.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 10/23/2023 8:58 AM, Simon Clubley wrote:
> On 2023-10-22, Arne Vajhøj <arne@vajhoej.dk> wrote:
>> If SYS$FILESCAN only accept 32 bit pointers then the data
>> need to be in P0 or P1 space. P2 space does not work.
>>
>> The documentation for SYS$FILESCAN does not mention anything
>> about 64 bit - I guess that means it does not support
>> 64 bit pointers.
>>
>> Either use another function that supports 64 bit pointers
>> or copy to a temporary area in P0 space. You can use
>> _malloc32 to allocate there.
>>
>
> Even if it's a user error on the part of Jake, it is still really
> a documentation error in the VMS documentation.
>
> The documentation should not talk about 64-bit pointers as if they
> are something special. The assumption should be that 64-bit pointers
> are valid, and if a specific system service doesn't support them, the
> following should be written in the documentation for the system service:
>
> NOTE: this system service supports 32-bit addresses only.
>
> The same comments apply to the RMS documentation as well.
>
> This concept of parts of the user's address space being unavailable
> to normal system services is utterly alien to people coming from other
> operating systems (and does not paint VMS in a good light to put it
> mildly).
>
> As such, the normal expectations elsewhere should be the expectations in
> the VMS documentation, and the documentation should draw _explicit_
> attention to the cases when that is not the situation.

In this specific case we really need SYS$FILESCAN64 to avoid a
temporary copy.

There is usually always room for improvements of documentation.

As David pointed out the documentation does show the "component
address" being written in a field with bits 0..31.

If VSI were to change documentation broadly then I would
suggest dropping the default assumption and always make a
note: a label "32 bit addresses only" / "64 bit addresses only" /
"Both 32 and 64 bit addresses" / "It is complicated - read
description for each argument carefully". Or something along those
lines.

Yes - the design is a bit tricky. DEC made some decisions
3 decades ago that made VAX -> Alpha migration easier. Today
we pay the price.

Arne

On Monday, October 23, 2023 at 2:02:39 AM UTC-7, h47...@gmail.com wrote:
> On Monday, October 23, 2023 at 12:32:33 AM UTC+2, Jake Hamby (Solid State Jake) wrote:
> > Finally, I've found an actual OS bug, I think! The previous times have been my own misunderstanding or the system service not supporting 64-bit descriptors and item lists, but here I'm actually seeing a crash with a well-known sample program. BTW, it crashes on clang with "/pointer_size=32" as well.
> Looks like a CXX compiler bug, not an OS bug. In the debugger, if you step through the code you will notice that the arguments for sys$check_access() aren't right. (OK, you need to know the VMS/x86 calling standard, the x86_86 instructions and ...)

I should have guessed the problem was in the compiler, not the OS.

Another bug I discovered in testing with Clang is there's no way for me to get the VMS behavior of exit() and _exit(). Instead, Clang/CXX always returns the status after POSIX conversion, even though _POSIX_EXIT isn't defined.. I know the CXX release notes say /MAIN isn't supported, but I didn't expect the behavior to default to UNIX-style. It looks like I'll have to call SYS$EXIT if I need to return a VMS status value, as Perl wants to be able to do.

Regards,
Jake

On Sunday, October 22, 2023 at 4:05:18 PM UTC-7, Arne Vajhøj wrote:
> On 10/22/2023 3:52 PM, Jake Hamby (Solid State Jake) wrote:
> I am a little curious. What is the goal of this exercise?
>
> If the goal is to make Perl work with 64 bit pointers
> and keep all data in P0 space, then it will probably help.
>
> If the goal is allow Perl to work on huge data in P2 space,
> then doing all dynamic allocations in P0 space is not the
> right solution. Instead you should generally allocated in
> P2 space and only allocate in P0 space for those few
> data items that has to be used by VMS SYS$ and LIB$
> functions only working with such. That will cause some
> copying between P0 and P2 space, but that is how the cards
> was dealt.

That's a great question! I should retrace my steps to explain how I got here. My original goals were to fix unit test failures in Perl, to use the latest features of the OS where available, and to compile Perl using Clang to take advantage of hopefully improved performance and C++'s stronger type checking.

I knew there was some support for building Perl with CXX on Alpha/Itanium, but not how up-to-date it is. I also know that switching to Clang would expose a lot of bugs in the new toolchain, which indeed it's been doing.

Your analysis of the memory spaces is correct. There's no practical reason why Perl needs to be compiled with 64-bit pointers. I can't think of a use case where a Perl script would ever need that much address space (if it did, you should almost certainly be using a faster programming language).

The original reason I went down the 64-bit pointer route was because I knew that I'd have to use va_list / va_copy on x86, and when I tried building with Clang, it defaults to 64-bit pointers and LP64 types, so I wanted to see if the new ABI is more portable than the old Alpha/IA64 "64-bit-pointers with 32-bit size_t and long" ABI, which it appears to be. More importantly, when I tried using 32-bit pointers with Clang, with NEED_VA_COPY defined in config.h, I got a build error about mismatched pointer types that I couldn't work around (when attempting to pass a va_list * to another function for processing).

My current status is working to bundle up my tiny (1-3 line) bug fixes to submit as individual pull requests. I'm pleasantly surprised to see that the Perl GitHub repo imported changelogs going back 25+ years, so when you look at the "blame" view of e.g. the VMS files, there are patches from last week mixed with code last updated 23-27 years ago. When I submitted my first PR, with a few cosmetic fixes, to get practice with the process, one of the check-in workflows complained that I should add my name and email to a ".mailmap" file, and it found my current email, presumably from my git config, but also an email address I haven't used in 25 years for an ISP that no longer exists. Amazing.

I think the quickest path to being able to build Perl on VMS x86-64 using Clang would be to use 32-bit pointers, get va_copy/va_list fully working in that mode, and fix any other compiler-specific gotchas, of which I've run into two: the sys$check_access() crash, and exit() / _exit() always have POSIX behavior (conversion of the exit status into VMS form), even without /main=posix_exit and without _POSIX_EXIT being defined.

Regards,
Jake

On Monday, October 23, 2023 at 11:01:03 AM UTC-7, Jake Hamby (Solid State Jake) wrote:
>
> I think the quickest path to being able to build Perl on VMS x86-64 using Clang would be to use 32-bit pointers, get va_copy/va_list fully working in that mode, and fix any other compiler-specific gotchas, of which I've run into two: the sys$check_access() crash, and exit() / _exit() always have POSIX behavior (conversion of the exit status into VMS form), even without /main=posix_exit and without _POSIX_EXIT being defined.

Here's the error message I can't get past when I try to build Perl using Clang and /Pointer=32:

CXX/NOANSI_ALIAS/NoExceptions/Pointer=32/Opt=(Lev=3)/l_double=64 /Include=[]/Standard=ANSI/Prefix=All/Obj=OP.obj/warn=(disable="c++11-compat-deprecated-writable-strings")/NoList/float=ieee/ieee=denorm/NAMES=(SHORTENED)/Define=(PERL_CORE,_USE_STD_STAT=1) OP.C
op.c:1649:1: warning: non-void function does not return a value in all control paths [-Wreturn-type]
} ^
op.c:1749:1: warning: non-void function does not return a value in all control paths [-Wreturn-type]
} ^
op.c:8117:23: error: first argument to 'va_arg' is of type '__sptr __ptr32 va_list' (aka '__sptr __ptr32 __builtin_va_list') and not 'va_list'
imop = va_arg(*args, OP*);
^~~~~
SYS$COMMON:[SYSLIB]DECC$RTLDEF.TLB;1(STDARG):127:54: note: expanded from macro 'va_arg'
# define va_arg(ap, type) __builtin_va_arg(ap, type)
^~
op.c:8122:21: error: first argument to 'va_arg' is of type '__sptr __ptr32 va_list' (aka '__sptr __ptr32 __builtin_va_list') and not 'va_list'
sv = va_arg(*args, SV*);
^~~~~
SYS$COMMON:[SYSLIB]DECC$RTLDEF.TLB;1(STDARG):127:54: note: expanded from macro 'va_arg'
# define va_arg(ap, type) __builtin_va_arg(ap, type)
^~
op.c:8125:25: error: first argument to 'va_arg' is of type '__sptr __ptr32 va_list' (aka '__sptr __ptr32 __builtin_va_list') and not 'va_list'
sv = va_arg(*args, SV*);
^~~~~
SYS$COMMON:[SYSLIB]DECC$RTLDEF.TLB;1(STDARG):127:54: note: expanded from macro 'va_arg'
# define va_arg(ap, type) __builtin_va_arg(ap, type)
^~
2 warnings and 3 errors generated.
%MMK-F-ERRUPD, error status %X15F61262 occurred when updating target OP.OBJ

> I think the quickest path to being able to build Perl on VMS x86-64 using Clang would be to use 32-bit pointers, get va_copy/va_list fully working in that mode, and fix any other compiler-specific gotchas, of which I've run into two: the sys$check_access() crash, and exit() / _exit() always have POSIX behavior (conversion of the exit status into VMS form), even without /main=posix_exit and without _POSIX_EXIT being defined.
>
Hello,
If it is possible to provide reproducers for the issues (for example sys_check_access.c, OP.C), that would be great.
I could add bugs to our bug list and have a look.

Thanks,
Armine

On Wednesday, October 25, 2023 at 4:52:33 AM UTC-7, Armine Arabyan wrote:
> > I think the quickest path to being able to build Perl on VMS x86-64 using Clang would be to use 32-bit pointers, get va_copy/va_list fully working in that mode, and fix any other compiler-specific gotchas, of which I've run into two: the sys$check_access() crash, and exit() / _exit() always have POSIX behavior (conversion of the exit status into VMS form), even without /main=posix_exit and without _POSIX_EXIT being defined.
> >
> Hello,
> If it is possible to provide reproducers for the issues (for example sys_check_access.c, OP.C), that would be great.
> I could add bugs to our bug list and have a look.
>
> Thanks,
> Armine

Sure. The sys$check_access crash can be repro'd with the example code at:
https://www.eight-cubed.com/examples/framework.php?file=sys_check_access.c

I can't come up with a shorter repro case, since 90% of the example is boilerplate to set up the data structures. Other than the short "errchk.h" header file that defines a few macros for error checking used by all the examples on that site, it's self-contained. Here's what I see on V9.2-1. Note that the crash when compiled with Clang happens before the privilege check has a chance to fail.

$ cc/version
VSI C X7.4-785 (GEM 50X65) on OpenVMS x86_64 V9.2-1
$ cxx/version
clang version 10.0.1 (git@bitbucket.org:vms_software/llvm-10.0.1.git b2ed69f05fb4239b21b6ff2f806caaf4691c9719)
Build date: 09-19-2023
Target: x86_64-OpenVMS
Thread model: posix
InstalledDir: VMSVM1$DKA0:[SYS0.SYSCOMMON.][SYSEXE]

$ cc sys_check_access
$ link sys_check_access
$ run sys_check_access
No access to SYSUAF - try running from a privileged account

$ cxx sys_check_access.c
DISK$X86DATA:[home.jhamby.Projects]sys_check_access.c;4:51:33: warning: conversion from string literal to 'char * __ptr32' is deprecated [-Wc++11-compat-deprecated-writable-strings]
static $DESCRIPTOR (filename_d, "SYS$SYSDEVICE:[000000]000000.DIR");
^
1 warning generated.
$ link sys_check_access
$ run sys_check_access
%SYSTEM-F-ARG_GTR_32_BITS, argument is not 32-bit sign-extended value
%TRACE-F-TRACEBACK, symbolic stack dump follows
image module routine line rel PC abs PC
Error: traceback pc= 00000220 was not found
sys_check_access SYS_CHECK_ACCESS 0 0000000000000220 0000000080000220
sys_check_access 0 0000000080000283 0000000080000283
PTHREAD$RTL 0 000000008004122C FFFF83000950922C
PTHREAD$RTL 0 0000000080002316 FFFF8300094CA316
0 FFFF8300081FC0A6 FFFF8300081FC0A6
DCL 0 000000008006778B 000000007ADEB78B
%TRACE-I-END, end of TRACE stack dump

The different behavior in _exit() can be shown with a much shorter test program:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main() {
printf("EXIT_SUCCESS is %d\n", EXIT_SUCCESS);
printf("EXIT_FAILURE is %d\n", EXIT_FAILURE);
#if defined(_POSIX_EXIT)
printf("_POSIX_EXIT is defined\n");
#else
printf("_POSIX_EXIT is not defined\n");
#endif
_exit(123);
}

$ cc test-exit.c
$ link test-exit
$ run test-exit
EXIT_SUCCESS is 0
EXIT_FAILURE is 268435458
_POSIX_EXIT is not defined
$ write sys$output $status
%X0000007B

$ cxx test-exit.c
$ link test-exit
$ run test-exit
EXIT_SUCCESS is 0
EXIT_FAILURE is 268435458
_POSIX_EXIT is not defined
$ write sys$output $status
%X0035A3D9

I just checked to see if the same thing happens with exit() or with returning the exit status from main, and I see that both of those methods behave the same with CC and CXX. It's only _exit() that has the bug of using the POSIX behavior when _POSIX_EXIT is not defined.

The last issue I noted that you may or may not have an open bug for is optarg not linking with CXX in the default 64-bit pointer mode. I can repro this with a do-nothing test program that just accesses optarg:

#include <stdio.h>
#include <stdlib.h>

int main()
{ printf("optarg is %p\n", optarg);
}

$ cc test-getopt.c
$ link test-getopt
$ run test-getopt
optarg is 0
$ cxx test-getopt.c
$ link test-getopt
%ILINK-W-NUDFSYMS, 1 undefined symbol:
%ILINK-I-UDFSYM, decc$ga__optarg64
%ILINK-W-USEUNDEF, undefined symbol decc$ga__optarg64 referenced
section: .text
offset: %X0000000000000012
module: TEST-GETOPT
file: DKA0:[local]test-getopt.OBJ;2
$ cxx/pointer=32 test-getopt.c
$ link test-getopt

As another user noted, this seems to be an issue with the header file not telling Clang that the symbol is there, but uppercase.

Thanks,
Jake Hamby

On Wednesday, October 25, 2023 at 10:15:27 PM UTC+2, Jake Hamby (Solid State Jake) wrote:
....
> The last issue I noted that you may or may not have an open bug for is optarg not linking with CXX in the default 64-bit pointer mode. I can repro this with a do-nothing test program that just accesses optarg:
>
> #include <stdio.h>
> #include <stdlib.h>
>
> int main()
> {
> printf("optarg is %p\n", optarg);
> }
>
> $ cc test-getopt.c
> $ link test-getopt
> $ run test-getopt
> optarg is 0
> $ cxx test-getopt.c
> $ link test-getopt
> %ILINK-W-NUDFSYMS, 1 undefined symbol:
> %ILINK-I-UDFSYM, decc$ga__optarg64
> %ILINK-W-USEUNDEF, undefined symbol decc$ga__optarg64 referenced
> section: .text
> offset: %X0000000000000012
> module: TEST-GETOPT
> file: DKA0:[local]test-getopt.OBJ;2
> $ cxx/pointer=32 test-getopt.c
> $ link test-getopt
>
> As another user noted, this seems to be an issue with the header file not telling Clang that the symbol is there, but uppercase.

It's a bug in the C RTL. As you can find with your favorite tool for displaying the exported, aka universal, symbols in sys$share:decc$shr.exe you will find that there are these optarg related symbols:
DECC$GA_OPTARG, decc$ga_optarg, and DECC$GA__OPTARG64. In other words, decc$ga__optarg64 is missing.

If it were a upcasing-header problem, you could use '/define=("decc$ga_optarg64=DECC$GA__OPTARG64")'. If you try it, you will see it doesn't change anything.

On Wednesday, October 25, 2023 at 1:15:27 PM UTC-7, Jake Hamby (Solid State Jake) wrote:
> On Wednesday, October 25, 2023 at 4:52:33 AM UTC-7, Armine Arabyan wrote:
> > Hello,
> > If it is possible to provide reproducers for the issues (for example sys_check_access.c, OP.C), that would be great.
> > I could add bugs to our bug list and have a look.
> >
> > Thanks,
> > Armine

Oops, there's one more bug that I ran into that I didn't give a test case for, which is va_list being broken with CXX and 32-bit pointers:

#include <stdio.h>
#include <stdarg.h>

static void test_va_arg_ptr(va_list *args) {
int i;
for (i = 0; i < 3; i++)
printf("args[i]: %d\n", va_arg(*args, int));
}

static void test_va_arg(int arg1, ...) {
printf("arg1 is %d\n", arg1);
va_list args;
va_start(args, arg1);
test_va_arg_ptr(&args);
va_end(args);
}

int main() {
test_va_arg(12, 34, 56, 78);
}

$ cc va_arg_test.c
$ link va_arg_test
$ run va_arg_test
arg1 is 12
args[i]: 34
args[i]: 56
args[i]: 78
$ cxx va_arg_test.c
$ link va_arg_test
$ run va_arg_test
arg1 is 12
args[i]: 34
args[i]: 56
args[i]: 78
$ cxx/pointer=32 va_arg_test.c
DKA0:[local]va_arg_test.c;3:7:40: error: first argument to 'va_arg' is of type '__sptr __ptr32 va_li
st' (aka '__sptr __ptr32 __builtin_va_list') and not 'va_list'
printf("args[i]: %d\n", va_arg(*args, int));
^~~~~
SYS$COMMON:[SYSLIB]DECC$RTLDEF.TLB;1(STDARG):127:54: note: expanded from macro 'va_arg'
# define va_arg(ap, type) __builtin_va_arg(ap, type)
^~
Regards,
Jake Hamby

On Wed, 2023-10-25 at 13:15 -0700, Jake Hamby (Solid State Jake) wrote:
> $ cc  test-exit.c
> $ link test-exit
> $ run test-exit
> EXIT_SUCCESS is 0
> EXIT_FAILURE is 268435458
> _POSIX_EXIT is not defined
> $ write sys$output $status
> %X0000007B
>
> $ cxx test-exit.c
> $ link test-exit
> $ run test-exit
> EXIT_SUCCESS is 0
> EXIT_FAILURE is 268435458
> _POSIX_EXIT is not defined
> $ write sys$output $status
> %X0035A3D9

I just got a chance to test this out on my x86 VMS installation. I get
different results, which is quite worrying.

Alex (KAKAPO) > dir

Directory DKA0:[USERS.ALEX.THINGS]

TEST_EXIT.C;1

Total of 1 file.
Alex (KAKAPO) > cc test_exit
Alex (KAKAPO) > link test_exit
Alex (KAKAPO) > run test_exit
EXIT_SUCCESS is 0
EXIT_FAILURE is 268435458
_POSIX_EXIT is not defined
%SYSTEM-F-DEVOFFLINE, device is not in configuration or not available
Alex (KAKAPO) > write sys$output $status
%X00000084
Alex (KAKAPO) > cxx test_exit.c
Alex (KAKAPO) > link test_exit
Alex (KAKAPO) > run test_exit
EXIT_SUCCESS is 0
EXIT_FAILURE is 268435458
_POSIX_EXIT is not defined
Alex (KAKAPO) > write sys$output $status
%X0035A421

Alex (KAKAPO) > cc/version
VSI C X7.4-785 (GEM 50X65) on OpenVMS x86_64 V9.2-1
Alex (KAKAPO) > cxx/version
clang version 10.0.1 (git@bitbucket.org:vms_software/llvm-10.0.1.git
b2ed69f05fb4239b21b6ff2f806caaf4691c9719)
Build date: 09-19-2023
Target: x86_64-OpenVMS
Thread model: posix
InstalledDir: KAKAPO$DKA0:[SYS0.SYSCOMMON.][SYSEXE]

--
Tactical Nuclear Kittens

On Saturday, October 28, 2023 at 9:01:32 PM UTC+2, Single Stage to Orbit wrote:
> On Wed, 2023-10-25 at 13:15 -0700, Jake Hamby (Solid State Jake) wrote:
> > $ cc test-exit.c
> > $ link test-exit
> > $ run test-exit
> > EXIT_SUCCESS is 0
> > EXIT_FAILURE is 268435458
> > _POSIX_EXIT is not defined
> > $ write sys$output $status
> > %X0000007B
> >
> > $ cxx test-exit.c
> > $ link test-exit
> > $ run test-exit
> > EXIT_SUCCESS is 0
> > EXIT_FAILURE is 268435458
> > _POSIX_EXIT is not defined
> > $ write sys$output $status
> > %X0035A3D9
> I just got a chance to test this out on my x86 VMS installation. I get
> different results, which is quite worrying.
>
> Alex (KAKAPO) > dir
>
> Directory DKA0:[USERS.ALEX.THINGS]
>
> TEST_EXIT.C;1
>
> Total of 1 file.
> Alex (KAKAPO) > cc test_exit
> Alex (KAKAPO) > link test_exit
> Alex (KAKAPO) > run test_exit
> EXIT_SUCCESS is 0
> EXIT_FAILURE is 268435458
> _POSIX_EXIT is not defined
> %SYSTEM-F-DEVOFFLINE, device is not in configuration or not available
> Alex (KAKAPO) > write sys$output $status
> %X00000084
> Alex (KAKAPO) > cxx test_exit.c
> Alex (KAKAPO) > link test_exit
> Alex (KAKAPO) > run test_exit
> EXIT_SUCCESS is 0
> EXIT_FAILURE is 268435458
> _POSIX_EXIT is not defined
> Alex (KAKAPO) > write sys$output $status
> %X0035A421
>
> Alex (KAKAPO) > cc/version
> VSI C X7.4-785 (GEM 50X65) on OpenVMS x86_64 V9.2-1
> Alex (KAKAPO) > cxx/version
> clang version 10.0.1 (g...@bitbucket.org:vms_software/llvm-10.0.1.git
> b2ed69f05fb4239b21b6ff2f806caaf4691c9719)
> Build date: 09-19-2023
> Target: x86_64-OpenVMS
> Thread model: posix
> InstalledDir: KAKAPO$DKA0:[SYS0.SYSCOMMON.][SYSEXE]
>
> --
> Tactical Nuclear Kittens

Can you show the output of "$ type test_exit.c"?
There is a chance that there is a typo - difference in the exit code:
$ set mess sys$message:DECC$MSG
$ write sys$output f$mess(%x35A421)
%C-S-EXIT132, exit value 132
$ x=132
$ sh symb x
X = 132 Hex = 00000084 Octal = 00000000204
$ write sys$output f$mess(132)
%SYSTEM-F-DEVOFFLINE, device is not in configuration or not available
$

On Sun, 2023-10-29 at 15:32 -0700, hb@end.of.inter.net wrote:
> Can you show the output of "$ type test_exit.c"?
> There is a chance that there is a typo - difference in the exit code:
> $ set mess sys$message:DECC$MSG
> $ write sys$output f$mess(%x35A421)
> %C-S-EXIT132, exit value 132
> $ x=132
> $ sh symb x
>   X = 132   Hex = 00000084  Octal = 00000000204
> $ write sys$output f$mess(132)
> %SYSTEM-F-DEVOFFLINE, device is not in configuration or not available
> $

Panic over. I mistyped the exit value as 132 instead of 123. Sorry
about that!
--
Tactical Nuclear Kittens