On 10/20/2023 5:51 AM, hb@end.of.inter.net wrote:
> On Friday, October 20, 2023 at 1:10:46 AM UTC+2, Arne Vajhøj wrote:
> ...
>> That is functional equivalent of what I am suggesting.
>
> and easy to implement. Just write a script to produce all the
> necessary source modules, compile them and tell the developer that at
> link time for feature A the object module NIX_A needs to be included
> and for feature B additionally the symbol NIX_B_VALUE needs to be
> defined. At least for the A type of features there can be an OPTIONS
> file.
>
>> But the LIB$INITIALIZE mechanism is a rather complex mechanism.
>>
>> Compared to:
>>
>> $ LINK ... + SYS$INPUT/OPT NIX_OPTIONS=(xxx,yyy,zzz) $
>>
>> or even simpler:
>>
>> $ LINK/NIX_OPTIONS=(xxx,yyy,zzz) ...
>
> You need to make that NIX_OPTIONS=(xxx,yyy=nnn,zzz) for a yyy
> features, which requires a value.

True.

> What do you want the linker to generate? Symbols the CRTL can look up
> at run time? Other than that would require CRTL changes, there are no
> symbols in main executable images. LIB$INITIALIZE code? That's
> probably possible but requires linker changes. The above approach
> seems simpler and works today.

I was thinking like a compatibility psect with values
in P0 space.

But if it generated lib$initialize code setting
logicals, then that would look the exact same from the
developer perspective.

I just want:
* application scope
* simple (simple enough that anyone writing a build
script can understand it)

Arne

On Friday, October 20, 2023 at 3:24:53 PM UTC+2, Arne Vajhøj wrote:

> I was thinking like a compatibility psect with values
> in P0 space.

I'm not sure I understand. What is a "compatibility psect"? Again, if the linker puts some values somewhere in the EXE, the C RTL code has to find them. In main images, neither PSECTs nor symbols are visible to the outside world. This approach would require a change in the linker, the image format and a change in the C RTL.

> But if it generated lib$initialize code setting
> logicals, then that would look the exact same from the
> developer perspective.

I'm note sure if you mean setting C RTL features or feature logicals. This requires "only" a change in the linker. The change has to be generic in the sense, that it should not depend on the features - that is, addding or removing a feature should not require a new linker.

> I just want:
> * application scope
> * simple (simple enough that anyone writing a build
> script can understand it)

So just do it! It's easy and possible today, took me half an hour to create a proof of concept.

FWIW, I spent some more time on this. The format of the source will very like not be preserved, but ...
Too long to read? Just ignore it.

Create a template, I used a .H file, but any other file name will do.

$ type CRTLINIT.H
#ifndef FEATURE
#error macro FEATURE undefined
// hack to abort compilation
#include <macro_FEATURE_undefined.h>
#endif
#define CONCAT(x,y) XCONCAT(x,y)
#define XCONCAT(x,y) x ## _ ## y
#define STRING(s) XSTRING(s)
#define XSTRING(s) #s

#include <unixlib.h>

// no longer needed on x86, but ...
extern void lib$initialize();
static void (*unused_function_pointer)(void) = lib$initialize ;

static void CONCAT(FEATURE,f)();
#pragma extern_model save
#pragma extern_model strict_refdef "LIB$INITIALIZE" gbl,noexe,nowrt,noshr,long
#if __INITIAL_POINTER_SIZE
# pragma __pointer_size __save
# pragma __pointer_size 32
#endif
extern void (* const CONCAT(FEATURE,a)[])() = {CONCAT(FEATURE,f)};
#if __INITIAL_POINTER_SIZE
# pragma __pointer_size __restore
#endif
#pragma extern_model restore
#ifdef VALUE
static void CONCAT(FEATURE,f) () {
decc$feature_set("DECC$" STRING(FEATURE), __FEATURE_MODE_CURVAL, VALUE);
#else
globalvalue int FEATURE;
static void CONCAT(FEATURE,f) () {
decc$feature_set("DECC$" STRING(FEATURE), __FEATURE_MODE_CURVAL, FEATURE);
#endif
} $

Compile it for all the C RTL features. Use your favorite tools. XHELP is HELP/NOPAGE/NOPROMPT,
GSED and GAWK are local copies of GNV's sed and awk.

$ pipe xhelp crtl feature_l |ggrep " \{12\}DECC\$" | -
gawk "{print ""$ cc/obj=""$1"" CRTLINIT.H/define=(feature=""$1,$2}" | -
gsed "s/\s\+DISABLE/,value=1)/" | gsed "s/\s\+ENABLE/,value=0)/" | -
gsed "s/\s\+\(RMS\|[0-9]\).*/)/" |gsed "s/DECC\$//g" |@sys$pipe

extern void (* const CONCAT(FEATURE,a)[])() = {CONCAT(FEATURE,f)};
......................^
%CC-W-LONGEXTERN, The external identifier name exceeds 31 characters; truncated to "DISABLE_TO_VMS_LOGNAME_TRANSLAT".
at line number 26 in file WORK18:[TMP.2023-10-20]CRTLINIT.H;1
$

Ugly, but you can ignore it or better, use cc:=cc/warning=disable=LONGEXTERN.

That' what you have to do only once, now an "application".

$ type CMAIN.C
#include <stdlib.h>
#include <stdio.h>
#include <time.h>
static struct tm time_struct;
int main (int argc, char *argv[]) {
strptime("01 January 90", "%d %B %y", &time_struct);
printf ("%s", asctime(&time_struct));
strptime("01 January 10", "%d %B %y", &time_struct);
printf ("%s", asctime(&time_struct));
for (int i=0; argv[i]; i++)
printf ("argv[%d]: '%s'\n", i, argv[i]);
return EXIT_SUCCESS;
} $ cc cmain
$ link cmain
$ set proces/parse=extended
$ mc []cmain a B c
Mon Jan 1 00:00:00 1990
Fri Jan 1 00:00:00 2010
argv[0]: '_dsa76:[tmp.2023-10-20]cmain.exe;1'
argv[1]: 'a'
argv[2]: 'b'
argv[3]: 'c'
$

Tell the developer to include the C RTL feature modules:

$ link cmain,ARGV_PARSE_STYLE
$ mc []cmain a B c
Mon Jan 1 00:00:00 1990
Fri Jan 1 00:00:00 2010
argv[0]: '_DSA76:[TMP.2023-10-20]CMAIN.EXE;2'
argv[1]: 'a'
argv[2]: 'B'
argv[3]: 'c'
$ $ link cmain,ARGV_PARSE_STYLE,XPG4_STRPTIME
$ mc []cmain a B c
Sun Jan 1 00:00:00 2090
Fri Jan 1 00:00:00 2010
argv[0]: '_DSA76:[TMP.2023-10-20]CMAIN.EXE;3'
argv[1]: 'a'
argv[2]: 'B'
argv[3]: 'c'
$

PIPE_BUFFER_SIZE isn't used, it's here only to demonstrate how to set a value
for a feature that requires a non-boolean value.

$ link cmain,ARGV_PARSE_STYLE,XPG4_STRPTIME,PIPE_BUFFER_SIZE
%ILINK-W-NUDFSYMS, 1 undefined symbol:
%ILINK-I-UDFSYM, PIPE_BUFFER_SIZE
%ILINK-W-USEUNDEF, undefined symbol PIPE_BUFFER_SIZE referenced
section: $CODE$
offset: %X0000000000000000 slot: 1
module: CRTLINIT
file: WORK18:[TMP.2023-10-20]PIPE_BUFFER_SIZE.OBJ;1
$ link cmain,ARGV_PARSE_STYLE,XPG4_STRPTIME,PIPE_BUFFER_SIZE,tt:/opt
symbol=PIPE_BUFFER_SIZE,1024
Exit
$

On 2023-10-18 13:48:32 +0000, Arne Vajhøj said:

> I am also very cautious about these logicals.
>
> I only add any when absolutely needed.

That approach works (sorta-kinda-mostly) because there isn't much code
being swapped around, and because lots of that is largely or entirely
self-contained only referencing itself and OpenVMS dependencies.

If source code and apps and dependencies start getting swapped around
more commonly, specifying the whole (current) list becomes "best
practice for the defensive developer", or some such corporate-speak.

The whole DECC$* design was bad, is bad, and will always remain bad, of course.

Same holds for most (all?) other designs using logical names for
configuration information.

The current translate-a-logical-name design was expedient, because
there wasn't (and isn't) an architected set of preferences stored for
an app.

The design—one intending to provide run-time compatibility—cannot be fixed.

One shorter-term alternative is to load the compatibility data into a
program section accessible to the linker and to some (more) SET
IMAGE-like commands, with anything not included being the original (and
usually bad) default.

The longer-term alternative would be to load the compatibility into a
program section, and force apps needing older (and backwards-focused)
defaults to explicitly select those. This to drift toward better
default behaviors.

Downside of loading this stuff directly into the executable is its lack
of extensibility; this only works for developer-provided settings.

Another downside of loading into the executable is different shareables
that need different default settings, too. That makes the image
activator and run-time handling that much more complex. Currently, the
run-time can YOLO with this, or the shareable can check and can exit.

This whole run-time compatibility area is adjacent to user- and
site-local configurations for applications, too. This is a mechanism
which just doesn't exist in any standard form on OpenVMS. Not past some
arcane X stuff, or the USER* parameters, or ilk.

The doc here is horrid, too—particularly given indirect effects with
some dependency getting side-swiped by a change to a shareable or to a
user- or site-local DECC$* or other logical name, and the results at
run-time then get... murky.

Wait, what, who among y'all thought I wouldn't unload on this DECC$*
design, and of using logical names for app-parsed non-device
configuration data, in this of all threads?

--
Pure Personal Opinion | HoffmanLabs LLC

On Wednesday, October 18, 2023 at 2:34:25 PM UTC-4, Arne Vajhøj wrote:
> I do not like the design due to logical having
> typical process scope where the need really is for
> application scope.
>
> An obvious question is: are there a better solution.

VSI could an additional ACE type that specifies a list of feature numbers
and values that the C init routine would scan and use to set the initial
features. The build script would add the ACE to the executable image's
ACL.

I suppose dlopen() could be modified as well to check the images it loads
for such ACE's as well and prevent incompatible combinations.

On Sunday, October 22, 2023 at 3:28:02 PM UTC-4, David Jones wrote:
> On Wednesday, October 18, 2023 at 2:34:25 PM UTC-4, Arne Vajhøj wrote:
> > I do not like the design due to logical having
> > typical process scope where the need really is for
> > application scope.
> >
> > An obvious question is: are there a better solution.
> VSI could an additional ACE type that specifies a list of feature numbers
> and values that the C init routine would scan and use to set the initial
> features. The build script would add the ACE to the executable image's
> ACL.
>
> I suppose dlopen() could be modified as well to check the images it loads
> for such ACE's as well and prevent incompatible combinations.
Actually, I talked to various folks about an ACE but the challenge is how to keep
the ACE with the image

On 10/20/2023 10:11 AM, hb@end.of.inter.net wrote:
> On Friday, October 20, 2023 at 3:24:53 PM UTC+2, Arne Vajhøj wrote:
>> I was thinking like a compatibility psect with values in P0 space.
>
> I'm not sure I understand. What is a "compatibility psect"?

Something like:

.psect nix_compatibility
feat1:: .long 0
feat2:: .long 1
feat3:: .long 777
....

> Again, if
> the linker puts some values somewhere in the EXE, the C RTL code has
> to find them.

Correct. Just like it today find those logicals.

> In main images, neither PSECTs nor symbols are visible
> to the outside world. This approach would require a change in the
> linker, the image format and a change in the C RTL.

Linker: yes. C RTL: yes. I am not sure that I understand the
need to change the image format - it is just a psect with global
symbols.

But.

David Letterman once said that, if you have to explain a
joke, then it is not funny.

I am starting to get the feeling that given how much I have
to explain my idea, then my idea may not be a good idea.

:-(

>> But if it generated lib$initialize code setting logicals, then that
>> would look the exact same from the developer perspective.
>
> I'm note sure if you mean setting C RTL features or feature logicals.
> This requires "only" a change in the linker. The change has to be
> generic in the sense, that it should not depend on the features -
> that is, addding or removing a feature should not require a new
> linker.
>
>> I just want:
>> * application scope
>> * simple (simple enough that anyone writing a build
>> script can understand it)
>
> So just do it! It's easy and possible today, took me half an hour to
> create a proof of concept.

LIB$INITIALIZE is not a feature I want VMS developers to
have to learn about.

But I will take a look at the code you posted.

Arne

On 2023-10-23 14:04:36 +0000, Arne Vajhøj said:

> On 10/20/2023 10:11 AM, hb@end.of.inter.net wrote:
>> On Friday, October 20, 2023 at 3:24:53 PM UTC+2, Arne Vajhøj wrote:
>>> I was thinking like a compatibility psect with values in P0 space.
>>
>> I'm not sure I understand. What is a "compatibility psect"?
>
> Something like:
>
> .psect nix_compatibility
> feat1:: .long 0
> feat2:: .long 1
> feat3:: .long 777
> ...

It's just like the rest of the image headers in VAX and Alpha terms,
and ELF terms too, with different sections for different executable
image metadata and data. Much like the image headers of old in other
ways too, with some cross-version compatibility expected upward and
downward.

For those unfamiliar with Ange Albertini's work:
https://github.com/corkami/pics/blob/master/binary/ELF101.png

This stuff would be an OpenVMS-specific p_type of PT_VMS_ABI_PREFS or
some such (or several such), and that probably assigned in the range
PT_LOOS to PT_HIOS, filled with a series of TLV type-length-value or
such with each entry. I would not use a fixed-size array here, because
somebody sooner or later is going to want or need something other than
a longword integer.

The image activator then loads that into P1 and provides an API for the
rest of the image activation and run-time processing to retrieve that
data. Calls requesting stuff not in the TLV gets a default answer (and
not an error!), much as happens with unimplemented methods on some
other platforms. There's an existing OpenVMS ABI that does this
generic-answer-for-compatibility stuff too IIRC, not that I can
immediately recall which ABI it is. This all ignoring that GSMATCH was
quietly nerfed a while back too, but that's fodder for another time.

>> Again, if the linker puts some values somewhere in the EXE, the C RTL
>> code has to find them.

Please stop thinking of this as solely a C and C++ issue. This is an
image activator and platform-compatibility limitation. Yes, C and C++
will be the current major consumers of this, but potential consumers
here are legion, not the least of which is the hypothetical future path
that gets rid of the 32-bit/64-bit support and goes native 64-bit APIs
throughout, among other existing limits. Other languages can want the
same things as C and C++ here, too. BASIC or some other language might
want to trash some older and problematic ABI, and this can provide an
incremental path to that.

Conceivably also useful for cross-platform features such as image
translation, and to OM if that or some analog to IBM AIX fdpr ever
makes a resurgence, and concievably caching previously-translated code
in a JIT environment, too. Pragmatically, none of those other uses will
happen this side of some new and breakout success of VSI with OpenVMS,
though.

One downside of embedding this data into the image—symbol tables or
headers or otherwise—is the increased difficulty of checksumming the
integrity of signed executable images, should that ever become
interesting.

In various ways, this is a much more flexible version of GSMATCH.

I don't see OpenVMS moving to app bundles or forks, but those are other
potential paths.

> Correct. Just like it today find those logicals.
>
>> In main images, neither PSECTs nor symbols are visible to the outside
>> world. This approach would require a change in the linker, the image
>> format and a change in the C RTL.
>
> Linker: yes. C RTL: yes. I am not sure that I understand the need to
> change the image format - it is just a psect with global symbols.

Probably best closer to image headers in classic OpenVMS usage, less to
global symbols.

> But.
>
> David Letterman once said that, if you have to explain a joke, then it
> is not funny.
>
> I am starting to get the feeling that given how much I have to explain
> my idea, then my idea may not be a good idea.

Your idea is fine. The implementation might shift slightly.

>
> :-(
>
>>> But if it generated lib$initialize code setting logicals, then that
>>> would look the exact same from the developer perspective.
>>
>> I'm note sure if you mean setting C RTL features or feature logicals.
>> This requires "only" a change in the linker. The change has to be
>> generic in the sense, that it should not depend on the features - that
>> is, addding or removing a feature should not require a new linker.
>>
>>> I just want:
>>> * application scope
>>> * simple (simple enough that anyone writing a build
>>> script can understand it)
>>
>> So just do it! It's easy and possible today, took me half an hour to
>> create a proof of concept.
>
> LIB$INITIALIZE is not a feature I want VMS developers to have to learn about.
>
> But I will take a look at the code you posted.

The use of the lib$initialize mechanism is a hack, probably because the
cost of architecting a solution that involves touching the linker and
image activator is presumably larger, and also with higher risk. Quick
and easy—using logical names as a management UI—was how we got here.

--
Pure Personal Opinion | HoffmanLabs LLC

On Monday, October 23, 2023 at 7:04:40 AM UTC-7, Arne Vajhøj wrote:
> On 10/20/2023 10:11 AM, h...@end.of.inter.net wrote:
> >
> > I'm note sure if you mean setting C RTL features or feature logicals.
> > This requires "only" a change in the linker. The change has to be
> > generic in the sense, that it should not depend on the features -
> > that is, addding or removing a feature should not require a new
> > linker.
> >
> >> I just want:
> >> * application scope
> >> * simple (simple enough that anyone writing a build
> >> script can understand it)
> >
> > So just do it! It's easy and possible today, took me half an hour to
> > create a proof of concept.
> LIB$INITIALIZE is not a feature I want VMS developers to
> have to learn about.
>
> But I will take a look at the code you posted.

The LIB$INITIALIZE hack isn't exactly pretty, but it's what GNV ports have been using for a while, and the Perl port as well. Adding a "vms_crtl_init.c" file or two to a project to set the decc$ features is how I've been handling the situation with my forks.

I only recently, from looking at the Perl port and docs, realized that VMS programs only look at the first character of boolean logicals, uppercased, to see if it's an "E", "T", or "1", and everything else is considered disabled. So at least VMS code that checks feature logicals, C RTL, Perl, or otherwise, are testing one byte and not doing a string compare, and usually the feature logicals get cached into lookup tables at startup.

Returning to my original concern of which DECC$ features are good/bad (regardless of how to activate them), I think DECC$STREAM_PIPE is relatively safe, and speeds up pipes, along with DECC$STDIO_CTX_EOL, which buffers fwrite() output until a terminator or buffer full is reached.

Experimenting with DECC$MAILBOX_CTX_STM led me to some more interesting successes and failures. "By default, an open on a local mailbox that is not a pipe treats mailbox records as having a record attribute of FAB$M_CR." That means that every line written is preceded by a newline and followed by a carriage return.

That explains perfectly the VMS quirk of moving the cursor to the beginning of the line before advancing it, instead of doing both at the same time, like UNIX and DOS/Windows.

There's a Perl test case related to temp files that spawns a perl process to run a subscript that returns a series of "ok" lines (or not, as the case may be). The test fails because the parent receives extra "\n" chars that it's not expecting ("test_pl/tempfile.t" tests 17-19). What's interesting is if I define DECC$MAILBOX_CTX_STM, then all 20 subtests pass, so the child is passing everything to the parent as expected, transparently.

Sadly, enabling that feature also causes the entire Perl test harness to fail for all tests, unable to find any subtests for any of the test scripts! So it seems like this feature could be very powerful to enable but only if it doesn't break handling of pipe data elsewhere, as it does for "perl harness".

Not coincidentally, a fair chunk of the 14,000 lines of code in vms/vms.c in Perl 5 are related to mailboxes and pipes and moving data around. There's one section in particular that copies from a mailbox to an fd that adds a "\n" after each fwrite(), which looks very suspicious to me, but removing that code seems to make no difference at all, at least relative to the extra "\n" that I'm seeing, or to any test cases passing or failing.

It wouldn't surprise me if it turned out to be faster for Perl to use the C RTL pipes exclusively, enabling locally DECC$STREAM_PIPE and perhaps DECC$MAILBOX_CTX_STM if it would help. I'm currently trying to determine if the Perl custom implementation of POSIX kill() that uses $SIGPRC behaves any differently than __posix_kill(). They appear to be about the same, but I'd prefer to use the C RTL if it works, especially for C++.

Regards,
Jake

On 10/23/2023 6:04 PM, Jake Hamby (Solid State Jake) wrote:
> The LIB$INITIALIZE hack isn't exactly pretty, but it's what GNV ports
> have been using for a while, and the Perl port as well.

Of course. It is the best option that exist today.

It is:

impact ugly exist
other
stuff

process yes yes yes
logical

LIB$INITIALIZE no yes yes

various "good" no no no
ideas on how to
put in in or on
the EXE

Arne

On 10/23/23 5:04 PM, Jake Hamby (Solid State Jake) wrote:

> Returning to my original concern of which DECC$ features are good/bad (regardless of how to activate them), I think DECC$STREAM_PIPE is relatively safe, and speeds up pipes, along with DECC$STDIO_CTX_EOL, which buffers fwrite() output until a terminator or buffer full is reached.
>
> Experimenting with DECC$MAILBOX_CTX_STM led me to some more interesting successes and failures. "By default, an open on a local mailbox that is not a pipe treats mailbox records as having a record attribute of FAB$M_CR." That means that every line written is preceded by a newline and followed by a carriage return.
>
> That explains perfectly the VMS quirk of moving the cursor to the beginning of the line before advancing it, instead of doing both at the same time, like UNIX and DOS/Windows.
>
> There's a Perl test case related to temp files that spawns a perl process to run a subscript that returns a series of "ok" lines (or not, as the case may be). The test fails because the parent receives extra "\n" chars that it's not expecting ("test_pl/tempfile.t" tests 17-19). What's interesting is if I define DECC$MAILBOX_CTX_STM, then all 20 subtests pass, so the child is passing everything to the parent as expected, transparently.
>
> Sadly, enabling that feature also causes the entire Perl test harness to fail for all tests, unable to find any subtests for any of the test scripts! So it seems like this feature could be very powerful to enable but only if it doesn't break handling of pipe data elsewhere, as it does for "perl harness".
>
> Not coincidentally, a fair chunk of the 14,000 lines of code in vms/vms.c in Perl 5 are related to mailboxes and pipes and moving data around. There's one section in particular that copies from a mailbox to an fd that adds a "\n" after each fwrite(), which looks very suspicious to me, but removing that code seems to make no difference at all, at least relative to the extra "\n" that I'm seeing, or to any test cases passing or failing.
>
> It wouldn't surprise me if it turned out to be faster for Perl to use the C RTL pipes exclusively, enabling locally DECC$STREAM_PIPE and perhaps DECC$MAILBOX_CTX_STM if it would help. I'm currently trying to determine if the Perl custom implementation of POSIX kill() that uses $SIGPRC behaves any differently than __posix_kill(). They appear to be about the same, but I'd prefer to use the C RTL if it works, especially for C++.
>

Perl pipes are not CRTL pipes. Defining DECC$MAILBOX_CTX_STM will have
no effect on pipes in Perl. It apparently affects *something else* or
you wouldn't have the whole test suite go belly up.

The history here is that pipes were so bad in the CRTL in the v6.x era
that Charles Lane contributed his own implementation to Perl based on
mailboxes. It wasn't just pipe() but also popen(), which used its own
hack (in DCL!) to hand off standard streams to child processes long
before decc$set_child_standard_streams existed. And closing down pipes
without hangs were also something this implementation did better than
what was available in the CRTL at the time. Whether any of that is
actually still true is a question worth asking. It's unlikely
mailbox-based pipes by any implementer would have significant
performance advantages over what we have now, but not maintaining all
that mailbox code could have some benefits.

Aside from being slow, the problem with mailboxes is that they are
record-oriented devices. There are various mechanisms to emulate
stream-oriented behavior, and as far as I understand them, they all
depend on buffering. But pipes are by nature unbuffered and if you
buffer them, the reader tends to hang at awkward moments.

There is an implementation called dmpipe based on global sections that's
going to perform a lot better than any mailbox-based pipe. I think we
may have conversed about it before and you appear to have your own fork
of it on github.

The reasons for Perl's own kill implementation are documented in the
configuration message it prints when checking if the system needs it:

---
Checking if kill() uses SYS$FORCEX, can't be called from a signal handler,
or fails to handle a signal value of zero...
---

It then runs this program:

#include <stdio.h>
#include <signal.h>
#include <unistd.h>
void handler1(int s) { printf("%d",s); kill(getpid(),2); }
void handler2(int s) { printf("%d",s); }

int main(){
printf("0");
signal(1,handler1);
signal(2,handler2);
kill(getpid(),1);
sleep(1);
kill(getpid(),0);
printf("3\n");
}

If you get anything other than "0123" from running this, then you need
the home-grown kill. I have never seen a VMS system that didn't need
it. Here's what I see with clang++:

$ cxx/pointer=64=argv check_kill.c
$ link check_kill
$ run check_kill
012

The fact that sys$sigprc is not available yet is not surprising. It
took a while to show up on Itanium. It involves stack unwinding and
other platform-specific nasties, but I'm pretty sure the OS itself uses
it and it will likely show up at some point.

On Monday, October 23, 2023 at 6:44:39 PM UTC-7, Craig A. Berry wrote:
>
> Perl pipes are not CRTL pipes. Defining DECC$MAILBOX_CTX_STM will have
> no effect on pipes in Perl. It apparently affects *something else* or
> you wouldn't have the whole test suite go belly up.
>
> The history here is that pipes were so bad in the CRTL in the v6.x era
> that Charles Lane contributed his own implementation to Perl based on
> mailboxes. It wasn't just pipe() but also popen(), which used its own
> hack (in DCL!) to hand off standard streams to child processes long
> before decc$set_child_standard_streams existed. And closing down pipes
> without hangs were also something this implementation did better than
> what was available in the CRTL at the time. Whether any of that is
> actually still true is a question worth asking. It's unlikely
> mailbox-based pipes by any implementer would have significant
> performance advantages over what we have now, but not maintaining all
> that mailbox code could have some benefits.
>
> Aside from being slow, the problem with mailboxes is that they are
> record-oriented devices. There are various mechanisms to emulate
> stream-oriented behavior, and as far as I understand them, they all
> depend on buffering. But pipes are by nature unbuffered and if you
> buffer them, the reader tends to hang at awkward moments.
>
> There is an implementation called dmpipe based on global sections that's
> going to perform a lot better than any mailbox-based pipe. I think we
> may have conversed about it before and you appear to have your own fork
> of it on github.

Yes, I will definitely have to revisit the "dmpipe" code and see how it compares to mailboxes.

Besides the performance and the issues with buffering and buffers, I'm a little wary of the difference in newline behavior. I'd like to have everything using UNIX-style "\n" endlines or perhaps "\r\n" when writing to a pseudo-terminal, so VMS putting the newline at the beginning and the CR at the end of each line seems like it could be a problem with interoperability.

I can see why VMS is delaying to output the newline: that way, you get a little extra time to read the top line of text before it scrolls out of view, rather than having a blank line at the bottom of your VT terminal waiting for the next line of output. :-)

> If you get anything other than "0123" from running this, then you need
> the home-grown kill. I have never seen a VMS system that didn't need
> it. Here's what I see with clang++:
>
> $ cxx/pointer=64=argv check_kill.c
> $ link check_kill
> $ run check_kill
> 012

That's true, but if you call __posix_kill() directly, you get the desired behavior. It's __posix_kill() vs. sys$sigprc that I've been comparing.

Normally you're supposed to define _POSIX_EXIT to get the POSIX version of kill(), but I can't define that and still get the VMS behavior for exit(). So I've patched my local branch to define kill() as __posix_kill() if the test case passes, with __posix_kill() being tested instead of kill().

There's no point calling the original test with VMS kill() if we already know that it's always going to fail. And if the test with __posix_kill() fails, then it must be an older version of the OS and we can fall through to the sys$sigprc behavior as before. If it succeeds, then we can use either.

> The fact that sys$sigprc is not available yet is not surprising. It
> took a while to show up on Itanium. It involves stack unwinding and
> other platform-specific nasties, but I'm pretty sure the OS itself uses
> it and it will likely show up at some point.

I should have clarified this. sys$sigprc is available on x86-64, and it works fine. What's not available in Clang yet (but does work in VSI C) is the lib$establish() call used by the test program to see if sys$sigprc works. That's how I went down the path of seeing if __posix_kill() might be more efficient, compatible, reliability, etc.. But so far I can't tell a difference in behavior between the two. You'd only see the test program failure in Clang and only until lib$establish support is added.

The only place where vaxc$establish (the old version of lib$establish) is actually used in Perl is in create_forked_xterm(), which looks up a private symbol from DECW$TERMINALSHR12 to call to create a new DECterm for use with the Perl debugger. That function has a comment that "LIB$FIND_IMAGE_SIGNAL" (actually "lib$find_image_symbol") needs a handler, and it sets "lib$sig_to_ret" to be that handler.

Regards,
Jake Hamby

On 2023-10-23 22:04:17 +0000, Jake Hamby (Solid State Jake) said:

> Returning to my original concern of which DECC$ features are good/bad
> (regardless of how to activate them), I think DECC$STREAM_PIPE is
> relatively safe, and speeds up pipes, along with DECC$STDIO_CTX_EOL,
> which buffers fwrite() output until a terminator or buffer full is
> reached.

I set all of the DECC$ latent bugs currently documented to either my
own non-default and expected values, or explicitly to the default
values, as app appropriate.

This works up until somebody at VSI creates a new latent bug, I miss
the instantiation of that new latent bug and the associated update to
my app, enough sites running the app update to the version with that
latent bug now available, and somebody then exposes that latent bug
somewhere upstream from my app.

Did I mention I dislike these out-of-band DECC$ latent bugs? These
latent bugs are more fun than the default values set by system
parameters bugs involving mailbox settings, though thankfully those
mailbox bugs are easier to avoid by explicitly specifying the mailbox
sizes and buffer quota values in the app.

--
Pure Personal Opinion | HoffmanLabs LLC

вторник, 24 октября 2023 г. в 21:20:12 UTC+2, Jake Hamby (Solid State Jake):

> Besides the performance and the issues with buffering and buffers, I'm a little wary of the difference in newline behavior. I'd like to have >everything using UNIX-style "\n" endlines or perhaps "\r\n" when writing to a pseudo-terminal, so VMS putting the newline at the beginning and >the CR at the end of each line seems like it could be a problem with interoperability.

Try enabling DECC$BASH_OUTPUT.

On Tuesday, October 24, 2023 at 1:22:24 PM UTC-7, Vitaly Pustovetov wrote:
> вторник, 24 октября 2023 г. в 21:20:12 UTC+2, Jake Hamby (Solid State Jake):
> > Besides the performance and the issues with buffering and buffers, I'm a little wary of the difference in newline behavior. I'd like to have >everything using UNIX-style "\n" endlines or perhaps "\r\n" when writing to a pseudo-terminal, so VMS putting the newline at the beginning and >the CR at the end of each line seems like it could be a problem with interoperability..
> Try enabling DECC$BASH_OUTPUT.

The DECC$BASH_OUTPUT logical doesn't seem to do anything for me, at least not here. I did some researching to find the history of Perl's custom VMS popen() implementation, which is over 1000 lines of dense VMS code, originally added in Aug. 2000:

https://github.com/Perl/perl5/commit/93d6612c1a533e775e2884e98da42e418edd3a83

I also found this thread on perl.vmsperl from 2003 that explores the rationale for the complexity and the vmspipe .com DCL script wrapper:

https://www.nntp.perl.org/group/perl.vmsperl/2003/05/msg11649.html

Even 20 years ago, there were people arguing to just use system(), popen(), etc. from the C RTL, and others saying that those functions don't really work well in VMS versions older than about V7.3-1, which had only just been released (in Aug. 2002). As Craig Berry said in the post I linked to:

"Yes, there has been *huge* improvement in the C RTL in recent years, but the piping implementation was still very buggy as of 7.2. Previous to Perl 5.6.1 we were using the C RTL for popen(), but a lot of piped I/O was just hanging in various deadlock conditions. Chuck cooked up what we currently have to get around these problems. It works rather well and has the added advantage of working on older VMS versions, which he and a lot of other people still use. Long term I'd like to see his piping implementation configurable so that with the flip of a switch we could get either his version or the C RTL version. That way if the C RTL version is ever better, we can use it when and where available."

What I'm going to try next is #ifdef'ing out Chuck Lane's custom pipe code and replacing it with popen() and pclose() to see what happens. In theory, that should be both fast and the most UNIX-compatible, especially if I enable DECC$STREAM_PIPE and increase DECC$PIPE_BUFFER_SIZE to 8192 (the default mailbox size that the custom pipe implementation uses).

BTW, the Perl configure script says VAX was supported until Perl 5.22, so the first version of Perl to require a 64-bit version of VMS was 5.24, released 8-May-2016. I wonder what the oldest version of Alpha OpenVMS is that people are still interested in running the latest version of Perl on. V7.3-2?

Regards,
Jake Hamby

On Wednesday, October 25, 2023 at 4:41:53 PM UTC-4, Jake Hamby (Solid State Jake) wrote:
> On Tuesday, October 24, 2023 at 1:22:24 PM UTC-7, Vitaly Pustovetov wrote:
>
> > Try enabling DECC$BASH_OUTPUT.
> The DECC$BASH_OUTPUT logical doesn't seem to do anything for me, at least not here. I did some researching to find the history of Perl's custom VMS popen() implementation, which is over 1000 lines of dense VMS code, originally added in Aug. 2000:

The available feature logicals for the C RTL are in the Introduction section 1.5 of the C run-time library reference manual
(https://docs.vmssoftware.com/vsi-c-run-time-library-reference-manual-for-openvms-systems/#RTINTRO_CHAP).

BASH_OUTPUT is not among the 75+ features.

If you have SQLite, the VFS will return the list using "SELECT * FROM pragma_vmsvfs_decc$;".

On 10/25/23 3:56 PM, David Jones wrote:
> On Wednesday, October 25, 2023 at 4:41:53 PM UTC-4, Jake Hamby (Solid State Jake) wrote:
>> On Tuesday, October 24, 2023 at 1:22:24 PM UTC-7, Vitaly Pustovetov wrote:
>>
>>> Try enabling DECC$BASH_OUTPUT.
>> The DECC$BASH_OUTPUT logical doesn't seem to do anything for me, at least not here. I did some researching to find the history of Perl's custom VMS popen() implementation, which is over 1000 lines of dense VMS code, originally added in Aug. 2000:
>
> The available feature logicals for the C RTL are in the Introduction section 1.5 of the C run-time library reference manual
> (https://docs.vmssoftware.com/vsi-c-run-time-library-reference-manual-for-openvms-systems/#RTINTRO_CHAP).
>
> BASH_OUTPUT is not among the 75+ features.

I believe what just happened is that a member of VMS Engineering told us
about an undocumented feature. The string does exist in the library:

$ SEARCH/FORMAT=NONULLS sys$library:DECC$SHR.EXE bash_output
DECC$FP_LOCKING_DISABLEDECC$STREAM_PIPEDECC$SYMLINKSDECC$COMMON_STDERR_STDOUTDECC$BASH_OUTPUTDECC$POSIX_COMPLIANT_PATHNAMESDECC$POSIX_COMPLIANT_V
ERSIONSDECC$MUNMAP_DELETEDECC$FWRITE_UNBUFFEREDDECC$FILE_WORLD_DELETEDECC$SSIODECC$SSIO_FOCDECC$EXIT_AFTER_FAILED_EXECDECC$SETVBUF_BUFFEREDDECC$P
RINTF_USES_VAX_ROUNDDECC$TERM_REC_CRLFDECC$PRN_PRE_BYTEDECC$FILENAME_ENCODING_UTF8DECC$VFC_REC_NOCFDECC$SLEEP_PENDING_WAKEDECC$FIL

I've never heard of this feature, but maybe it's a new one? Or has been
around but undocumented? If I weren't tired and lazy, I would check
whether the bash in GNV uses it.

On 10/25/23 3:41 PM, Jake Hamby (Solid State Jake) wrote:

> What I'm going to try next is #ifdef'ing out Chuck Lane's custom pipe code and replacing it with popen() and pclose() to see what happens. In theory, that should be both fast and the most UNIX-compatible, especially if I enable DECC$STREAM_PIPE and increase DECC$PIPE_BUFFER_SIZE to 8192 (the default mailbox size that the custom pipe implementation uses).

Please do try this. It's been on my list for a while, but I haven't
heard of any major changes to the pipe() implementation in the CRTL in
the last 20 years, so I'd expect about the same results now as those
that led to Perl's own pipe implementation. There was an alternate pipe
implementation of pipe() by VMS Engineering in India a few years ago. I
tested it and it did not solve any of the problems of the existing
implementation and as far as I know never saw the light of day.

With Perl's pipes, you can set PERL_MBX_SIZE instead of
DECC$PIPE_BUFFER_SIZE to do essentially the same thing if the default of
8192 is too small for you.

I think DECC$STREAM_PIPE just allows you to read a partial record from
the pipe's mailbox, using IO$M_STREAM on the underlying mailbox. I
experimented with IO$M_STREAM in Perl at some point and it made no
difference in anything I could see. The problem we've had with mailbox
pipes is that when writes are flushed all the way to the device, you
introduce a record boundary because mailboxes are fundamentally
record-oriented. Reading a partial record doesn't help you with those
spurious record boundaries.

For more details on IO$M_STREAM and mailboxes in general, have a look here:

http://h41379.www4.hpe.com/openvms/journal/v9/mailboxes.pdf

> BTW, the Perl configure script says VAX was supported until Perl 5.22, so the first version of Perl to require a 64-bit version of VMS was 5.24, released 8-May-2016. I wonder what the oldest version of Alpha OpenVMS is that people are still interested in running the latest version of Perl on. V7.3-2?

Don't know. With the end of the HPE Hobbyist program, I can't try it
out on anything pre-v8.4. I will point out that Perl has been working
continuously on VMS all these years, so if someone on an older version
of VMS can't get the latest Perl to build, they probably can easily get
a decent Perl coeval with or newer than their VMS release.

четверг, 26 октября 2023 г. в 00:17:29 UTC+2, Craig A. Berry:
> On 10/25/23 3:56 PM, David Jones wrote:

> I believe what just happened is that a member of VMS Engineering told us
> about an undocumented feature. The string does exist in the library:

Yes, this is one of the many undocumented decc$ features. We've added it to our bash port (not the GNV bash). Although it looks like it solves another issue with newlines.

On Wednesday, October 25, 2023 at 5:59:53 PM UTC-7, Craig A. Berry wrote:
> On 10/25/23 3:41 PM, Jake Hamby (Solid State Jake) wrote:
>
> > What I'm going to try next is #ifdef'ing out Chuck Lane's custom pipe code and replacing it with popen() and pclose() to see what happens. In theory, that should be both fast and the most UNIX-compatible, especially if I enable DECC$STREAM_PIPE and increase DECC$PIPE_BUFFER_SIZE to 8192 (the default mailbox size that the custom pipe implementation uses).
> Please do try this. It's been on my list for a while, but I haven't
> heard of any major changes to the pipe() implementation in the CRTL in
> the last 20 years, so I'd expect about the same results now as those
> that led to Perl's own pipe implementation. There was an alternate pipe
> implementation of pipe() by VMS Engineering in India a few years ago. I
> tested it and it did not solve any of the problems of the existing
> implementation and as far as I know never saw the light of day.
>
> With Perl's pipes, you can set PERL_MBX_SIZE instead of
> DECC$PIPE_BUFFER_SIZE to do essentially the same thing if the default of
> 8192 is too small for you.
>
> I think DECC$STREAM_PIPE just allows you to read a partial record from
> the pipe's mailbox, using IO$M_STREAM on the underlying mailbox. I
> experimented with IO$M_STREAM in Perl at some point and it made no
> difference in anything I could see. The problem we've had with mailbox
> pipes is that when writes are flushed all the way to the device, you
> introduce a record boundary because mailboxes are fundamentally
> record-oriented. Reading a partial record doesn't help you with those
> spurious record boundaries.
>
> For more details on IO$M_STREAM and mailboxes in general, have a look here:
>
> http://h41379.www4.hpe.com/openvms/journal/v9/mailboxes.pdf

I spent some time yesterday experimenting with the Perl source code and came to a few conclusions about how to proceed.

There are two interrelated sets of issues. The current version of Perl runs all external commands via lib$spawn() via a "vmspipe" DCL script which reads some temporarily-defined DCL symbols containing the sys$input, sys$output, and sys$error to set, along with a DCL version of the command to run (arguments quoted and merged into a single command string). The command is split into up to 4 symbols (PERL_POPEN_CMD0 to PERL_POPEN_CMD3) in case it's too long to fit in a single symbol.

What does this hack solve? It can run commands in the current directory or path (with MCR or RUN prepended), DCL scripts (with "@"), and DCL commands. What problems does it have if you try to use it with system mailboxes? The first that I ran into was having to add "/trans=term" to the lines redefining sys$input or sys$output or else a mailbox name like _MBA123: gets a directory and filename extension added, like "_MBA123:[local.perl].LIS", which doesn't open, of course. I'm not sure why I ran into this with my code but not the original version. At least it's easy to write debug output from the vmspipe script into a temp file to see what it's getting from Perl.

Here's where it gets tricky. lib$spawn() doesn't interoperate with waitpid(). The Perl custom pipe code gets the child process completion status from the completion code that's passed by reference in the lib$spawn() call, along with an AST completion callback. Whenever Perl's waitpid() is called, the VMS implementation checks the list of pipes and uses that to return the completion status, or waits in a loop for the status to change. This doesn't look very efficient:

for (info = open_pipes; info != NULL; info = info->next)
if (info->pid == pid) break;

if (info != NULL) { /* we know about this child */
while (!info->done) {
_ckvmssts(sys$setast(0));
done = info->done;
if (!done) _ckvmssts(sys$clref(pipe_ef));
_ckvmssts(sys$setast(1));
if (!done) _ckvmssts(sys$waitfr(pipe_ef));
}

if (statusp) *statusp = info->completion;
return pid;
}

There's a single event flag for all pipes. If I try to use waitpid() on a pid that wasn't created with one of the POSIX methods (exec, system, popen), it returns -1 immediately.

Conclusion #1: in order to replace the custom mailbox pipes with pipe(), the UNIX-style fork()/exec() code implementing Perl's popen (the OS's popen() is never used) using pipes, fork, and exec, can be modified somewhat to work with VMS vfork, as long as you remember (which I'd forgotten yesterday) that you can't close or move around file descriptors in the child, because when exec() returns to the parent, they'll be modified there. So I need to make sure to call decc$set_child_standard_streams() instead of dup2() and close().

The generic process-spawning and popen/pclose code is all in util.c, and already heavily #ifdef'd for Windows ("DOSISH" and "WIN32"), OS/2, z/OS ("#if defined(OEMVS)"), and AmigaOS 4, most notably. So modifying the existing #ifdef VMS code won't be too intrusive. I'm using "#ifdef MBX_PIPES" for the old code, partly to minimize the diff size by not deleting the code blocks..

Conclusion #2: if I modify the code to use vfork()/execvp() instead of lib$spawn(), then waitpid() will work and stream-oriented pipe setup should be as efficient as possible via the "VAXC$EXECMBX" set up between parent and child process, but the biggest downside is some of the VMS-specific behavior has to be dropped, or else emulated in extra code. The exec functions will work with both .EXE and .COM files, if found in DCL$PATH or the current directory iff DCL$PATH is undefined, and it automatically tries those filename extensions. You can't start an argv[0] with "@", even if it's a script. I don't think it's possible to execute built-in DCL commands this way.

For DCL foreign commands, I can borrow the approach of GNV bash and use lib$get_symbol() to look up any symbol with the command name, and then substitute it if found, removing any leading "$" or "@". The bash version (vms_source/bash/vms_get_foreign_cmd.c) truncates the returned value at the first space character.

This leads me to a few questions. Is it worth recursively looking up symbols in case a command is defined in terms of another symbol? What about arguments added to a command without spaces, like a MYCC defined as "CC/Opt/Define=(FOO)"? It seems like ideally any such options or parameters should be inserted as argv[1], before any parameters from Perl. But execvp() can't call "CC" or "DIR" no matter what's in argv.

Therefore, for DCL built-in commands to continue to work, I think a hack like the vmspipe script may still be needed, because then vfork/execvp can be used to call the script, enabling waitpid(), while the pipe script can in turn call the DCL command as before. Thoughts?

The last bit of special processing that VMS Perl does that's interwoven with the custom mailbox pipes is the call to getredirection(argcp, argvp) at the end of vms_image_init(), which is called early in Perl startup. This function parses the argv for UNIX-style redirection and background-process operators: "<", ">", ">>", "|", "&", sets up the needed file descriptors and background process for a pipe output, and modifies argc/argv in place. If the last argv[] entry is "&" or ends in "&", then the entire command is re-executed via background_process(), and the non-background original interpreter exits. This entire section of code was pasted into Perl in 1994 from a file written by Mark Pizzolato ((C) 1989-1994, 2007).

I know that Perl, at least to build itself, needs to be able to exec foreign commands such as "MMK", but I don't know if it ever runs DCL commands directly, except internally, in prime_env_iter(), to get defined symbols and logicals by calling lib$spawn on "Show Symbol/Global *", "Show Symbol/Local *", and "Show Logical *" with "/Table=". Even if not needed for the build, it does seem like you should still be able to open("DIR|") or open("CC|") in VMS Perl and get the output.

Is there no way to make that happen using only POSIX C RTL functions without calling a DCL script? It feels like a lot of VMS programs work around this by making a DCL script as a temp file, running it, then deleting the script, so in retrospect, Perl's use of a single shared "vmspipe .com" DCL script is likely much more efficient than a temp file create/write/close/open/read/close/delete. If only there were yet another DECC$ feature logical that one could set to enable vfork()/execvp() to run subprocesses using the same syntax as DCL accepts (split into command + argv parameters).

Regards,
Jake Hamby

On 10/26/23 12:06 PM, Jake Hamby (Solid State Jake) wrote:
> On Wednesday, October 25, 2023 at 5:59:53 PM UTC-7, Craig A. Berry wrote:
>> On 10/25/23 3:41 PM, Jake Hamby (Solid State Jake) wrote:
>>
>>> What I'm going to try next is #ifdef'ing out Chuck Lane's custom pipe code and replacing it with popen() and pclose() to see what happens. In theory, that should be both fast and the most UNIX-compatible, especially if I enable DECC$STREAM_PIPE and increase DECC$PIPE_BUFFER_SIZE to 8192 (the default mailbox size that the custom pipe implementation uses).
>> Please do try this. It's been on my list for a while, but I haven't
>> heard of any major changes to the pipe() implementation in the CRTL in
>> the last 20 years, so I'd expect about the same results now as those
>> that led to Perl's own pipe implementation. There was an alternate pipe
>> implementation of pipe() by VMS Engineering in India a few years ago. I
>> tested it and it did not solve any of the problems of the existing
>> implementation and as far as I know never saw the light of day.
>>
>> With Perl's pipes, you can set PERL_MBX_SIZE instead of
>> DECC$PIPE_BUFFER_SIZE to do essentially the same thing if the default of
>> 8192 is too small for you.
>>
>> I think DECC$STREAM_PIPE just allows you to read a partial record from
>> the pipe's mailbox, using IO$M_STREAM on the underlying mailbox. I
>> experimented with IO$M_STREAM in Perl at some point and it made no
>> difference in anything I could see. The problem we've had with mailbox
>> pipes is that when writes are flushed all the way to the device, you
>> introduce a record boundary because mailboxes are fundamentally
>> record-oriented. Reading a partial record doesn't help you with those
>> spurious record boundaries.
>>
>> For more details on IO$M_STREAM and mailboxes in general, have a look here:
>>
>> http://h41379.www4.hpe.com/openvms/journal/v9/mailboxes.pdf
>
> I spent some time yesterday experimenting with the Perl source code and came to a few conclusions about how to proceed.
>
> There are two interrelated sets of issues. The current version of Perl runs all external commands via lib$spawn() via a "vmspipe" DCL script which reads some temporarily-defined DCL symbols containing the sys$input, sys$output, and sys$error to set, along with a DCL version of the command to run (arguments quoted and merged into a single command string). The command is split into up to 4 symbols (PERL_POPEN_CMD0 to PERL_POPEN_CMD3) in case it's too long to fit in a single symbol.
>
> What does this hack solve? It can run commands in the current directory or path (with MCR or RUN prepended), DCL scripts (with "@"), and DCL commands. What problems does it have if you try to use it with system mailboxes? The first that I ran into was having to add "/trans=term" to the lines redefining sys$input or sys$output or else a mailbox name like _MBA123: gets a directory and filename extension added, like "_MBA123:[local.perl].LIS", which doesn't open, of course. I'm not sure why I ran into this with my code but not the original version. At least it's easy to write debug output from the vmspipe script into a temp file to see what it's getting from Perl.
>
> Here's where it gets tricky. lib$spawn() doesn't interoperate with waitpid(). The Perl custom pipe code gets the child process completion status from the completion code that's passed by reference in the lib$spawn() call, along with an AST completion callback. Whenever Perl's waitpid() is called, the VMS implementation checks the list of pipes and uses that to return the completion status, or waits in a loop for the status to change. This doesn't look very efficient:
>
> for (info = open_pipes; info != NULL; info = info->next)
> if (info->pid == pid) break;
>
> if (info != NULL) { /* we know about this child */
> while (!info->done) {
> _ckvmssts(sys$setast(0));
> done = info->done;
> if (!done) _ckvmssts(sys$clref(pipe_ef));
> _ckvmssts(sys$setast(1));
> if (!done) _ckvmssts(sys$waitfr(pipe_ef));
> }
>
> if (statusp) *statusp = info->completion;
> return pid;
> }
>
> There's a single event flag for all pipes. If I try to use waitpid() on a pid that wasn't created with one of the POSIX methods (exec, system, popen), it returns -1 immediately.
>
> Conclusion #1: in order to replace the custom mailbox pipes with pipe(), the UNIX-style fork()/exec() code implementing Perl's popen (the OS's popen() is never used) using pipes, fork, and exec, can be modified somewhat to work with VMS vfork, as long as you remember (which I'd forgotten yesterday) that you can't close or move around file descriptors in the child, because when exec() returns to the parent, they'll be modified there. So I need to make sure to call decc$set_child_standard_streams() instead of dup2() and close().
>
> The generic process-spawning and popen/pclose code is all in util.c, and already heavily #ifdef'd for Windows ("DOSISH" and "WIN32"), OS/2, z/OS ("#if defined(OEMVS)"), and AmigaOS 4, most notably. So modifying the existing #ifdef VMS code won't be too intrusive. I'm using "#ifdef MBX_PIPES" for the old code, partly to minimize the diff size by not deleting the code blocks.
>
> Conclusion #2: if I modify the code to use vfork()/execvp() instead of lib$spawn(), then waitpid() will work and stream-oriented pipe setup should be as efficient as possible via the "VAXC$EXECMBX" set up between parent and child process, but the biggest downside is some of the VMS-specific behavior has to be dropped, or else emulated in extra code. The exec functions will work with both .EXE and .COM files, if found in DCL$PATH or the current directory iff DCL$PATH is undefined, and it automatically tries those filename extensions. You can't start an argv[0] with "@", even if it's a script. I don't think it's possible to execute built-in DCL commands this way.
>
> For DCL foreign commands, I can borrow the approach of GNV bash and use lib$get_symbol() to look up any symbol with the command name, and then substitute it if found, removing any leading "$" or "@". The bash version (vms_source/bash/vms_get_foreign_cmd.c) truncates the returned value at the first space character.
>
> This leads me to a few questions. Is it worth recursively looking up symbols in case a command is defined in terms of another symbol? What about arguments added to a command without spaces, like a MYCC defined as "CC/Opt/Define=(FOO)"? It seems like ideally any such options or parameters should be inserted as argv[1], before any parameters from Perl. But execvp() can't call "CC" or "DIR" no matter what's in argv.
>
> Therefore, for DCL built-in commands to continue to work, I think a hack like the vmspipe script may still be needed, because then vfork/execvp can be used to call the script, enabling waitpid(), while the pipe script can in turn call the DCL command as before. Thoughts?
>
> The last bit of special processing that VMS Perl does that's interwoven with the custom mailbox pipes is the call to getredirection(argcp, argvp) at the end of vms_image_init(), which is called early in Perl startup. This function parses the argv for UNIX-style redirection and background-process operators: "<", ">", ">>", "|", "&", sets up the needed file descriptors and background process for a pipe output, and modifies argc/argv in place. If the last argv[] entry is "&" or ends in "&", then the entire command is re-executed via background_process(), and the non-background original interpreter exits. This entire section of code was pasted into Perl in 1994 from a file written by Mark Pizzolato ((C) 1989-1994, 2007).
>
> I know that Perl, at least to build itself, needs to be able to exec foreign commands such as "MMK", but I don't know if it ever runs DCL commands directly, except internally, in prime_env_iter(), to get defined symbols and logicals by calling lib$spawn on "Show Symbol/Global *", "Show Symbol/Local *", and "Show Logical *" with "/Table=". Even if not needed for the build, it does seem like you should still be able to open("DIR|") or open("CC|") in VMS Perl and get the output.
>
> Is there no way to make that happen using only POSIX C RTL functions without calling a DCL script? It feels like a lot of VMS programs work around this by making a DCL script as a temp file, running it, then deleting the script, so in retrospect, Perl's use of a single shared "vmspipe .com" DCL script is likely much more efficient than a temp file create/write/close/open/read/close/delete. If only there were yet another DECC$ feature logical that one could set to enable vfork()/execvp() to run subprocesses using the same syntax as DCL accepts (split into command + argv parameters).
>

On Thursday, October 26, 2023 at 1:52:03 PM UTC-7, Craig A. Berry wrote:
> >
> I don't think I know the answers to all your questions, but here are a
> few things to consider, in no particular order.
>
> Look for isdcl in setup_cmddsc(). There is already code that attempts to
> detect whether the command being spawned is DCL or is a program that can
> be run with MCR. Someone recommended a while back that we use exec() and
> decc$set_child_standard_streams for the latter cases and only use
> lib$spawn for the former. It's a reasonable suggestion that never got
> implemented.
>
> I'm not sure it's reasonable to abandon the lib$spawn entirely because
> Perl definitely needs the ability to run DCL commands in subprocesses.
> posix_spawn() is supposedly getting added to the CRTL but I don't know
> the status of that or whether it could handle both the exec and spawn
> cases. It might be possible to create all subprocesses via sys$creprc,
> specifying SYS$SYSTEM:LOGINOUT.EXE when you want DCL and the actual
> image if you want more exec-like behavior.
>
> Perl's home-grown waitpid() first looks for processes created by the
> home-grown popen(). If it doesn't find one it falls back to the CRTL,
> and if it still doesn't find one, it calls $getjpi once a second until
> it gets SS$_NONEXPR. So I'm skeptical of the claim that it always
> returns -1 if the process is not created by popen() or exec(). Possibly
> you don't have privileges to check the status of the target process?
>
> I've tried pretty hard to get rid of vmspipe.com and never found a way
> to do so. There is already fallback code that will create one on the
> fly if it's not found for some reason, so going the temp file route
> would basically just involve not installing the pre-built vmspipe.com.

After much trial and error, and figuring out how to wrap all the code blocks in order for the parent vfork() code path to free all of the memory properly (I've found that the vfork() and execv() really need to be in the same function and stack frame), I have a working prototype, and it even seems to be faster than the vmspipe version.

Sadly, you're right about DCL commands and having to use lib$spawn to execute them. My build fails as soon as it gets to ext/errno/errno_pm .pl which tries to run " LIBRARY/EXTRACT=ERRNO/OUTPUT=SYS\$OUTPUT $file |" (with the leading space). Before that, I had to patch make_ext .pl to not add quotes around the arguments, because they weren't getting stripped off, and then, after looking up foreign commands as symbols, peeling off any leading "@" or "$", and skipping any leading "mcr", at least I have system() working reliably and not leaving any orphaned subprocesses behind like some of my earlier experiments did.

The worst of all possible worlds would appear to be UNIX pipes with vfork()/exec() to run vmspipe. It works, but slowly, and not very well.

I think the best approach from here will be to use the isdcl() logic you mentioned to decide which way to go after creating the pipe() pair, and then I can keep around a very stripped-down version of the Pipe struct that just has the pid and completion status, and reuse the pclose()/waitpid() logic for that case, and use POSIX calls otherwise. The relevant files I'm looking at are:

- pp_sys.c - wrapper for system()
- util.c - wrappers for popen()/pclose()
- doio.c - wrappers do_aexec5() and do_exec3()

I'll make sure to clean up my diffs as best I can before checking them into my experimental branch. The one really ugly hack that I'd like to clean up was that in order to combine the fork() and exec() in my versions of do_aexec5() and do_exec3(), I had to change their function signatures to return Pid_t instead of bool, so the parent (with #ifdef'd changes for VMS) code path can execute with the pid that would have been returned by fork(). So that required a patch to embed.fnc and then regenerating proto.h.

To avoid the ugliness of changing a function signature for everyone, I'll probably add new vms_do_aexec5() and vms_do_exec3() prototypes in vms/vmsish..h with the Pid_t return type. The code can still be shared in doio.c, but with a different function name for the VMS version.

Regards,
Jake Hamby

On 10/26/23 10:10 PM, Jake Hamby (Solid State Jake) wrote:

>
> I think the best approach from here will be to use the isdcl() logic you mentioned to decide which way to go after creating the pipe() pair, and then I can keep around a very stripped-down version of the Pipe struct that just has the pid and completion status, and reuse the pclose()/waitpid() logic for that case, and use POSIX calls otherwise. The relevant files I'm looking at are:
>
> - pp_sys.c - wrapper for system()
> - util.c - wrappers for popen()/pclose()
> - doio.c - wrappers do_aexec5() and do_exec3()
>

I don't understand the reason for changes to those. It should be
possible to do everything as a modifications to my_popen() in vms/vms.c.

On Friday, October 27, 2023 at 7:12:57 AM UTC-7, Craig A. Berry wrote:
> On 10/26/23 10:10 PM, Jake Hamby (Solid State Jake) wrote:
>
> >
> > I think the best approach from here will be to use the isdcl() logic you mentioned to decide which way to go after creating the pipe() pair, and then I can keep around a very stripped-down version of the Pipe struct that just has the pid and completion status, and reuse the pclose()/waitpid() logic for that case, and use POSIX calls otherwise. The relevant files I'm looking at are:
> >
> > - pp_sys.c - wrapper for system()
> > - util.c - wrappers for popen()/pclose()
> > - doio.c - wrappers do_aexec5() and do_exec3()
> >
> I don't understand the reason for changes to those. It should be
> possible to do everything as a modifications to my_popen() in vms/vms.c.

You're absolutely right. The reason I went down that route was that I wanted to avoid code duplication while experimenting by reusing the more UNIX-like code paths. In retrospect, it's much too messy to commit, as it only makes the existing code paths more chaotic. I'll reset things back to the way they were, using the original my_popen() and the spawn() methods that get called for the "no-fork" paths in the files that I'd modified to use (somewhat) the "fork" code paths.

It's a good time for me to reset my local tree back to the current version of the code and reapply my diffs anyway, because I have some good and bad news about using vfork/exec vs. using lib$spawn. The good news is that for everything except DCL commands, you can use a very similar approach to the UNIX code paths, with things slightly rearranged and with decc$set_child_standard_streams(). In fact, it's very speedy this way as long as it's calling "MMK" and not "LIBRARY".

The bad news as far as lib$spawn is that while I think it's possible to call commands like "library" directly and not through a DCL script, I can't read any output of the DCL command from the read side of the UNIX file descriptor. I think it's returning an EOF immediately because it goes into the waitpid() and the DCL command never finishes so I think it needs the Perl process to read its output and it isn't. I think the write end of the pipe hasn't been opened by the subprocess by the time it tries to read.

I can see now why the current code has all the copying between mailboxes and explicit handling and sending of EOF's. I hadn't thought through the case where sys$input or sys$output might be a file or something that the subprocess might not have direct access to and might have to shuttle data back and forth between the file and the mailbox for the spawned subprocess. So I think all of that code may be necessary for those cases, but only for lib$spawn because the filename doesn't include file position and there could be locking issues, etc.. For the vfork/exec case, things seem to automatically get handed over in the way I'd expect.

I know now that it doesn't work to call lib$spawn() with the output descriptor set to the name of the mailbox I get from calling getname() on one of the file descriptors from pipe(), and then immediately start reading from the other end of the pipe. I think that what I may be able to do is open a second read channel on the pipe after closing the write end (or the read end, depending on the pipe direction) and then use sys$qiow on IO$_SETMODE | IO$M_WRITERWAIT (or IO$M_READERWAIT) so that I can delay returning from the my_popen() until after the spawned subprocess has actually opened the mailbox to write, and then hopefully it'll read until the (real) EOF.

For this approach to work, I'm assuming that the C RTL keeps an open I/O channel for the read file descriptor and one for the write file descriptor, and that by closing one or the other end, it'll drop the reference count down to 0 so that I can wait for the lib$spawn'd process to open it. In the less-than-ideal case where this approach returns immediately or doesn't fix the problem, then at least I can fall back on the code I know works.

Regards,
Jake Hamby

On 10/26/23 9:10 PM, Jake Hamby (Solid State Jake) wrote:
> On Thursday, October 26, 2023 at 1:52:03 PM UTC-7, Craig A. Berry wrote:
>>>
>> I don't think I know the answers to all your questions, but here are a
>> few things to consider, in no particular order.
>>
>> Look for isdcl in setup_cmddsc(). There is already code that attempts to
>> detect whether the command being spawned is DCL or is a program that can
>> be run with MCR. Someone recommended a while back that we use exec() and
>> decc$set_child_standard_streams for the latter cases and only use
>> lib$spawn for the former. It's a reasonable suggestion that never got
>> implemented.
>>
>> I'm not sure it's reasonable to abandon the lib$spawn entirely because
>> Perl definitely needs the ability to run DCL commands in subprocesses.
>> posix_spawn() is supposedly getting added to the CRTL but I don't know
>> the status of that or whether it could handle both the exec and spawn
>> cases. It might be possible to create all subprocesses via sys$creprc,
>> specifying SYS$SYSTEM:LOGINOUT.EXE when you want DCL and the actual
>> image if you want more exec-like behavior.
>>
>> Perl's home-grown waitpid() first looks for processes created by the
>> home-grown popen(). If it doesn't find one it falls back to the CRTL,
>> and if it still doesn't find one, it calls $getjpi once a second until
>> it gets SS$_NONEXPR. So I'm skeptical of the claim that it always
>> returns -1 if the process is not created by popen() or exec(). Possibly
>> you don't have privileges to check the status of the target process?
>>
>> I've tried pretty hard to get rid of vmspipe.com and never found a way
>> to do so. There is already fallback code that will create one on the
>> fly if it's not found for some reason, so going the temp file route
>> would basically just involve not installing the pre-built vmspipe.com.
>
> After much trial and error, and figuring out how to wrap all the code blocks in order for the parent vfork() code path to free all of the memory properly (I've found that the vfork() and execv() really need to be in the same function and stack frame), I have a working prototype, and it even seems to be faster than the vmspipe version.
>
> Sadly, you're right about DCL commands and having to use lib$spawn to execute them. My build fails as soon as it gets to ext/errno/errno_pm .pl which tries to run " LIBRARY/EXTRACT=ERRNO/OUTPUT=SYS\$OUTPUT $file |" (with the leading space). Before that, I had to patch make_ext .pl to not add quotes around the arguments, because they weren't getting stripped off, and then, after looking up foreign commands as symbols, peeling off any leading "@" or "$", and skipping any leading "mcr", at least I have system() working reliably and not leaving any orphaned subprocesses behind like some of my earlier experiments did.
>
> The worst of all possible worlds would appear to be UNIX pipes with vfork()/exec() to run vmspipe. It works, but slowly, and not very well.
>
> I think the best approach from here will be to use the isdcl() logic you mentioned to decide which way to go after creating the pipe() pair, and then I can keep around a very stripped-down version of the Pipe struct that just has the pid and completion status, and reuse the pclose()/waitpid() logic for that case, and use POSIX calls otherwise. The relevant files I'm looking at are:
>
> - pp_sys.c - wrapper for system()
> - util.c - wrappers for popen()/pclose()
> - doio.c - wrappers do_aexec5() and do_exec3()
>
> I'll make sure to clean up my diffs as best I can before checking them into my experimental branch. The one really ugly hack that I'd like to clean up was that in order to combine the fork() and exec() in my versions of do_aexec5() and do_exec3(), I had to change their function signatures to return Pid_t instead of bool, so the parent (with #ifdef'd changes for VMS) code path can execute with the pid that would have been returned by fork(). So that required a patch to embed.fnc and then regenerating proto.h.
>
> To avoid the ugliness of changing a function signature for everyone, I'll probably add new vms_do_aexec5() and vms_do_exec3() prototypes in vms/vmsish.h with the Pid_t return type. The code can still be shared in doio.c, but with a different function name for the VMS version.

In all cases where I have needed to "fork" a process with I/O passing
back and forth between the two processes, I have found that using
socketpair instead of pipes works much better. Socketpair is designed
for streams and completely eliminates the issue of pipes trying to be
record-oriented.

FYI in case you'd care to try it.

Mark Berryman