I recently got interested in the Rexx language as part of learning about IBM mainframes, and discovered that it's a good scripting language for other platforms as well. So why not port Regina (free Rexx interpreter) to VMS? After all, I've always been disappointed in DCL as a scripting language, and Rexx just seems like it'd be a good fit for VMS.

I discovered that Regina already *had* been ported to VMS, many years ago, and in fact had 3000+ lines of code to implement all of the useful F$ lexical functions from DCL directly as Rexx functions, which is pretty cool. I've been fixing bugs and updating it on my GitHub project page (link below), and I noticed some bizarre behavior in the vfork()/exec() implementation, at least in OpenVMS Alpha V8.4-2L1 (I'll try to test on Itanium as soon as I get access to my old boxes) that I wanted to ask the experts about. This is with VMS842L1A_RTL V3.0 installed.

My first issue was that vfork() doesn't preserve the file descriptors, unlike BSD or Linux vfork(), so I had to dup() the old stdin, stdout, and stderr before replacing them and calling vfork(), and then, in the parent path, reset everything the way it was before returning. That's annoying, but expected. The manual didn't say that the C RTL would preserve and restore the status of file descriptors that you dup2()'d or close()'d in the child.

But I think I've found an actual bug. When execvp() fails due to e.g. file not found, the manual, and traditional vfork() behavior, would have me call either _exit() or exit() in order to return to the parent. But if I do that, my process exits! As a workaround, I tried *returning* from my function, as the child, to see if the CRTL would jump back to the parent code path, and that actually works, but it feels really wrong to do something as hacky as returning from a function as the vfork child, rather than calling _exit().

I did try undefining _POSIX_EXIT, which I'd enabled, and that didn't make any difference. I've copied the vms_crtl_init.c file from GNV bash into the project to enable more UNIX behaviors, but didn't add anything unusual. Is this a known issue/bug in the V8.4-2L1 VMS CRTL?

This feels like a rather severe bug to be showing up suddenly in such core POSIX functionality, but if it's been this way for a long time, that's even more worrisome.

Thanks in advance for any thoughts/suggestions.

Link to my fork of Regina with VMS fixes: https://github.com/jhamby/vms-regina/tree/jhamby/vms-fixes

Regards,
Jake Hamby

On 5/4/22 2:19 AM, Jake Hamby wrote:
> My first issue was that vfork() doesn't preserve the file
> descriptors, unlike BSD or Linux vfork(), so I had to dup() the old stdin, stdout,
> and stderr before replacing them and calling vfork(), and then, in the
> parent path, reset everything the way it was before returning. That's
> annoying, but expected. The manual didn't say that the C RTL would
> preserve and restore the status of file descriptors that you dup2()'d or
> close()'d in the child.

You may want to look into decc$set_child_standard_streams and/or
decc$set_child_default_dir.

> But I think I've found an actual bug. When execvp() fails due to
> e.g. file not found, the manual, and traditional vfork() behavior, would have
> me call either _exit() or exit() in order to return to the parent. But
> if I do that, my process exits!

As far as I know, until the exec succeeds, you don't actually have a
child process. This is the documented behavior of vfork() on VMS, which
is really just a set-up routine comparable to setjmp(). So it makes
sense that exiting after a failed exec is actually exiting from the parent.

On Wednesday, May 4, 2022 at 5:55:23 AM UTC-7, Craig A. Berry wrote:
> You may want to look into decc$set_child_standard_streams and/or
> decc$set_child_default_dir.

I'll do that, thanks.

> As far as I know, until the exec succeeds, you don't actually have a
> child process. This is the documented behavior of vfork() on VMS, which
> is really just a set-up routine comparable to setjmp(). So it makes
> sense that exiting after a failed exec is actually exiting from the parent.

Yes, I'm aware of that, which is why the current behavior does make sense, but the problem is that the C RTL reference manual tells me specifically to call exit() or _exit() from the child in case of failure. Other vfork() man pages say _exit() is safer because it doesn't flush streams, so that's what I tried first, and I assumed there was a hook in both of those functions that would detect when it's being called by a vfork() "child" and handle it specially, because otherwise that means the C RTL reference manual is wrong and it needs to be updated to tell us what we're supposed to do in the child when the exec() fails, to return to the parent context. I'll return from the function for now, but it's not what I'm supposed to be doing according to the docs.

-Jake

On 5/4/22 1:14 PM, Jake Hamby wrote:
> On Wednesday, May 4, 2022 at 5:55:23 AM UTC-7, Craig A. Berry wrote:
>
>> As far as I know, until the exec succeeds, you don't actually have a
>> child process. This is the documented behavior of vfork() on VMS, which
>> is really just a set-up routine comparable to setjmp(). So it makes
>> sense that exiting after a failed exec is actually exiting from the parent.

> Yes, I'm aware of that, which is why the current behavior does make
> sense, but the problem is that the C RTL reference manual tells me
> specifically to call exit() or _exit() from the child in case of
> failure. Other vfork() man pages say _exit() is safer because it doesn't
> flush streams, so that's what I tried first, and I assumed there was a
> hook in both of those functions that would detect when it's being called
> by a vfork() "child" and handle it specially, because otherwise that
> means the C RTL reference manual is wrong and it needs to be updated to
> tell us what we're supposed to do in the child when the exec() fails, to
> return to the parent context. I'll return from the function for now, but
> it's not what I'm supposed to be doing according to the docs.

You're right. I hadn't noticed the docs said this, i.e., bottom of page
118 here:

https://vmssoftware.com/docs/VSI_CRTL_REF.pdf

However, in the reference section for exit(), page 307 of that manual,
it also says:

-----
Beginning with OpenVMS Version 8.3, C RTL contains a fix for the problem
in which a call to _exit after a failed execl really exits but must not.

In the OpenVMS implementation of vfork, a child process is not actually
started as it is started on most UNIX systems. However, the C RTL
creates some internal data structures intended to mimic child-process
functionality (called the "child context").

A bug occurred whereby after a vfork while in the child context, a call
to an exec function justifiably fails, then calls _exit. On UNIX
systems, after the failed exec call, the child process continues to
execute. A subsequent call to _exit terminates the child. In the OpenVMS
implementation, after the failed exec call, the child context
terminates. A subsequent call to _exit terminates the parent. The C RTL
fix is enabled by a feature logical switch, DECC$EXIT_AFTER_FAILED_EXEC.
Enabling this feature logical allows the child context to continue
execution.

With DECC$EXIT_AFTER_FAILED_EXEC disabled or not defined, the current
behavior remains the default.
----

So if you want to preserve some degree of portability, I guess you could
add DECC$EXIT_AFTER_FAILED_EXEC to your vms_crtl_init.c. Or you could
just not call exit after a failed exec.

On Wednesday, May 4, 2022 at 11:14:41 AM UTC-7, jake....@gmail.com wrote:
> but the problem is that the C RTL reference manual tells me specifically to call exit() or _exit()
> from the child in case of failure.

yeah, well, the manual also says that sem_timedwait takes an absolute timeout. it doesn't.
--
roger ivie
roger.ivie@gmail.com

On Wednesday, May 4, 2022 at 1:12:01 PM UTC-7, Craig A. Berry wrote:
>
> In the OpenVMS implementation of vfork, a child process is not actually
> started as it is started on most UNIX systems. However, the C RTL
> creates some internal data structures intended to mimic child-process
> functionality (called the "child context").
>
> A bug occurred whereby after a vfork while in the child context, a call
> to an exec function justifiably fails, then calls _exit. On UNIX
> systems, after the failed exec call, the child process continues to
> execute. A subsequent call to _exit terminates the child. In the OpenVMS
> implementation, after the failed exec call, the child context
> terminates. A subsequent call to _exit terminates the parent. The C RTL
> fix is enabled by a feature logical switch, DECC$EXIT_AFTER_FAILED_EXEC.
> Enabling this feature logical allows the child context to continue
> execution.
>
> With DECC$EXIT_AFTER_FAILED_EXEC disabled or not defined, the current
> behavior remains the default.
> ----
>
> So if you want to preserve some degree of portability, I guess you could
> add DECC$EXIT_AFTER_FAILED_EXEC to your vms_crtl_init.c. Or you could
> just not call exit after a failed exec.

Thanks, that's *exactly* what I was hoping to hear! I'll give it a try.

Cheers,
Jake

On Wednesday, May 4, 2022 at 12:19:46 AM UTC-7, jake....@gmail.com wrote:

(snip)

>
> My first issue was that vfork() doesn't preserve the file descriptors,
> unlike BSD or Linux vfork(), so I had to dup() the old stdin, stdout,
> and stderr before replacing them and calling vfork(), and then,
> in the parent path, reset everything the way it was before returning.
> That's annoying, but expected. The manual didn't say that the C RTL
> would preserve and restore the status of file descriptors that you
> dup2()'d or close()'d in the child.

From man vfork() on my system:

"vfork() can be used to create new processes without fully copying the address space of the old
process, which is horrendously inefficient in a paged environment. It is useful when the pur-
pose of fork(2) would have been to create a new system context for an execve. vfork() differs
from fork in that the child borrows the parent's memory and thread of control until a call to
execve(2) or an exit (either by a call to exit(2) or abnormally.) The parent process is sus-
pended while the child is using its resources.

vfork() returns 0 in the child's context and (later) the pid of the child in the parent's con-
text.

vfork() can normally be used just like fork. It does not work, however, to return while run-
ning in the childs context from the procedure that called vfork() since the eventual return
from vfork() would then return to a no longer existent stack frame. Be careful, also, to call
_exit rather than exit if you can't execve, since exit will flush and close standard I/O chan-
nels, and thereby mess up the parent processes standard I/O data structures. (Even with fork
it is wrong to call exit since buffered data would then be flushed twice.)"