I recently had to deal with some strange effects around process-pipes:

The one bit where I might have left the "beaten path" is, that the
sub-process writes to both stdout and stderr, but both shall be
captured from the pipe, so there is a "2>@1" redirection involved.
Not sure, if this is relevant, though.

Case 1:
the process writes to stdout, closes stdout, then writes something
to stderr, and finally exits. (stderr only gets closed by exit.)

Observation:
Once the sub-process closes stdout, it can still write to stderr,
but that isn't seen by tclsh. From standard Unix experience, I
would have expected that the subprocess would have two fd for
stdout and stderr that just to to the same channel. closing one
of them would leave the channel intact.
According to "truss -ff ..." output, the child writes to stdout,
closes stdout, writes to stderr and exits, but tclsh only reads
what the subprocess wroe before the close of stdout. Everything
afterwards just wasn't received by the tcl script-process.
(actually, the child process was a tcl-script as well.)

I could workaround it, by avoiding the explicit close for stdout.

Case 2:
the child process writes data on stderr and stdout, and eventually
exits with a non-zero exit code.

Observation:
tcl notices the eof on the channel, does a "NOHANG"-wait to see
for the child process' exit code, but doesn't hit it.
(despite truss showing the child's exit() a few lines before the wait.)
Only a bit later, truss shows the parent receiving SIGCHLD.
Result is, that "close $fd" finishes fine, and the child's
exitcode gets lost.

My workaround is, to write "Exitcode=..." to stderr in child, and
check for it in the parent. quite ugly ;-)

If that is an inevitable race-condition between the child closing the
channel and exiting, and the parent acting too fast on closed
channels, then I think we could need something for "close",
which currently accepts a second argument for half-closing, to
accept a third value like "p(rocess)" there to enforce waiting on the
child process (without NOHANG) and be sure to capture the exitcode.

I vaguely remember some discussion about process management features
in Tcl, but that seems to be not in 8.6.

I'd do the TIP for a "really-wait-for-subprocess-exit" close, but hope
to get some opinions on it. Maybe the future process management
in 8.7 or 9 obviates the use for it, anyway.

Case 3:
I do an "exec hd $filename | head" to get the first 10 lines
of hexdump of a given file.

On shell (bash), hd foo | head works fine. No one sees, that
hd gets a broken pipe, once head has finished.

in tcl, however, exec throws an error, for "hd"'s exit-code.
Is there a way to tell exec to ignore exitcodes of all but the
last command of a pipe line?

wrapping it in a catch doesn't help, because exec adds the text
"child killed: write on pipe with no readers" to the output.
redirecting stderr doesn't help... the line isn't written by
the process afterall, but added by exec based on exit-status
of some process in the pipe.

Andreas Leitgeb <avl@logic.at> wrote:
> I recently had to deal with some strange effects around process-pipes:
>
> The one bit where I might have left the "beaten path" is, that the
> sub-process writes to both stdout and stderr, but both shall be
> captured from the pipe, so there is a "2>@1" redirection involved.
> Not sure, if this is relevant, though.

I suspect it is relevant.

> Case 1:
> the process writes to stdout, closes stdout, then writes something
> to stderr, and finally exits. (stderr only gets closed by exit.)
>
> Observation:
> Once the sub-process closes stdout, it can still write to stderr,
> but that isn't seen by tclsh. From standard Unix experience, I
> would have expected that the subprocess would have two fd for
> stdout and stderr that just to to the same channel. closing one
> of them would leave the channel intact.
> According to "truss -ff ..." output, the child writes to stdout,
> closes stdout, writes to stderr and exits, but tclsh only reads
> what the subprocess wroe before the close of stdout. Everything
> afterwards just wasn't received by the tcl script-process.
> (actually, the child process was a tcl-script as well.)
>
> I could workaround it, by avoiding the explicit close for stdout.

The equivalent Bash operator "2>&1" is a file descriptor duplication
operation. It means to take the current file descriptor that is fd1
(stdout) and duplicate it to be fd2 (stderr). The result is that while
the process stdout and stderr channels open in the standard positions,
both refer to the same "destination" file descriptor. So closing
either one is also a close of the other. Which would explain exactly
the behavior you see.

> Case 2:
> the child process writes data on stderr and stdout, and eventually
> exits with a non-zero exit code.

I can't offer any explanations on this one.

> Case 3:
> I do an "exec hd $filename | head" to get the first 10 lines
> of hexdump of a given file.
>
> On shell (bash), hd foo | head works fine. No one sees, that
> hd gets a broken pipe, once head has finished.
>
> in tcl, however, exec throws an error, for "hd"'s exit-code.
> Is there a way to tell exec to ignore exitcodes of all but the
> last command of a pipe line?

Not so elegant, but it does suppress the error:

exec bash -c "hd $filename || echo -n" | head

* Andreas Leitgeb <avl@logic.at>
| I recently had to deal with some strange effects around process-pipes:
>
| The one bit where I might have left the "beaten path" is, that the
| sub-process writes to both stdout and stderr, but both shall be
| captured from the pipe, so there is a "2>@1" redirection involved.
| Not sure, if this is relevant, though.

As Rich pointed out, it is relevant for case 1. As an alternative you
could create a pipe and redirect stderr to that, see chan(n) for details
(this case is explicitely noted there)

https://www.tcl-lang.org/man/tcl/TclCmd/chan.htm#M29

chan pipe
Creates a standalone pipe whose read- and write-side channels are
returned as a 2-element list, the first element being the read side
and the second the write side. Can be useful e.g. to redirect
separately stderr and stdout from a subprocess. To do this, spawn
with "2>@" or ">@" redirection operators onto the write side of a
pipe, and then immediately close it in the parent. This is necessary
to get an EOF on the read side once the child has exited or
otherwise closed its output.

| Case 2:
| the child process writes data on stderr and stdout, and eventually
| exits with a non-zero exit code.
>
| Observation:
| tcl notices the eof on the channel, does a "NOHANG"-wait to see
| for the child process' exit code, but doesn't hit it.
| (despite truss showing the child's exit() a few lines before the wait.)
| Only a bit later, truss shows the parent receiving SIGCHLD.
| Result is, that "close $fd" finishes fine, and the child's
| exitcode gets lost.

Do you by any chance have the channel configured non-blocking?
The channel needs to be in blocking mode to get the exit status of
processes.
https://www.tcl-lang.org/man/tcl/TclCmd/open.htm#M21
https://www.tcl-lang.org/man/tcl/TclCmd/close.htm

| Case 3:
| I do an "exec hd $filename | head" to get the first 10 lines
| of hexdump of a given file.
>
| On shell (bash), hd foo | head works fine. No one sees, that
| hd gets a broken pipe, once head has finished.

If you have control over 'hd' source code, you could make it silently
ignore SIGPIPE... other than that, again see Rich's suggestion.

HTH
R'

Rich <rich@example.invalid> wrote:
> Andreas Leitgeb <avl@logic.at> wrote:
>> I recently had to deal with some strange effects around process-pipes:
>> The one bit where I might have left the "beaten path" is, that the
>> sub-process writes to both stdout and stderr, but both shall be
>> captured from the pipe, so there is a "2>@1" redirection involved.
>> Not sure, if this is relevant, though.
> I suspect it is relevant.

>> Case 1:
>> the process writes to stdout, closes stdout, then writes something
>> to stderr, and finally exits. (stderr only gets closed by exit.)
>>
>> Observation:
>> Once the sub-process closes stdout, it can still write to stderr,
>> but that isn't seen by tclsh.
>> I could workaround it, by avoiding the explicit close for stdout.
>
> The equivalent Bash operator "2>&1" is a file descriptor duplication
> operation. It means to take the current file descriptor that is fd1
> (stdout) and duplicate it to be fd2 (stderr). The result is that while
> the process stdout and stderr channels open in the standard positions,
> both refer to the same "destination" file descriptor.

So far, I agree, except that I'd refer to some channel as the thing
both fd now refer to, rather than yet another fd.

> So closing either one is also a close of the other.

in bash:
$ bash -c 'echo a;echo b >&2;exec>/dev/null;echo c >&2' |& sed 's/^/x/'
xa
xb
xc
$

The "inner" bash inherits the pipe as it's stdout, and sees stderr
redirected to the same pipe. The "|&" in bash is just a shorthand
for "2>&1 |" .

It writes to both stdout and stderr, and both get through to sed.
Then, by redirecting stdout to /dev/null, the previous fd1 is closed
(and replaced by a new fd1), but fd2 (stderr) is still connected to
the pipe, and "c" still gets through to sed.

Apparently, tclsh does it somehow "differently" than bash.

I just can't tell yet, if that is a bug or a feature ;-)

>> Case 3:
>> I do an "exec hd $filename | head" to get the first 10 lines
>> of hexdump of a given file.
>> in tcl, however, exec throws an error, for "hd"'s exit-code.
>> Is there a way to tell exec to ignore exitcodes of all but the
>> last command of a pipe line?
>
> Not so elegant, but it does suppress the error:
> exec bash -c "hd $filename || echo -n" | head

not bad... I'd just need a safe bash-escaper for the filename :-)
Wrapping it in a pair of ' and replacing all ' inside the name to
something like '\'' should do it. Given that hassle, however, I'm
likely going to re-implement "hd" in tcl, instead :-)

Thanks a lot for the thoughts!

Ralf Fassel <ralfixx@gmx.de> wrote:
> * Andreas Leitgeb <avl@logic.at>
>| I recently had to deal with some strange effects around process-pipes:
>>
>| The one bit where I might have left the "beaten path" is, that the
>| sub-process writes to both stdout and stderr, but both shall be
>| captured from the pipe, so there is a "2>@1" redirection involved.
>| Not sure, if this is relevant, though.
>
> As Rich pointed out, it is relevant for case 1. As an alternative you
> could create a pipe and redirect stderr to that, see chan(n) for details
> (this case is explicitely noted there)

> https://www.tcl-lang.org/man/tcl/TclCmd/chan.htm#M29
> chan pipe
> Creates a standalone pipe whose read- and write-side channels are
> returned as a 2-element list, >

Cool! Wasn't aware of that. Did it arrive with "chan" ?
Will come very handy, if I ever need stdout and stderr separately. I always
thought of memchan, but that wouldn't work, whereas "chan pipe" is just what
the doctor prescribed :-)

>| Case 2:
>| the child process writes data on stderr and stdout, and eventually
>| exits with a non-zero exit code.
>| Observation:
>| tcl notices the eof on the channel, does a "NOHANG"-wait to see
>| for the child process' exit code, but doesn't hit it.
> Do you by any chance have the channel configured non-blocking?

You nailed it. And I'm embarrassed, because I was aware of the
blocking for half-closing only the write-side, but didn't think of
it for the final full-close.

Thanks! That of course obviates any TIP I had in mind for that.

>| Case 3:
>| I do an "exec hd $filename | head" to get the first 10 lines
>| of hexdump of a given file.

> If you have control over 'hd' source code, you could make it silently
> ignore SIGPIPE... other than that, again see Rich's suggestion.

The particular "hd" is actually my own (http://avl.enemy.org/utils/hextools/)
but its rather designed for "KISS", so I rather won't add any signal-handliing
to it.
Rich's suggestion is fine, too, but I'll probably just end up regexp'ing
away the pipe error from output.

Maybe for Case 3, a TIP for a new option to prevent exec from adding such
diagnose to the output might be worth it...

Andreas Leitgeb <avl@logic.at> wrote:
> in bash:
> $ bash -c 'echo a;echo b >&2;exec>/dev/null;echo c >&2' |& sed 's/^/x/'
> xa
> xb
> xc
> $

Maybe that test wasn't perfect, because

set fd [open "|[list bash -c {echo a;echo b >&2;exec>/dev/null;echo c >&2} 2>@1]" r]
puts [read $fd]
close $fd

does receive all three letter lines...

Seems like I need to research this a bit further...
and I'll let this case rest for now.