Suppose I have a program that, in part, runs another program
using FILE *outptr=popen("some command string","r") and then
fread()'s its stdout from outptr. Now, if that other program
is otherpgm which reads input from its stdin, and I also want
that input to come from a file inputfile, then I could easily
just write FILE *outptr=popen("cat inputfile|otherpgm","r");

But here's the rub: rather than some inputfile on disk, my
program has an internal unsigned char buffer[9999] which is
what I want piped to otherpgm, but which I don't want written
to disk first. How can I get buffer piped to otherpgm's stdin
using popen()? Or some other mechanism, as long as I can read
otherpgm's stdout from within my program.
--
John Forkosh ( mailto: j@f.com where j=john and f=forkosh )

On Saturday, 1 January 2022 at 09:35:59 UTC, John Forkosh wrote:
> Suppose I have a program that, in part, runs another program
> using FILE *outptr=popen("some command string","r") and then
> fread()'s its stdout from outptr. Now, if that other program
> is otherpgm which reads input from its stdin, and I also want
> that input to come from a file inputfile, then I could easily
> just write FILE *outptr=popen("cat inputfile|otherpgm","r");
>
> But here's the rub: rather than some inputfile on disk, my
> program has an internal unsigned char buffer[9999] which is
> what I want piped to otherpgm, but which I don't want written
> to disk first. How can I get buffer piped to otherpgm's stdin
> using popen()? Or some other mechanism, as long as I can read
> otherpgm's stdout from within my program.
> --
> John Forkosh ( mailto: j...@f.com where j=john and f=forkosh )
popen() is part of the POSIX spec, I believe.

This is possibly better suited to comp.unix.programming or a similar newsgroup. I'd probably dig out a copy of https://en.wikipedia.org/wiki/Advanced_Programming_in_the_Unix_Environment and remind myself how to do this sort of thing. It would probably involve fork/exec and explicit pipe management.

Mark Bluemel <mark.bluemel@gmail.com> writes:

> On Saturday, 1 January 2022 at 09:35:59 UTC, John Forkosh wrote:
>> Suppose I have a program that, in part, runs another program
>> using FILE *outptr=popen("some command string","r") and then
>> fread()'s its stdout from outptr. Now, if that other program
>> is otherpgm which reads input from its stdin, and I also want
>> that input to come from a file inputfile, then I could easily
>> just write FILE *outptr=popen("cat inputfile|otherpgm","r");
>>
>> But here's the rub: rather than some inputfile on disk, my
>> program has an internal unsigned char buffer[9999] which is
>> what I want piped to otherpgm, but which I don't want written
>> to disk first. How can I get buffer piped to otherpgm's stdin
>> using popen()? Or some other mechanism, as long as I can read
>> otherpgm's stdout from within my program.
>> --
>> John Forkosh ( mailto: j...@f.com where j=john and f=forkosh )
> popen() is part of the POSIX spec, I believe.
>
> This is possibly better suited to comp.unix.programming or a similar
> newsgroup.

You mean

comp.unix.programmer

On 2022-01-01, John Forkosh <forkosh@panix.com> wrote:
> Suppose I have a program that, in part, runs another program
> using FILE *outptr=popen("some command string","r") and then
> fread()'s its stdout from outptr. Now, if that other program
> is otherpgm which reads input from its stdin, and I also want
> that input to come from a file inputfile, then I could easily
> just write FILE *outptr=popen("cat inputfile|otherpgm","r");

That would be a useless use of cat; it could be done with:

FILE *outptr=popen("otherpgm < inputfile","r");
>
> But here's the rub: rather than some inputfile on disk, my
> program has an internal unsigned char buffer[9999] which is
> what I want piped to otherpgm, but which I don't want written
> to disk first. How can I get buffer piped to otherpgm's stdin
> using popen()? Or some other mechanism, as long as I can read
> otherpgm's stdout from within my program.

A program to which you send input while reading its output
is a coprocess. The popen function is for simple uses; it doesn't
support coprocesses. You have to cob those together yourself
with fork, exec, and various other functions.

With a coprocess, there is the risk of deadlock. It happens
like this: you write output for the process to read, while
it concurrently reads it and processes output. Because you're
writing, you're not reading. The process blocks trying to
write the data to you, and therefore stops reading. Since it
stops reading, you block writing to it.

In the general case, coprocesses require non-blocking I/O,
polling for input and output simultaneously, or else threads.
When the amount of data exchanged is small (so small that you can you
can rely on it all fitting into the pipe size on the given system)
deadlock may be avoided that way.

In the case of just wanting to feed a canned piece of data to
the process from memory, while reading its output, we can use
fork() to create a process which will do the feeding. We let
that run in the background. (Thus we have a multi-threaded solution via
multi-process, and do not need to juggle non-blocking I/O and polling.)

Just like in your original popen above, you have a "cat inputfile |
otherpgm" where cat is useless, a forked process is instantiated which
"cats" the memory buffer. That "memory cat" is not useless because a
piece of memory will not serve itself as a file.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal

In article <20220101101839.914@kylheku.com>,
Kaz Kylheku <480-992-1380@kylheku.com> wrote:
>On 2022-01-01, John Forkosh <forkosh@panix.com> wrote:
>> Suppose I have a program that, in part, runs another program
>> using FILE *outptr=popen("some command string","r") and then
>> fread()'s its stdout from outptr. Now, if that other program
>> is otherpgm which reads input from its stdin, and I also want
>> that input to come from a file inputfile, then I could easily
>> just write FILE *outptr=popen("cat inputfile|otherpgm","r");
>
>That would be a useless use of cat; it could be done with:
>
> FILE *outptr=popen("otherpgm < inputfile","r");

Yeah, I noticed that, too. I think that, in this day and age, we can
probably ignore it - assume it was there only for pedagogic purposes, and
that OP actually knows better.

That said, two observations on the original question:

1) This is, of course, completely OT in clc. People who live their lives
around keeping clc pure and untainted by OT posts will no doubt start
jumping up and down about this any post now...

2) There are at least a half-dozen possible solutions to OP's problem that
spring to mind almost immediately. Which one is best will depend on two
things:
a) How deep OP's problem actually is. I.e., how much rigor is needed
in a solution - i.e., which solution is going to be "good enough".
b) OP's skill level - and how much grit-and-grime he is willing to put
up with.

That all said, one solution that comes to mind is to use bash's <<<
operator, like this:

sprintf(buffer,"otherpgm <<< '%s'",myinputstring);
FILE *outptr=popen(buffer,"r");

Now, the problem with this is that, on most systems, things run with
popen() don't get bash, even if bash is the standard and most commonly used
shell on the system. Sometimes, they get bash, but invoke it in a way that
disables most of the bash goodies (e.g., <<<).

Fixing this is left as an exercise for the reader...
(It *can* be done, but it gets messy)

--
The randomly chosen signature file that would have appeared here is more than 4
lines long. As such, it violates one or more Usenet RFCs. In order to remain
in compliance with said RFCs, the actual sig can be found at the following URL:
http://user.xmission.com/~gazelle/Sigs/FreeCollege

John Forkosh <forkosh@panix.com> writes:
> Suppose I have a program that, in part, runs another program
> using FILE *outptr=popen("some command string","r") and then
> fread()'s its stdout from outptr. Now, if that other program
> is otherpgm which reads input from its stdin, and I also want
> that input to come from a file inputfile, then I could easily
> just write FILE *outptr=popen("cat inputfile|otherpgm","r");
>
> But here's the rub: rather than some inputfile on disk, my
> program has an internal unsigned char buffer[9999] which is
> what I want piped to otherpgm, but which I don't want written
> to disk first. How can I get buffer piped to otherpgm's stdin
> using popen()? Or some other mechanism, as long as I can read
> otherpgm's stdout from within my program.

popen() is POSIX, not ISO C. Try comp.unix.programmer.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips
void Void(void) { Void(); } /* The recursive call of the void */

In article <87fsq712p5.fsf@nosuchdomain.example.com>,
Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
>John Forkosh <forkosh@panix.com> writes:
>> Suppose I have a program that, in part, runs another program
>> using FILE *outptr=popen("some command string","r") and then
>> fread()'s its stdout from outptr. Now, if that other program
>> is otherpgm which reads input from its stdin, and I also want
>> that input to come from a file inputfile, then I could easily
>> just write FILE *outptr=popen("cat inputfile|otherpgm","r");
>>
>> But here's the rub: rather than some inputfile on disk, my
>> program has an internal unsigned char buffer[9999] which is
>> what I want piped to otherpgm, but which I don't want written
>> to disk first. How can I get buffer piped to otherpgm's stdin
>> using popen()? Or some other mechanism, as long as I can read
>> otherpgm's stdout from within my program.
>
>popen() is POSIX, not ISO C. Try comp.unix.programmer.

What'd I tell ya???

--
There are two kinds of Republicans: Billionaires and suckers.
Republicans: Please check your bank account and decide which one is you.

Kenny McCormack <gazelle@shell.xmission.com> wrote:
> Kaz Kylheku <480-992-1380@kylheku.com> wrote:
>> John Forkosh <forkosh@panix.com> wrote:
>>> Suppose I have a program that, in part, runs another program
>>> using FILE *outptr=popen("some command string","r") and then
>>> fread()'s its stdout from outptr. Now, if that other program
>>> is otherpgm which reads input from its stdin, and I also want
>>> that input to come from a file inputfile, then I could easily
>>> just write FILE *outptr=popen("cat inputfile|otherpgm","r");
>>
>>That would be a useless use of cat; it could be done with:
>>
>> FILE *outptr=popen("otherpgm < inputfile","r");
>
> Yeah, I noticed that, too. I think that, in this day and age, we can
> probably ignore it - assume it was there only for pedagogic purposes,
> and that OP actually knows better.

Yup (maybe "illustrative" rather than "pedagogic").

> That said, two observations on the original question:
>
> 1) This is, of course, completely OT in clc. People who live their
> lives around keeping clc pure and untainted by OT posts will no doubt
> start jumping up and down about this any post now...

Sorry about that, though I think maybe in stackexchange there'd possibly
be two simultaneously relevant tags: Unix C. I'm obviously interested
in implementing a solution, in C, whereby the C-relevance.

> 2) There are at least a half-dozen possible solutions to OP's problem
> that spring to mind almost immediately. Which one is best will depend
> on two things:
> a) How deep OP's problem actually is. I.e., how much rigor is needed
> in a solution - i.e., which solution is going to be "good enough".
If it works, it's good enough.

> b) OP's skill level - and how much grit-and-grime he is willing to put
> up with.
If it works, I'm probably willing to put up with it.

> That all said, one solution that comes to mind is to use bash's <<<
> operator, like this:
>
> sprintf(buffer,"otherpgm <<< '%s'",myinputstring);
> FILE *outptr=popen(buffer,"r");

Problem is that myinputstring (which is what I called
unsigned char buffer[9999] in original question) can be
many megabytes long, containing lots of hex chars (including \000).
If it were short and ascii, I might have illustratively
alternatively written
sprintf(buffer,"echo \"%s\" | otherpgm",myinputstring);
along the same lines as our "cat" discussion above.

> Now, the problem with this is that, on most systems,
> things run with popen() don't get bash, even if bash
> is the standard and most commonly used shell on the system.
> Sometimes, they get bash, but invoke it in a way that
> disables most of the bash goodies (e.g., <<<).
>
> Fixing this is left as an exercise for the reader...
> (It *can* be done, but it gets messy)

--
John Forkosh ( mailto: j@f.com where j=john and f=forkosh )

Kaz Kylheku <480-992-1380@kylheku.com> wrote:
> On 2022-01-01, John Forkosh <forkosh@panix.com> wrote:
>> Suppose I have a program that, in part, runs another program
>> using FILE *outptr=popen("some command string","r") and then
>> fread()'s its stdout from outptr. Now, if that other program
>> is otherpgm which reads input from its stdin, and I also want
>> that input to come from a file inputfile, then I could easily
>> just write FILE *outptr=popen("cat inputfile|otherpgm","r");
>
> That would be a useless use of cat; it could be done with:
> FILE *outptr=popen("otherpgm < inputfile","r");

Thanks, Kaz:
Yeah, I'm not actually doing either; the chosen illustration
was basically just pipes versus redirection, and since I was
kind of guessing the ultimate answer more likely involves pipes,
I illustrated it along those lines.

>> But here's the rub: rather than some inputfile on disk, my
>> program has an internal unsigned char buffer[9999] which is
>> what I want piped to otherpgm, but which I don't want written
>> to disk first. How can I get buffer piped to otherpgm's stdin
>> using popen()? Or some other mechanism, as long as I can read
>> otherpgm's stdout from within my program.

> A program to which you send input while reading its output
> is a coprocess. The popen function is for simple uses; it doesn't
> support coprocesses. You have to cob those together yourself
> with fork, exec, and various other functions.

Yeah, I've done a little, but not much, of that. And, as
suggested in a preceding followup, I actually have a hardback
paper copy of Stevens' APUE, which I occasionally refer to.
But I couldn't quite figure out what to do from his Chapters
14 and 15, assuming that's where an answer's buried.

> With a coprocess, there is the risk of deadlock. It happens
> like this: you write output for the process to read, while
> it concurrently reads it and processes output. Because you're
> writing, you're not reading. The process blocks trying to
> write the data to you, and therefore stops reading. Since it
> stops reading, you block writing to it.

Not a problem in my case. The 'otherpgm' reads all its input
before it writes anything.

> In the general case, coprocesses require non-blocking I/O,
> polling for input and output simultaneously, or else threads.
> When the amount of data exchanged is small (so small that you can you
> can rely on it all fitting into the pipe size on the given system)
> deadlock may be avoided that way.
>
> In the case of just wanting to feed a canned piece of data to
> the process from memory, while reading its output, we can use
> fork() to create a process which will do the feeding.

Feed how, exactly??? That had occurred to me (at least if I
correctly understand what you're saying), but as far as I could
tell that forked child is in exactly the same predicament as the
parent was in my original question: how does it pipe (or redirect
or whatever) a large unsigned char buffer[9999] (actually
malloc/realloc'ed, often larger than 20MB) from its memory to
the stdin of that 'otherpgm'?

> We let that run in the background. (Thus we have a multi-threaded
> solution via multi-process, and do not need to juggle non-blocking
> I/O and polling.)
>
> Just like in your original popen above, you have a "cat inputfile |
> otherpgm" where cat is useless, a forked process is instantiated which
> "cats" the memory buffer. That "memory cat" is not useless because a
> piece of memory will not serve itself as a file.

--
John Forkosh ( mailto: j@f.com where j=john and f=forkosh )

Mark Bluemel <mark.bluemel@gmail.com> wrote:
> John Forkosh wrote:
>> Suppose I have a program that, in part, runs another program
>> using FILE *outptr=popen("some command string","r") and then
>> fread()'s its stdout from outptr. Now, if that other program
>> is otherpgm which reads input from its stdin, and I also want
>> that input to come from a file inputfile, then I could easily
>> just write FILE *outptr=popen("cat inputfile|otherpgm","r");
>>
>> But here's the rub: rather than some inputfile on disk, my
>> program has an internal unsigned char buffer[9999] which is
>> what I want piped to otherpgm, but which I don't want written
>> to disk first. How can I get buffer piped to otherpgm's stdin
>> using popen()? Or some other mechanism, as long as I can read
>> otherpgm's stdout from within my program.
>> --
>> John Forkosh ( mailto: j...@f.com where j=john and f=forkosh )
> popen() is part of the POSIX spec, I believe.
>
> This is possibly better suited to comp.unix.programming
> or a similar newsgroup. I'd probably dig out a copy of
> https://en.wikipedia.org/wiki/Advanced_Programming_in_the_Unix_Environment
> and remind myself how to do this sort of thing.
> It would probably involve fork/exec and explicit pipe management.

Thanks, Mark. Yeah, I actually have a hardback paper copy of APUE,
purchased in the 1990's. I've only occasionally (very occasionally)
used fork, etc, and couldn't quite figure out how to do what I want
from Chapters 14 and 15. The answer may well be there, but I couldn't
quite figure it out.
--
John Forkosh ( mailto: j@f.com where j=john and f=forkosh )

On Sat, 01 Jan 2022 09:35:48 +0000, John Forkosh wrote:

> Suppose I have a program that, in part, runs another program
> using FILE *outptr=popen("some command string","r") and then
> fread()'s its stdout from outptr. Now, if that other program
> is otherpgm which reads input from its stdin, and I also want
> that input to come from a file inputfile, then I could easily
> just write FILE *outptr=popen("cat inputfile|otherpgm","r");
>
> But here's the rub: rather than some inputfile on disk, my
> program has an internal unsigned char buffer[9999] which is
> what I want piped to otherpgm, but which I don't want written
> to disk first. How can I get buffer piped to otherpgm's stdin
> using popen()? Or some other mechanism, as long as I can read
> otherpgm's stdout from within my program.

Assuming a Unix-like environment,
pipe() to create a pair of unidirectional pipes
fork() to create a child process
in parent, close() the [0] pipe, and write() your data to the [1] pipe
in child, close(STDIN_FILENO), dup() the [1] pipe, and then exec() your
target "some command string" executable

If you want, we can have a more detailed discussion on comp.unix.programmer

--
Lew Pitcher
"In Skills, We Trust"

In article <sqs7a1$o59$1@dont-email.me>,
Lew Pitcher <lew.pitcher@digitalfreehold.ca> wrote:
....
>Assuming a Unix-like environment,
> pipe() to create a pair of unidirectional pipes
> fork() to create a child process
> in parent, close() the [0] pipe, and write() your data to the [1] pipe
> in child, close(STDIN_FILENO), dup() the [1] pipe, and then exec() your
> target "some command string" executable

The problem with this is that, as Kaz alluded to, pipes are only guaranteed
to be at least 4096 bytes in size - and deadlock is possible. The original
post suggested that the input string was about 10K (which is already too
big) in size, but subsequent followups by OP have stated that it could be
multi-megabytes. So, you can't just shove it all into the pipe in one go
(w/o having the reader be reading some of it as you go).

Also, OP was using popen() to encapsulate the gory details of pipe/fork/dup/exec.
Going this route, he would lose the simplicity/elegance of popen().

>If you want, we can have a more detailed discussion on comp.unix.programmer

Quite so. I hope OP starts a thread there, so we can continue.

I wouldn't want to give any of the Not-Cs in this group anymore mental pain.

--
Faced with the choice between changing one's mind and proving that there is
no need to do so, almost everyone gets busy on the proof.

- John Kenneth Galbraith -

On Sun, 02 Jan 2022 14:54:18 +0000, Kenny McCormack wrote:

> In article <sqs7a1$o59$1@dont-email.me>,
> Lew Pitcher <lew.pitcher@digitalfreehold.ca> wrote:
> ...

I posted this breadcrumb clue to give the OP a start in his
research. APUE covers all of these steps, and discusses the
resulting program design implications.

>>Assuming a Unix-like environment,
>> pipe() to create a pair of unidirectional pipes
>> fork() to create a child process
>> in parent, close() the [0] pipe, and write() your data to the [1] pipe
>> in child, close(STDIN_FILENO), dup() the [1] pipe, and then exec() your
>> target "some command string" executable
>
> The problem with this is that, as Kaz alluded to, pipes are only guaranteed
> to be at least 4096 bytes in size - and deadlock is possible.
[snip]

It was precisely this detail that prompted me to suggest that we continue the
conversation on comp.unix.programmer.

>>If you want, we can have a more detailed discussion on comp.unix.programmer
>
> Quite so. I hope OP starts a thread there, so we can continue.
>
> I wouldn't want to give any of the Not-Cs in this group anymore mental pain.

--
Lew Pitcher
"In Skills, We Trust"

Lew Pitcher <lew.pitcher@digitalfreehold.ca> wrote:
> John Forkosh wrote:
>> Suppose I have a program that, in part, runs another program
>> using FILE *outptr=popen("some command string","r") and then
>> fread()'s its stdout from outptr. Now, if that other program
>> is otherpgm which reads input from its stdin, and I also want
>> that input to come from a file inputfile, then I could easily
>> just write FILE *outptr=popen("cat inputfile|otherpgm","r");
>>
>> But here's the rub: rather than some inputfile on disk, my
>> program has an internal unsigned char buffer[9999] which is
>> what I want piped to otherpgm, but which I don't want written
>> to disk first. How can I get buffer piped to otherpgm's stdin
>> using popen()? Or some other mechanism, as long as I can read
>> otherpgm's stdout from within my program.
>
> Assuming a Unix-like environment,
> pipe() to create a pair of unidirectional pipes
> fork() to create a child process
> in parent, close() the [0] pipe, and write() your data to the [1] pipe
> in child, close(STDIN_FILENO), dup() the [1] pipe, and then exec() your
> target "some command string" executable
> If you want, we can have a more detailed discussion on comp.unix.programmer

Thanks, Lew, that looks like a real good synopsis, modulo Kenny's
pipe size followup remark which might pose a problem. Let me take
another look through APUE with your synopsis in mind, and maybe I
can figure it out from there without too much more detailed disc-
ussion. Google, e.g.,
http://unixwiz.net/techtips/remap-pipe-fds.html
also turned up what seems to be some useful discussion.

And, again, apologies to those (as Kenny put it), "Not-Cs in this
group". However, I did notice a whole bunch of "Beautiful Russian
Women" posts (from "outpost season") in comp.unix.programmer, and
nobody seemed to be complaining much. I suppose beautiful Russian
women are just more interesting than Unix pipes. (Maybe we should
instead be discussing their 'stdin's and 'stdout's?:)
--
John Forkosh ( mailto: j@f.com where j=john and f=forkosh )

Kenny McCormack <gazelle@shell.xmission.com> wrote:
> Lew Pitcher <lew.pitcher@digitalfreehold.ca> wrote:
> ...
>>Assuming a Unix-like environment,
>> pipe() to create a pair of unidirectional pipes
>> fork() to create a child process
>> in parent, close() the [0] pipe, and write() your data to the [1] pipe
>> in child, close(STDIN_FILENO), dup() the [1] pipe, and then exec() your
>> target "some command string" executable
>
> The problem with this is that, as Kaz alluded to, pipes are only guaranteed
> to be at least 4096 bytes in size - and deadlock is possible. The original
> post suggested that the input string was about 10K (which is already too
> big) in size, but subsequent followups by OP have stated that it could be
> multi-megabytes. So, you can't just shove it all into the pipe in one go
> (w/o having the reader be reading some of it as you go).

Yeah, definitely can be multi-MB input to stdin (when I originally
wrote "unsigned char buffer[9999]", I hadn't meant to imply that as
a maximum). Thanks for the additional warning, and I'll explicitly
look for that discussion while munging through APUE again.

> Also, OP was using popen() to encapsulate the gory details of
> pipe/fork/dup/exec. Going this route, he would lose the
> simplicity/elegance of popen().

Yeah, I already pretty much figured out from the preceding discussion
that "lose the simplicity/elegance" was a foregone conclusion.
--
John Forkosh ( mailto: j@f.com where j=john and f=forkosh )

In article <sqsp80$fmk$2@reader1.panix.com>,
John Forkosh <forkosh@panix.com> wrote:
....
>> Also, OP was using popen() to encapsulate the gory details of
>> pipe/fork/dup/exec. Going this route, he would lose the
>> simplicity/elegance of popen().
>
>Yeah, I already pretty much figured out from the preceding discussion
>that "lose the simplicity/elegance" was a foregone conclusion.

It's not.

I have in mind a solution that is simple, elegant and Linux-specific. It
does not require you to give up using popen().

Once we get this moved to comp.unix.programmer, we can discuss it further.

--
I love the poorly educated.

On 2022-01-02, John Forkosh <forkosh@panix.com> wrote:
> Kaz Kylheku <480-992-1380@kylheku.com> wrote:
>> On 2022-01-01, John Forkosh <forkosh@panix.com> wrote:
>>> Suppose I have a program that, in part, runs another program
>>> using FILE *outptr=popen("some command string","r") and then
>>> fread()'s its stdout from outptr. Now, if that other program
>>> is otherpgm which reads input from its stdin, and I also want
>>> that input to come from a file inputfile, then I could easily
>>> just write FILE *outptr=popen("cat inputfile|otherpgm","r");
>>
>> That would be a useless use of cat; it could be done with:
>> FILE *outptr=popen("otherpgm < inputfile","r");
>
> Thanks, Kaz:
> Yeah, I'm not actually doing either; the chosen illustration
> was basically just pipes versus redirection, and since I was
> kind of guessing the ultimate answer more likely involves pipes,
> I illustrated it along those lines.
>
>>> But here's the rub: rather than some inputfile on disk, my
>>> program has an internal unsigned char buffer[9999] which is
>>> what I want piped to otherpgm, but which I don't want written
>>> to disk first. How can I get buffer piped to otherpgm's stdin
>>> using popen()? Or some other mechanism, as long as I can read
>>> otherpgm's stdout from within my program.
>
>> A program to which you send input while reading its output
>> is a coprocess. The popen function is for simple uses; it doesn't
>> support coprocesses. You have to cob those together yourself
>> with fork, exec, and various other functions.
>
> Yeah, I've done a little, but not much, of that. And, as
> suggested in a preceding followup, I actually have a hardback
> paper copy of Stevens' APUE, which I occasionally refer to.
> But I couldn't quite figure out what to do from his Chapters
> 14 and 15, assuming that's where an answer's buried.
>
>> With a coprocess, there is the risk of deadlock. It happens
>> like this: you write output for the process to read, while
>> it concurrently reads it and processes output. Because you're
>> writing, you're not reading. The process blocks trying to
>> write the data to you, and therefore stops reading. Since it
>> stops reading, you block writing to it.
>
> Not a problem in my case. The 'otherpgm' reads all its input
> before it writes anything.

That only exacerbates deadlock.

>
>> In the general case, coprocesses require non-blocking I/O,
>> polling for input and output simultaneously, or else threads.
>> When the amount of data exchanged is small (so small that you can you
>> can rely on it all fitting into the pipe size on the given system)
>> deadlock may be avoided that way.
>>
>> In the case of just wanting to feed a canned piece of data to
>> the process from memory, while reading its output, we can use
>> fork() to create a process which will do the feeding.
>
> Feed how, exactly??? That had occurred to me (at least if I

The feeding process (forked child) just sits in a loop sending the bytes
to one end of a pipe. That pipe is installed as the standard input
of the coprocess.

> correctly understand what you're saying), but as far as I could
> tell that forked child is in exactly the same predicament as the
> parent was in my original question: how does it pipe (or redirect
> or whatever) a large unsigned char buffer[9999] (actually
> malloc/realloc'ed, often larger than 20MB) from its memory to
> the stdin of that 'otherpgm'?

Your shell can do it:

cat_memory()
{
echo "string in shell's memory"
}

upcased=$(cat_memory | tr '[a-z]' '[A-Z]')

# upcased is now "STRING IN SHELL'S MEMORY"

The shell forks a process which calls the function (i.e. does not exec
anything external). And it forks a process for the "tr" command. As it's
doing that, it hooks up the pipes in all the right ways and so it goes.

The shell itself goes into a loop reading the output of the pipe,
so that it can gather it into a string via the $(...) syntax, and then
assign that to a variable when it's done.

Kaz Kylheku <480-992-1380@kylheku.com> wrote:
> John Forkosh <forkosh@panix.com> wrote:
>> Kaz Kylheku <480-992-1380@kylheku.com> wrote:
>>> On 2022-01-01, John Forkosh <forkosh@panix.com> wrote:
>>>> Suppose I have a program that, in part, runs another program
>>>> using FILE *outptr=popen("some command string","r") and then
>>>> fread()'s its stdout from outptr. Now, if that other program
>>>> is otherpgm which reads input from its stdin, and I also want
>>>> that input to come from a file inputfile, then I could easily
>>>> just write FILE *outptr=popen("cat inputfile|otherpgm","r");
>>>
>>> That would be a useless use of cat; it could be done with:
>>> FILE *outptr=popen("otherpgm < inputfile","r");
>>
>> Thanks, Kaz:
>> Yeah, I'm not actually doing either; the chosen illustration
>> was basically just pipes versus redirection, and since I was
>> kind of guessing the ultimate answer more likely involves pipes,
>> I illustrated it along those lines.
>>
>>>> But here's the rub: rather than some inputfile on disk, my
>>>> program has an internal unsigned char buffer[9999] which is
>>>> what I want piped to otherpgm, but which I don't want written
>>>> to disk first. How can I get buffer piped to otherpgm's stdin
>>>> using popen()? Or some other mechanism, as long as I can read
>>>> otherpgm's stdout from within my program.
>>
>>> A program to which you send input while reading its output
>>> is a coprocess. The popen function is for simple uses; it doesn't
>>> support coprocesses. You have to cob those together yourself
>>> with fork, exec, and various other functions.
>>
>> Yeah, I've done a little, but not much, of that. And, as
>> suggested in a preceding followup, I actually have a hardback
>> paper copy of Stevens' APUE, which I occasionally refer to.
>> But I couldn't quite figure out what to do from his Chapters
>> 14 and 15, assuming that's where an answer's buried.
>>
>>> With a coprocess, there is the risk of deadlock. It happens
>>> like this: you write output for the process to read, while
>>> it concurrently reads it and processes output. Because you're
>>> writing, you're not reading. The process blocks trying to
>>> write the data to you, and therefore stops reading. Since it
>>> stops reading, you block writing to it.
>>
>> Not a problem in my case. The 'otherpgm' reads all its input
>> before it writes anything.
>
> That only exacerbates deadlock.

Oops, my bad. Guess I've got even more reading to do than I thought,
before I can wrap my head around how to do this properly.

>>> In the general case, coprocesses require non-blocking I/O,
>>> polling for input and output simultaneously, or else threads.
>>> When the amount of data exchanged is small (so small that you can you
>>> can rely on it all fitting into the pipe size on the given system)
>>> deadlock may be avoided that way.
>>>
>>> In the case of just wanting to feed a canned piece of data to
>>> the process from memory, while reading its output, we can use
>>> fork() to create a process which will do the feeding.
>>
>> Feed how, exactly??? That had occurred to me (at least if I
>
> The feeding process (forked child) just sits in a loop sending the bytes
> to one end of a pipe. That pipe is installed as the standard input
> of the coprocess.

Yeah, the synopsis posted by Lew
>> pipe() to create a pair of unidirectional pipes
>> fork() to create a child process
>> in parent, close() the [0] pipe, and write() your data to the [1] pipe
>> in child, close(STDIN_FILENO), dup() the [1] pipe, and then exec() your
>> target "some command string" executable
seems like a good outline to guide my further APUE reading,
maybe along with the example (also cited earlier) at
http://unixwiz.net/techtips/remap-pipe-fds.html

>> correctly understand what you're saying), but as far as I could
>> tell that forked child is in exactly the same predicament as the
>> parent was in my original question: how does it pipe (or redirect
>> or whatever) a large unsigned char buffer[9999] (actually
>> malloc/realloc'ed, often larger than 20MB) from its memory to
>> the stdin of that 'otherpgm'?
>
> Your shell can do it:
>
> cat_memory()
> {
> echo "string in shell's memory"
> }
>
> upcased=$(cat_memory | tr '[a-z]' '[A-Z]')
>
> # upcased is now "STRING IN SHELL'S MEMORY"
>
> The shell forks a process which calls the function (i.e. does not exec
> anything external). And it forks a process for the "tr" command. As it's
> doing that, it hooks up the pipes in all the right ways and so it goes.
>
> The shell itself goes into a loop reading the output of the pipe,
> so that it can gather it into a string via the $(...) syntax, and then
> assign that to a variable when it's done.

Thanks, Kaz, Kenny, Lew, Keith...
Sounded to me like straightforward functionality when I initially
asked how to do it, and therefore supposed there'd be a pretty easy
and straightforward way to implement it. But now seems to be a bit
harder than I expected. I'll nevertheless continue studying APUE and
all the stuff you've suggested, but for the time being just continue
using the already-coded-and-working (but inelegant) solution -- just
write the first program's output to a /tmp/file and then popen() the
second program with an "-f /tmp/file" to read that file (and finally
rm it). But I'll eventually re-implement that with your suggestions.
--
John Forkosh ( mailto: j@f.com where j=john and f=forkosh )

On 1/3/2022 9:07 AM, John Forkosh wrote:
> Kaz Kylheku <480-992-1380@kylheku.com> wrote:
>> John Forkosh <forkosh@panix.com> wrote:
>>> Kaz Kylheku <480-992-1380@kylheku.com> wrote:
>>>> On 2022-01-01, John Forkosh <forkosh@panix.com> wrote:
>>>>> Suppose I have a program that, in part, runs another program
>>>>> using FILE *outptr=popen("some command string","r") and then
>>>>> fread()'s its stdout from outptr. Now, if that other program
>>>>> is otherpgm which reads input from its stdin, and I also want
>>>>> that input to come from a file inputfile, then I could easily
>>>>> just write FILE *outptr=popen("cat inputfile|otherpgm","r");
>>>>
>>>> That would be a useless use of cat; it could be done with:
>>>> FILE *outptr=popen("otherpgm < inputfile","r");
>>>
>>> Thanks, Kaz:
>>> Yeah, I'm not actually doing either; the chosen illustration
>>> was basically just pipes versus redirection, and since I was
>>> kind of guessing the ultimate answer more likely involves pipes,
>>> I illustrated it along those lines.
>>>
>>>>> But here's the rub: rather than some inputfile on disk, my
>>>>> program has an internal unsigned char buffer[9999] which is
>>>>> what I want piped to otherpgm, but which I don't want written
>>>>> to disk first. How can I get buffer piped to otherpgm's stdin
>>>>> using popen()? Or some other mechanism, as long as I can read
>>>>> otherpgm's stdout from within my program.
>>>
>>>> A program to which you send input while reading its output
>>>> is a coprocess. The popen function is for simple uses; it doesn't
>>>> support coprocesses. You have to cob those together yourself
>>>> with fork, exec, and various other functions.
>>>
>>> Yeah, I've done a little, but not much, of that. And, as
>>> suggested in a preceding followup, I actually have a hardback
>>> paper copy of Stevens' APUE, which I occasionally refer to.
>>> But I couldn't quite figure out what to do from his Chapters
>>> 14 and 15, assuming that's where an answer's buried.
>>>
>>>> With a coprocess, there is the risk of deadlock. It happens
>>>> like this: you write output for the process to read, while
>>>> it concurrently reads it and processes output. Because you're
>>>> writing, you're not reading. The process blocks trying to
>>>> write the data to you, and therefore stops reading. Since it
>>>> stops reading, you block writing to it.
>>>
>>> Not a problem in my case. The 'otherpgm' reads all its input
>>> before it writes anything.
>>
>> That only exacerbates deadlock.
>
> Oops, my bad. Guess I've got even more reading to do than I thought,
> before I can wrap my head around how to do this properly.
>
>>>> In the general case, coprocesses require non-blocking I/O,
>>>> polling for input and output simultaneously, or else threads.
>>>> When the amount of data exchanged is small (so small that you can you
>>>> can rely on it all fitting into the pipe size on the given system)
>>>> deadlock may be avoided that way.
>>>>
>>>> In the case of just wanting to feed a canned piece of data to
>>>> the process from memory, while reading its output, we can use
>>>> fork() to create a process which will do the feeding.
>>>
>>> Feed how, exactly??? That had occurred to me (at least if I
>>
>> The feeding process (forked child) just sits in a loop sending the bytes
>> to one end of a pipe. That pipe is installed as the standard input
>> of the coprocess.
>
> Yeah, the synopsis posted by Lew
> >> pipe() to create a pair of unidirectional pipes
> >> fork() to create a child process
> >> in parent, close() the [0] pipe, and write() your data to the [1] pipe

I don't want to confuse things, but, while the above is correct (parent
closes p[0] and writes to p[1]), the part below is not:
child should close p[1], and read from p[0]. If you need to read from
stdin, then I believe the [0] pipe should be dup'd to stdin accordingly
(I've not used dup myself too much).

See 'man pipe' for a clear description and example code of this part.

> >> in child, close(STDIN_FILENO), dup() the [1] pipe, and then exec() your
> >> target "some command string" executable
> seems like a good outline to guide my further APUE reading,
> maybe along with the example (also cited earlier) at
> http://unixwiz.net/techtips/remap-pipe-fds.html
>
>>> correctly understand what you're saying), but as far as I could
>>> tell that forked child is in exactly the same predicament as the
>>> parent was in my original question: how does it pipe (or redirect
>>> or whatever) a large unsigned char buffer[9999] (actually
>>> malloc/realloc'ed, often larger than 20MB) from its memory to
>>> the stdin of that 'otherpgm'?
>>
>> Your shell can do it:
>>
>> cat_memory()
>> {
>> echo "string in shell's memory"
>> }
>>
>> upcased=$(cat_memory | tr '[a-z]' '[A-Z]')
>>
>> # upcased is now "STRING IN SHELL'S MEMORY"
>>
>> The shell forks a process which calls the function (i.e. does not exec
>> anything external). And it forks a process for the "tr" command. As it's
>> doing that, it hooks up the pipes in all the right ways and so it goes.
>>
>> The shell itself goes into a loop reading the output of the pipe,
>> so that it can gather it into a string via the $(...) syntax, and then
>> assign that to a variable when it's done.
>
> Thanks, Kaz, Kenny, Lew, Keith...
> Sounded to me like straightforward functionality when I initially
> asked how to do it, and therefore supposed there'd be a pretty easy
> and straightforward way to implement it. But now seems to be a bit
> harder than I expected. I'll nevertheless continue studying APUE and
> all the stuff you've suggested, but for the time being just continue
> using the already-coded-and-working (but inelegant) solution -- just
> write the first program's output to a /tmp/file and then popen() the
> second program with an "-f /tmp/file" to read that file (and finally
> rm it). But I'll eventually re-implement that with your suggestions.