Hi,

I have a command which produces output like the
following:

Job ID: 9431211
Cluster: curta
User/Group: build/staff
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 8
CPU Utilized: 01:30:53
CPU Efficiency: 83.63% of 01:48:40 core-walltime
Job Wall-clock time: 00:13:35
Memory Utilized: 6.45 GB
Memory Efficiency: 80.68% of 8.00 GB

I want to parse this and am using subprocess.Popen and accessing the
contents via Popen.stdout. However, for testing purposes I want to save
various possible outputs of the command as text files and use those as
inputs.

What format should I use to pass data to the actual parsing function?

I could in both production and test convert the entire input to a string
and pass the string to the parsing method.

However, I could use something like

test_input_01 = subprocess.Popen(
["cat test_input_01.txt"],
stdout=subprocess.PIPE,
)

for the test input and then pass a Popen object to the parsing function.

Any comments on these alternative or suggestions for doing something
completely different?

Cheers,

Loris
--
This signature is currently under construction.

Loris Bennett wrote at 2022-3-10 13:16 +0100:
>I have a command which produces output like the
>following:
>
> Job ID: 9431211
> Cluster: curta
> User/Group: build/staff
> State: COMPLETED (exit code 0)
> Nodes: 1
> Cores per node: 8
> CPU Utilized: 01:30:53
> CPU Efficiency: 83.63% of 01:48:40 core-walltime
> Job Wall-clock time: 00:13:35
> Memory Utilized: 6.45 GB
> Memory Efficiency: 80.68% of 8.00 GB
>
>I want to parse this and am using subprocess.Popen and accessing the
>contents via Popen.stdout. However, for testing purposes I want to save
>various possible outputs of the command as text files and use those as
>inputs.

What do you want to test? the parsing? the "popen" interaction?
You can separately test both tasks (I, at your place, would do this).

For the parsing test, it is not relevant that the actual text
comes from an external process. You can directly read it from a file
or have it in your text.

In my view, you do not need a test for the `Popen` interaction:
if it works once, it will work always.

Dieter Maurer <dieter@handshake.de> writes:

> Loris Bennett wrote at 2022-3-10 13:16 +0100:
>>I have a command which produces output like the
>>following:
>>
>> Job ID: 9431211
>> Cluster: curta
>> User/Group: build/staff
>> State: COMPLETED (exit code 0)
>> Nodes: 1
>> Cores per node: 8
>> CPU Utilized: 01:30:53
>> CPU Efficiency: 83.63% of 01:48:40 core-walltime
>> Job Wall-clock time: 00:13:35
>> Memory Utilized: 6.45 GB
>> Memory Efficiency: 80.68% of 8.00 GB
>>
>>I want to parse this and am using subprocess.Popen and accessing the
>>contents via Popen.stdout. However, for testing purposes I want to save
>>various possible outputs of the command as text files and use those as
>>inputs.
>
> What do you want to test? the parsing? the "popen" interaction?
> You can separately test both tasks (I, at your place, would do this).

I just want to test the parsing.

> For the parsing test, it is not relevant that the actual text
> comes from an external process. You can directly read it from a file
> or have it in your text.

As mentioned in the original post, for the tests I indeed want to read
the input from files.

> In my view, you do not need a test for the `Popen` interaction:
> if it works once, it will work always.

Sorry if I was unclear but my question is:

Given that the return value from Popen is a Popen object and given that
the return value from reading a file is a single string or maybe a list
of strings, what should the common format for the argument which is
passed to the actual parsing function be?

Cheers,

Loris
--
This signature is currently under construction.

Loris Bennett wrote at 2022-3-11 07:40 +0100:
> ... I want to test the parsing ...
>Sorry if I was unclear but my question is:
>
>Given that the return value from Popen is a Popen object and given that
>the return value from reading a file is a single string or maybe a list
>of strings, what should the common format for the argument which is
>passed to the actual parsing function be?

What methods (of its input argument) does the parsing use?
If it uses `Popen` methods, then you pass a `POpen` object;
if it uses only (typical) file methods, then you pass a file object;
if it assumes its input to be a (line) interator, you pass
a (line) iterator (such as a "file" object).

I would design the parsing that it makes as few assumptions
about its input as possible -- to ease testing
and increase the chance for reuse.

That said, I would not design it to work with `Popen` objects
but likely to have a line iterator as input.

Op 10/03/2022 om 13:16 schreef Loris Bennett:
> Hi,
>
> I have a command which produces output like the
> following:
>
> Job ID: 9431211
> Cluster: curta
> User/Group: build/staff
> State: COMPLETED (exit code 0)
> Nodes: 1
> Cores per node: 8
> CPU Utilized: 01:30:53
> CPU Efficiency: 83.63% of 01:48:40 core-walltime
> Job Wall-clock time: 00:13:35
> Memory Utilized: 6.45 GB
> Memory Efficiency: 80.68% of 8.00 GB
>
> I want to parse this and am using subprocess.Popen and accessing the
> contents via Popen.stdout. However, for testing purposes I want to save
> various possible outputs of the command as text files and use those as
> inputs.
>
> What format should I use to pass data to the actual parsing function?
>
Is this a command you run, produces that output, and then stops (as
opposed to a long-running program that from time to time generates a
bunch of output)?
Because in that case I would use subprocess.run() with
capture_output=True instead of subprocess.Popen(). subprocess.run()
returns a CompletedProcess instance wich has stdout and stderr members
that contain the captured output as byte sequences or strings, depending
on the parameters you passed.

So in your case I would simply read the content of each text file as a
whole into a string, and use subprocess.run() to get the command's
output also as a string. Then you can have a parse function that accepts
such strings, and works exactly the same for the text files as for the
command output. Your parse function can then use splitlines() to access
the lines individually. The data size is very small so it's not a
problem to have it all in memory at the same time (i.e. no need to worry
about trying to stream it instead).

--
"Don't Panic."
-- Douglas Adams, The Hitchhiker's Guide to the Galaxy

Op 11/03/2022 om 10:11 schreef Roel Schroeven:
> Op 10/03/2022 om 13:16 schreef Loris Bennett:
>> Hi,
>>
>> I have a command which produces output like the
>> following:
>>
>>    Job ID: 9431211
>>    Cluster: curta
>>    User/Group: build/staff
>>    State: COMPLETED (exit code 0)
>>    Nodes: 1
>>    Cores per node: 8
>>    CPU Utilized: 01:30:53
>>    CPU Efficiency: 83.63% of 01:48:40 core-walltime
>>    Job Wall-clock time: 00:13:35
>>    Memory Utilized: 6.45 GB
>>    Memory Efficiency: 80.68% of 8.00 GB
>>
>> I want to parse this and am using subprocess.Popen and accessing the
>> contents via Popen.stdout.  However, for testing purposes I want to save
>> various possible outputs of the command as text files and use those as
>> inputs.
>>
>> What format should I use to pass data to the actual parsing function?
>>
> Is this a command you run, produces that output, and then stops (as
> opposed to a long-running program that from time to time generates a
> bunch of output)?
> Because in that case I would use subprocess.run() with
> capture_output=True instead of subprocess.Popen(). subprocess.run()
> returns a CompletedProcess instance wich has stdout and stderr members
> that contain the captured output as byte sequences or strings,
> depending on the parameters you passed.
>
> So in your case I would simply read the content of each text file as a
> whole into a string, and use subprocess.run() to get the command's
> output also as a string. Then you can have a parse function that
> accepts such strings, and works exactly the same for the text files as
> for the command output. Your parse function can then use splitlines()
> to access the lines individually. The data size is very small so it's
> not a problem to have it all in memory at the same time (i.e. no need
> to worry about trying to stream it instead).
>
Very simple example:

    import subprocess
    from pprint import pprint

    def parse(state_data):
        lines = state_data.splitlines(keepends=False)
        state_dict = {}
        for line in lines:
            key, value = line.split(': ')
            state_dict[key] = value
        return state_dict

    def read_from_command():
        return subprocess.run(['./jobstate'], capture_output=True,
check=True, encoding='UTF-8').stdout

    def read_from_file(fn):
        with open(fn, 'rt', encoding='UTF-8') as f:
            return f.read()

    pprint(parse(read_from_command()))
    pprint(parse(read_from_file('jobfile')))

--
"Iceland is the place you go to remind yourself that planet Earth is a
machine... and that all organic life that has ever existed amounts to a greasy
film that has survived on the exterior of that machine thanks to furious
improvisation."
-- Sam Hughes, Ra

On Friday, 11 March 2022 at 15:01:43 UTC+5:30, Roel Schroeven wrote:
> Op 11/03/2022 om 10:11 schreef Roel Schroeven:
> > Op 10/03/2022 om 13:16 schreef Loris Bennett:
> >> Hi,
> >>
> >> I have a command which produces output like the
> >> following:
> >>
> >> Job ID: 9431211
> >> Cluster: curta
> >> User/Group: build/staff
> >> State: COMPLETED (exit code 0)
> >> Nodes: 1
> >> Cores per node: 8
> >> CPU Utilized: 01:30:53
> >> CPU Efficiency: 83.63% of 01:48:40 core-walltime
> >> Job Wall-clock time: 00:13:35
> >> Memory Utilized: 6.45 GB
> >> Memory Efficiency: 80.68% of 8.00 GB
> >>
> >> I want to parse this and am using subprocess.Popen and accessing the
> >> contents via Popen.stdout. However, for testing purposes I want to save
> >> various possible outputs of the command as text files and use those as
> >> inputs.
> >>
> >> What format should I use to pass data to the actual parsing function?
> >>
> > Is this a command you run, produces that output, and then stops (as
> > opposed to a long-running program that from time to time generates a
> > bunch of output)?
> > Because in that case I would use subprocess.run() with
> > capture_output=True instead of subprocess.Popen(). subprocess.run()
> > returns a CompletedProcess instance wich has stdout and stderr members
> > that contain the captured output as byte sequences or strings,
> > depending on the parameters you passed.
> >
> > So in your case I would simply read the content of each text file as a
> > whole into a string, and use subprocess.run() to get the command's
> > output also as a string. Then you can have a parse function that
> > accepts such strings, and works exactly the same for the text files as
> > for the command output. Your parse function can then use splitlines()
> > to access the lines individually. The data size is very small so it's
> > not a problem to have it all in memory at the same time (i.e. no need
> > to worry about trying to stream it instead).
> >
> Very simple example:
>
> import subprocess
> from pprint import pprint
>
> def parse(state_data):
> lines = state_data.splitlines(keepends=False)
> state_dict = {}
> for line in lines:
> key, value = line.split(': ')
> state_dict[key] = value
> return state_dict
>
> def read_from_command():
> return subprocess.run(['./jobstate'], capture_output=True,
> check=True, encoding='UTF-8').stdout
>
> def read_from_file(fn):
> with open(fn, 'rt', encoding='UTF-8') as f:
> return f.read()
>
> pprint(parse(read_from_command()))
> pprint(parse(read_from_file('jobfile')))
>
> --
> "Iceland is the place you go to remind yourself that planet Earth is a
> machine... and that all organic life that has ever existed amounts to a greasy
> film that has survived on the exterior of that machine thanks to furious
> improvisation."
> -- Sam Hughes, Ra

------------------------------------------------------------------------------------------------------------------------------------
Want to learn how self-driving cars work and how the future is being built today using AI?
Understand the tech behind the 'Tesla Auto Pilot'.

Get Enrolled Today - https://bit.ly/3hUIKuy

What you’ll learn in this session?
- What is an autopilot?
- What are the features of autopilot?
- What is computer vision?
- Computer vision applications
- What is YOLO(You only look once)?
- What is Deep Learning?
- Deep Learning applications

Get Practice Material - You will get Notebook to revise, practice, test yourself and interact with mentor.
Earn a Certificate upon completion of Course

Book your session today for How Tesla Self Driving Car Work
Last date is Mar 14, 2022 | 2:00 PM

Why not just have scripts that echo out the various sets of test
data you are interested in? That way, Popen would
always be your interface and you wouldn't have to
make two cases in the consumer script.

In other words, make program that outputs test
data just like your main data source program.
Then the consumer would only have to work in one way.

On 3/10/22 04:16, Loris Bennett wrote:
> Hi,
>
> I have a command which produces output like the
> following:
>
> Job ID: 9431211
> Cluster: curta
> User/Group: build/staff
> State: COMPLETED (exit code 0)
> Nodes: 1
> Cores per node: 8
> CPU Utilized: 01:30:53
> CPU Efficiency: 83.63% of 01:48:40 core-walltime
> Job Wall-clock time: 00:13:35
> Memory Utilized: 6.45 GB
> Memory Efficiency: 80.68% of 8.00 GB
>
> I want to parse this and am using subprocess.Popen and accessing the
> contents via Popen.stdout. However, for testing purposes I want to save
> various possible outputs of the command as text files and use those as
> inputs.
>
> What format should I use to pass data to the actual parsing function?
>
> I could in both production and test convert the entire input to a string
> and pass the string to the parsing method.
>
> However, I could use something like
>
> test_input_01 = subprocess.Popen(
> ["cat test_input_01.txt"],
> stdout=subprocess.PIPE,
> )
>
> for the test input and then pass a Popen object to the parsing function.
>
> Any comments on these alternative or suggestions for doing something
> completely different?
>
> Cheers,
>
> Loris
>