On 31.01.2024 18:20, Scott Lurndal wrote:
> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>> On 31.01.2024 17:18, Michael S wrote:
>>> On Wed, 31 Jan 2024 17:05:23 +0100
>>> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
>
>> (See for example 'man ksh' Section "Input/Output". But careful; ksh
>> has additional non-standard additions. So a peek into the POSIX docs
>> might serve you better.)
>
> FWIW, the POSIX shell language was based on a subset of ksh88.

Yes. POSIX was based pretty much on ksh88. The differences are not
very large. If you're interested to know about the few differences
have a look into the book of Bolsky and Korn (it's in the appendix).

For more than three decades, though, there's ksh93 around. This is
what we typically mean when we speak about ksh. (But careful, since
the first version of ksh93 there had been a lot changes until now.)

Quite some time passed, though, until the commercial Unixes changed
their default ksh to ksh93. (I haven't any more access to commercial
Unixes, so I cannot tell about the current state.) See what you get.

On Linux you got only a ksh clone, first basically based on ksh88
(but I anyway never used it but downloaded the original AT&T ksh93).
Meanwhile, I think, you should get a ksh93 on Linux per default.
But I suggest to use the ksh93u+m (the version maintained by Martijn
Dekker), instead; it has a lot errors fixed.

A lot has changed between ksh88 and ksh93 already, and the current
ksh93 version developed even further. (Its man page shows details.)

Janis

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>On 31.01.2024 18:20, Scott Lurndal wrote:

<snip commentary about POSIX adopting most of ksh88>

>On Linux you got only a ksh clone, first basically based on ksh88
>(but I anyway never used it but downloaded the original AT&T ksh93).
>Meanwhile, I think, you should get a ksh93 on Linux per default.
>But I suggest to use the ksh93u+m (the version maintained by Martijn
>Dekker), instead; it has a lot errors fixed.

The AT&T ksh93 has been distributed for a long time. My 2012
install of Fedora Core 20, OSX, and my recent Ubuntu install use:

$ Version AJM 93u+ 2012-08-01

It's been decades since pdksh/mksh were defaults.

On 31/01/2024 13:47, Janis Papanagnou wrote:
> On 30.01.2024 20:39, Richard Harnden wrote:
>>
>> Nobody uses printf to output binary data. fwrite(3) would be common, as
>> would write(2).
>
> Right. I'm using the OS'es write(2), but also printf with ANSI escapes,
> e.g. sprintf (buf, "\033[%d;%dH", ...

I meant 'binary' as in has \0s

It seems to work fine with ESC's and utf8 (and i abuse it thus often)
.... but, from what James said, that is not actually guarenteed.

>
>>
>> Maybe you could use printf("%c%c%c" ... but it'd be beyond tedious.
>
> Since I recall to have used it in some thread I want to clarify that
> it was just meant as an example countering an incorrect argument of
> "not being able to output binary data on stdout", or some such.
>
> Janis
>

Richard Harnden <richard.nospam@gmail.invalid> writes:
> On 31/01/2024 13:47, Janis Papanagnou wrote:
>> On 30.01.2024 20:39, Richard Harnden wrote:
>>>
>>> Nobody uses printf to output binary data. fwrite(3) would be common, as
>>> would write(2).
>> Right. I'm using the OS'es write(2), but also printf with ANSI
>> escapes,
>> e.g. sprintf (buf, "\033[%d;%dH", ...
>
> I meant 'binary' as in has \0s

I don't think that's what "binary" means.

Loosely, "text" consists only of printable characters and a selected few
control characters like tabs and newlines, while "binary" can
potentially have any of 256 byte values (strictly speaking,
2**CHAR_BIT).

Terminal control sequences (almost always based on VT100 these days) are
typically not printable, but tend to avoid null characters, which means
you can very probably use printf to print them (assuming you're on a
POSIX-like system).

[...]

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

In article <upe60i$1latn$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
>[snip]
>On Linux you got only a ksh clone, first basically based on ksh88
>(but I anyway never used it but downloaded the original AT&T ksh93).
>Meanwhile, I think, you should get a ksh93 on Linux per default.
>[snip]

I'm not sure I understand this. Linux is just the kernel; to
get a useful operating system, you need a number of other
components such as utilities, system libraries, a load, and
other tools. A number of such distributions are based on GNU
tools; at least one is based on BSD tools. So writing, "On
Linux you got only a ksh clone..." doesn't make much sense to me
and seems like a category error.

Certainly, one can install `ksh93` in a Linux distribution.

- Dan C.

On Wed, 31 Jan 2024 20:16:12 -0000 (UTC)
cross@spitfire.i.gajendra.net (Dan Cross) wrote:

> In article <upe60i$1latn$1@dont-email.me>,
> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
> >[snip]
> >On Linux you got only a ksh clone, first basically based on ksh88
> >(but I anyway never used it but downloaded the original AT&T ksh93).
> >Meanwhile, I think, you should get a ksh93 on Linux per default.
> >[snip]
>
> I'm not sure I understand this. Linux is just the kernel; to
> get a useful operating system, you need a number of other
> components such as utilities, system libraries, a load, and
> other tools. A number of such distributions are based on GNU
> tools; at least one is based on BSD tools. So writing, "On
> Linux you got only a ksh clone..." doesn't make much sense to me
> and seems like a category error.
>
> Certainly, one can install `ksh93` in a Linux distribution.
>
> - Dan C.
>

Come on, in informal everyday's language the word Linux is very often
used as a shortcut for Gnu/Linux.

On 2024-01-31, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
> Terminal control sequences (almost always based on VT100 these days) are
> typically not printable, but tend to avoid null characters, which means
> you can very probably use printf to print them (assuming you're on a
> POSIX-like system).

They use text. For instance, a cursor position is both accepted and
reported in a decimal format like 13;17. All the commands and
delimiting characters are textual, except for part of the CSI (control
sequence introducer). The 7 bit CSI uses two characters, ESC and [.
Except for that one ESC, everything is printable.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

Michael S <already5chosen@yahoo.com> writes:
>On Wed, 31 Jan 2024 20:16:12 -0000 (UTC)
>cross@spitfire.i.gajendra.net (Dan Cross) wrote:
>
>> In article <upe60i$1latn$1@dont-email.me>,
>> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
>> >[snip]
>> >On Linux you got only a ksh clone, first basically based on ksh88
>> >(but I anyway never used it but downloaded the original AT&T ksh93).
>> >Meanwhile, I think, you should get a ksh93 on Linux per default.
>> >[snip]
>>
>> I'm not sure I understand this. Linux is just the kernel; to
>> get a useful operating system, you need a number of other
>> components such as utilities, system libraries, a load, and
>> other tools. A number of such distributions are based on GNU
>> tools; at least one is based on BSD tools. So writing, "On
>> Linux you got only a ksh clone..." doesn't make much sense to me
>> and seems like a category error.
>>
>> Certainly, one can install `ksh93` in a Linux distribution.
>>
>> - Dan C.
>>
>
>Come on, in informal everyday's language the word Linux is very often
>used as a shortcut for Gnu/Linux.
>

Other than providing a compiler and C library, GNU has nothing to
do with linux. There are other compilers, and other C libraries.

On Wed, 31 Jan 2024 20:50:55 GMT
scott@slp53.sl.home (Scott Lurndal) wrote:

> Michael S <already5chosen@yahoo.com> writes:
> >On Wed, 31 Jan 2024 20:16:12 -0000 (UTC)
> >cross@spitfire.i.gajendra.net (Dan Cross) wrote:
> >
> >> In article <upe60i$1latn$1@dont-email.me>,
> >> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
> >> >[snip]
> >> >On Linux you got only a ksh clone, first basically based on ksh88
> >> >(but I anyway never used it but downloaded the original AT&T
> >> >ksh93). Meanwhile, I think, you should get a ksh93 on Linux per
> >> >default. [snip]
> >>
> >> I'm not sure I understand this. Linux is just the kernel; to
> >> get a useful operating system, you need a number of other
> >> components such as utilities, system libraries, a load, and
> >> other tools. A number of such distributions are based on GNU
> >> tools; at least one is based on BSD tools. So writing, "On
> >> Linux you got only a ksh clone..." doesn't make much sense to me
> >> and seems like a category error.
> >>
> >> Certainly, one can install `ksh93` in a Linux distribution.
> >>
> >> - Dan C.
> >>
> >
> >Come on, in informal everyday's language the word Linux is very often
> >used as a shortcut for Gnu/Linux.
> >
>
> Other than providing a compiler and C library, GNU has nothing to
> do with linux. There are other compilers, and other C libraries.

https://en.wikipedia.org/wiki/GNU/Linux_naming_controversy

In article <20240131230048.0000513c@yahoo.com>,
Michael S <already5chosen@yahoo.com> wrote:
>On Wed, 31 Jan 2024 20:50:55 GMT
>scott@slp53.sl.home (Scott Lurndal) wrote:
>
>> Michael S <already5chosen@yahoo.com> writes:
>> >On Wed, 31 Jan 2024 20:16:12 -0000 (UTC)
>> >cross@spitfire.i.gajendra.net (Dan Cross) wrote:
>> >
>> >> In article <upe60i$1latn$1@dont-email.me>,
>> >> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
>> >> >[snip]
>> >> >On Linux you got only a ksh clone, first basically based on ksh88
>> >> >(but I anyway never used it but downloaded the original AT&T
>> >> >ksh93). Meanwhile, I think, you should get a ksh93 on Linux per
>> >> >default. [snip]
>> >>
>> >> I'm not sure I understand this. Linux is just the kernel; to
>> >> get a useful operating system, you need a number of other
>> >> components such as utilities, system libraries, a load, and
>> >> other tools. A number of such distributions are based on GNU
>> >> tools; at least one is based on BSD tools. So writing, "On
>> >> Linux you got only a ksh clone..." doesn't make much sense to me
>> >> and seems like a category error.
>> >>
>> >> Certainly, one can install `ksh93` in a Linux distribution.
>> >
>> >Come on, in informal everyday's language the word Linux is very often
>> >used as a shortcut for Gnu/Linux.
>>
>> Other than providing a compiler and C library, GNU has nothing to
>> do with linux. There are other compilers, and other C libraries.
>
>https://en.wikipedia.org/wiki/GNU/Linux_naming_controversy

I really don't care what Stallman thinks about the naming. DMR
is said to have once quipped in the Unix room, when Stallman was
complaining about the Plan 9 license, "Stallman won't be happy
until we call it Plan-Guh-Nine."

That said, when one's talking about whether ksh93 is shipped
with "Linux" or not, it's surely important to talk about what
distribution one is referring to!

- Dan C.

Kaz Kylheku <433-929-6894@kylheku.com> writes:
> On 2024-01-31, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
>> Terminal control sequences (almost always based on VT100 these days) are
>> typically not printable, but tend to avoid null characters, which means
>> you can very probably use printf to print them (assuming you're on a
>> POSIX-like system).
>
> They use text. For instance, a cursor position is both accepted and
> reported in a decimal format like 13;17. All the commands and
> delimiting characters are textual, except for part of the CSI (control
> sequence introducer). The 7 bit CSI uses two characters, ESC and [.
> Except for that one ESC, everything is printable.

I'd describe "printable except for ESC" as binary. And some sequences
use other non-printable characters like ASCII BEL (Ctl-G) (perhaps not
VT100 standard, but for example commands to change fonts and colors for
xterm).

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

On 31/01/2024 20:14, Keith Thompson wrote:
> Richard Harnden <richard.nospam@gmail.invalid> writes:
>> On 31/01/2024 13:47, Janis Papanagnou wrote:
>>> On 30.01.2024 20:39, Richard Harnden wrote:
>>>>
>>>> Nobody uses printf to output binary data. fwrite(3) would be common, as
>>>> would write(2).
>>> Right. I'm using the OS'es write(2), but also printf with ANSI
>>> escapes,
>>> e.g. sprintf (buf, "\033[%d;%dH", ...
>>
>> I meant 'binary' as in has \0s
>
> I don't think that's what "binary" means.
>
> Loosely, "text" consists only of printable characters and a selected few
> control characters like tabs and newlines, while "binary" can
> potentially have any of 256 byte values (strictly speaking,
> 2**CHAR_BIT).
>
> Terminal control sequences (almost always based on VT100 these days) are
> typically not printable, but tend to avoid null characters, which means
> you can very probably use printf to print them (assuming you're on a
> POSIX-like system).
>
> [...]
>
In ASCII, 0 means NUL, or "ignore". So an ASCII sequence may contain any
number of embedded zero bytes, which the receiver ignores. That's
because for technical reasons some communications channels have to send
data every cycle, and if there is no data, they will send a signal
indistinguishable from all bits zero.

--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
> On 31/01/2024 20:14, Keith Thompson wrote:
[...]
>> Terminal control sequences (almost always based on VT100 these days)
>> are typically not printable, but tend to avoid null characters, which
>> means you can very probably use printf to print them (assuming you're
>> on a POSIX-like system).
>> [...]
>>
> In ASCII, 0 means NUL, or "ignore". So an ASCII sequence may contain
> any number of embedded zero bytes, which the receiver ignores. That's
> because for technical reasons some communications channels have to
> send data every cycle, and if there is no data, they will send a
> signal indistinguishable from all bits zero.

Not particularly relevant. A quick experiment with xterm indicates that
embedding null bytes in a control sequence prevents it from being
recognized. There may be some standards that require embedded zero
bytes to be ignored, but xterm doesn't any such standard. Similarly, if
you embed null bytes in text written to a file, the result is corrupted
text file.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:

> On 30/01/2024 07:27, Tim Rentsch wrote:
>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>
>>> On 29/01/2024 20:10, Tim Rentsch wrote:
>>>
>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>>
>>>> [...]
>>>>
>>>>> I've never used standard output for binary data.
>>>>> [...] it strikes me as a poor design decision.
>>>>
>>>> How so?
>>>
>>> Because the output can't be inspected by humans, and because it might
>>> have unusual effects if passed though systems designed to handle
>>> human-readable text.

Maybe you are not used to a system where it's trivial to inspect such
data. When "some_prog" produces data that are not compatible with the
current terminal settings, "some_prog | hd" shows a hex dump instead.
The need to do this does not make "some_prog" poorly designed. It may
simply mean that the output is /intended/ for further processing.

> For instance in some systems designed to receive
>>> ASCII text, there is no distinction between the nul byte and "waiting
>>> for next data byte". Obviously this will cause difficuties if the data
>>> is binary.
>>> Also many binary formats can't easily be extended, so you can pass one
>>> image and that's all. While it is possible to devise a text format
>>> which is similar, in practice text formats usually have enough
>>> redundancy to be easily extended.
>>>
>>> So it's harder to correct errors, more prone to errors, and harder to
>>> extend.
>> Your reasoning is all gobbledygook. Your comments reflect only
>> limitations in your thinking, not any essential truth about using
>> standard out for binary data.
>>
> I must admit that it's nothing I have ever done or considered doing.
>
> However standard output is designed for text and not binary ouput.

What is your evidence? stdout was just designed for output (as far as I
can tell) and, anyway, what is the distinction you are making between
binary and text? iconv --from ACSII --to EBCDIC-UK will produce
something that is "logically" text on stdout, but it might look like
binary to you.

An example where it's really useful not to care: I have a suite of tools
for doing toy cryptanalysis. Some apply various transformations and/or
filters to byte streams and others collect and output (on stderr)
various statistics. Plugging them together in various pipelines is very
handy when investigating an encrypted text. The output is almost always
"binary" in the sense that there would be not point in looking at on a
terminal.

According to you, these tools are poorly designed. I don't think so.
How would you design them? Endless input and output file names to be
juggled and tidied up afterwards?

> That you couldn't actually mount a defence of your position whilst I could
> also strongly implies that I am right.

There is also the possibility that readers are just exhausted by the
fire-hose of bizarre, unsubstantiated options coming from you. I've
been tied up with personal matters recently, but I've seen many posts
from you where I have been tempted to make a one-word reply like "no",
"nonsense" or "daft", but I see that you would have taken these to be
string confirmation that you are right!

--
Ben.

On 31/01/2024 16:33, Keith Thompson wrote:
> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>> On 31/01/2024 07:18, Tim Rentsch wrote:
> [...]
>>> It is FAR more cumbersome to accomplish what this command
>>> is doing without sending binary data through standard out.
>>> Anyone who doesn't understand this doesn't understand Unix.
>>>
>> Yes. I don't do that sort of thing.
>
> You don't. Others do. What was your point again?
>
There's kind of an implication that it's something I ought to be doing.

--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On 31/01/2024 23:36, Ben Bacarisse wrote:
> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>
>> On 30/01/2024 07:27, Tim Rentsch wrote:
>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>
>>>> On 29/01/2024 20:10, Tim Rentsch wrote:
>>>>
>>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>>>
>>>>> [...]
>>>>>
>>>>>> I've never used standard output for binary data.
>>>>>> [...] it strikes me as a poor design decision.
>>>>>
>>>>> How so?
>>>>
>>>> Because the output can't be inspected by humans, and because it might
>>>> have unusual effects if passed though systems designed to handle
>>>> human-readable text.
>
> Maybe you are not used to a system where it's trivial to inspect such
> data. When "some_prog" produces data that are not compatible with the
> current terminal settings, "some_prog | hd" shows a hex dump instead.
> The need to do this does not make "some_prog" poorly designed. It may
> simply mean that the output is /intended/ for further processing.
>
>> For instance in some systems designed to receive
>>>> ASCII text, there is no distinction between the nul byte and "waiting
>>>> for next data byte". Obviously this will cause difficuties if the data
>>>> is binary.
>>>> Also many binary formats can't easily be extended, so you can pass one
>>>> image and that's all. While it is possible to devise a text format
>>>> which is similar, in practice text formats usually have enough
>>>> redundancy to be easily extended.
>>>>
>>>> So it's harder to correct errors, more prone to errors, and harder to
>>>> extend.
>>> Your reasoning is all gobbledygook. Your comments reflect only
>>> limitations in your thinking, not any essential truth about using
>>> standard out for binary data.
>>>
>> I must admit that it's nothing I have ever done or considered doing.
>>
>> However standard output is designed for text and not binary ouput.
>
> What is your evidence? stdout was just designed for output (as far as I
> can tell) and, anyway, what is the distinction you are making between
> binary and text? iconv --from ACSII --to EBCDIC-UK will produce
> something that is "logically" text on stdout, but it might look like
> binary to you.
>
> An example where it's really useful not to care: I have a suite of tools
> for doing toy cryptanalysis. Some apply various transformations and/or
> filters to byte streams and others collect and output (on stderr)
> various statistics. Plugging them together in various pipelines is very
> handy when investigating an encrypted text. The output is almost always
> "binary" in the sense that there would be not point in looking at on a
> terminal.
>
> According to you, these tools are poorly designed. I don't think so.
> How would you design them? Endless input and output file names to be
> juggled and tidied up afterwards?

I think they're poorly designed too.

From the POV of interactive console programs, they /are/ poor. But the
mistake is thinking that they are actual programs or commands, when
really they are just filters. They are not designed to be standalone
commands.

Even 'cat', if I type it by itself, just sits there. (I wonder what use
it has in a sequence like ... | cat | ...; what does it add to the data?)

AFAICS, this stuff mainly works inside scripts. Or do people here spend
all day manually piping stuff between programs?

As for alternatives, I don't know. There are any number of ways this
could be done. But if everyone has become inured to this piping
business, they will not be receptive to anything different.

Here however are some ideas:

* Have versions of these tools for use as filters with no UI, just
a default input and output, and versions for interactive use with
helpful prompts. Or even just a sensibly named output! Instead of
every program writing a.out.

* Have a concept of a current block of data, analogous to a clipboard.
Then separate commands can load data, sort it, count it, display it,
write it etc, with no need for intermediate named files.

But I'd be happier if this was all contained within a separate
application from an OS shell program.

Such things are ludicrously easy to write. I just did one in 15 minutes
and 50 lines of code, but set up for line-based text files. This is it
in action; the test input contains 4 lines "one two three four":

Type exit quit or q to finish

> load fred
Data loaded

> lc
4 lines

> list
1 one
2 two
3 three
4 four

> rev
Reversed

> list
1 four
2 three
3 two
4 one

> sort
Sorted

> upper

> list
1 FOUR
2 ONE
3 THREE
4 TWO

> save bill
Written to bill

> q

There are any number of transformations that can be applied without
needed to name and write intermediate files. The commands can even just
invoke the same filters I mentioned, but now within a friendlier UI.

A more sophisticated version can permanently show the data without my
having to type 'list'.

bart <bc@freeuk.com> writes:
>On 31/01/2024 23:36, Ben Bacarisse wrote:
>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:

>> According to you, these tools are poorly designed. I don't think so.
>> How would you design them? Endless input and output file names to be
>> juggled and tidied up afterwards?
>
>I think they're poorly designed too.

Of course you do. They're not bart programs.

>
> From the POV of interactive console programs, they /are/ poor.

You don't provide any reason why - do elucidate!

>But the
>mistake is thinking that they are actual programs or commands, when
>really they are just filters. They are not designed to be standalone
>commands.
>
>Even 'cat', if I type it by itself, just sits there.

It is _designed_ to do that. If that's not the behavior you
want, don't use that command.

If you read the command documentation, you would understand
what the behavior of the command would be in that context
and realize that it would be pointless.

Although, it is actually useful, since it echos back what
you type, for e.g. checking serial lines, terminal emulators,
etc.

>(I wonder what use
>it has in a sequence like ... | cat | ...; what does it add to the data?)

The manual page provides the answer to that question, and it
should be obvious even to the meanest programmer.

>
>AFAICS, this stuff mainly works inside scripts. Or do people here spend
>all day manually piping stuff between programs?

Yes and Yes.

>
>As for alternatives, I don't know.

Indeed.

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
> On 31/01/2024 16:33, Keith Thompson wrote:
>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>> On 31/01/2024 07:18, Tim Rentsch wrote:
>> [...]
>>>> It is FAR more cumbersome to accomplish what this command
>>>> is doing without sending binary data through standard out.
>>>> Anyone who doesn't understand this doesn't understand Unix.
>>>>
>>> Yes. I don't do that sort of thing.
>> You don't. Others do. What was your point again?
>>
> There's kind of an implication that it's something I ought to be doing.

Nope.

You seemed to be implying that it's something nobody else ought to be
doing. If that's not what you meant, we can just drop this.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>On 31/01/2024 16:33, Keith Thompson wrote:
>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>> On 31/01/2024 07:18, Tim Rentsch wrote:
>> [...]
>>>> It is FAR more cumbersome to accomplish what this command
>>>> is doing without sending binary data through standard out.
>>>> Anyone who doesn't understand this doesn't understand Unix.
>>>>
>>> Yes. I don't do that sort of thing.
>>
>> You don't. Others do. What was your point again?
>>
>There's kind of an implication that it's something I ought to be doing.

No, we're saying that what you do or don't do is irrelevent. Nobody is
forcing you to do anything other than acknowledge that it is a useful
capability that others use regularly.

On 01/02/2024 00:49, Keith Thompson wrote:
> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>> On 31/01/2024 16:33, Keith Thompson wrote:
>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>> On 31/01/2024 07:18, Tim Rentsch wrote:
>>> [...]
>>>>> It is FAR more cumbersome to accomplish what this command
>>>>> is doing without sending binary data through standard out.
>>>>> Anyone who doesn't understand this doesn't understand Unix.
>>>>>
>>>> Yes. I don't do that sort of thing.
>>> You don't. Others do. What was your point again?
>>>
>> There's kind of an implication that it's something I ought to be doing.
>
> Nope.
>
> You seemed to be implying that it's something nobody else ought to be
> doing. If that's not what you meant, we can just drop this.
>
No. It's something which has no value in and of itself, so you should
only do it or learn much about it if faced with a practical need to set
up or debug and troubleshoot such systems. Unlike some other types of
computer programming. But I don't want to imply that there is never a
need for it or that you're a lesser person if you do have a need for it.
But it's nothing to be proud of either. It's like knowing how to use MS
Word if you are doing a PhD. Often you would and we even had formal
courses run by the university on Word for long documents. But it doesn't
make you a scientist.

But as you say, I think I've made the point. And you weren't the person
to imply that I ought to know more about Unix pipelines than I do.
--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On 2024-01-31, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
> Kaz Kylheku <433-929-6894@kylheku.com> writes:
>> On 2024-01-31, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
>>> Terminal control sequences (almost always based on VT100 these days) are
>>> typically not printable, but tend to avoid null characters, which means
>>> you can very probably use printf to print them (assuming you're on a
>>> POSIX-like system).
>>
>> They use text. For instance, a cursor position is both accepted and
>> reported in a decimal format like 13;17. All the commands and
>> delimiting characters are textual, except for part of the CSI (control
>> sequence introducer). The 7 bit CSI uses two characters, ESC and [.
>> Except for that one ESC, everything is printable.
>
> I'd describe "printable except for ESC" as binary. And some sequences
> use other non-printable characters like ASCII BEL (Ctl-G) (perhaps not
> VT100 standard, but for example commands to change fonts and colors for
> xterm).

But ESC is related text; it's a character described in ASCII used for
signaling in the middle of text, which is what it's doing here.

I'd say that if the terminal control sequences contained features like:

- records delimited by length (number of bytes) rather than any stable
delimiter, with an emphasis on either fixed length data structures,
or else length + data encodings for variable ones.

- numbers encoded in binary: for instance screen coordinates (5,257)
being encoded as bytes 05 00 01 01 (little endian, 16 bit)
rathe than the text "5;257"

.... then that would be "binary data".

In the terminal sequences, when the byte 27/0x1B occurs, it is
always being ESC.

In binary data, when such a byte value occurs, it might be anything,
with any meaning. It could be part of a number, e.g. lower byte of
0x1D1B. It could be some flags meaning 00011101, with various meanings.

Binary data has the characteristic that it's not concerned with the
character interpretation of many of the bytes, other than ones which are
embedded text fields. Thus in many parts of the binary data, any
character can potentially occur by chance. An 8 bit flags field can
reproduce any character, given the right combination of the flag values.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

On 01/02/2024 00:47, Scott Lurndal wrote:
> bart <bc@freeuk.com> writes:
>> On 31/01/2024 23:36, Ben Bacarisse wrote:
>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>
>>> According to you, these tools are poorly designed. I don't think so.
>>> How would you design them? Endless input and output file names to be
>>> juggled and tidied up afterwards?
>>
>> I think they're poorly designed too.
>
> Of course you do. They're not bart programs.
>
>>
>> From the POV of interactive console programs, they /are/ poor.
>
> You don't provide any reason why - do elucidate!

They only do one thing, like you can't first do A, then B. They don't
give any prompts. They often apparently do nothing (so you can't tell if
they're busy, waiting for input, or hanging). There is no dialog.

But I'm sure this has been mentioned a few times.

NONE of my console apps are that poor. But then I've written programs
with proper CLIs and GUIs that had to used by non-programmers.

>> AFAICS, this stuff mainly works inside scripts. Or do people here spend
>> all day manually piping stuff between programs?
>
> Yes and Yes.

OK. I spent most creative time on my machine doing actual coding.

Kaz Kylheku <433-929-6894@kylheku.com> writes:
>On 2024-01-31, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
>> Kaz Kylheku <433-929-6894@kylheku.com> writes:
>>> On 2024-01-31, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
>>>> Terminal control sequences (almost always based on VT100 these days) are
>>>> typically not printable, but tend to avoid null characters, which means
>>>> you can very probably use printf to print them (assuming you're on a
>>>> POSIX-like system).
>>>
>>> They use text. For instance, a cursor position is both accepted and
>>> reported in a decimal format like 13;17. All the commands and
>>> delimiting characters are textual, except for part of the CSI (control
>>> sequence introducer). The 7 bit CSI uses two characters, ESC and [.
>>> Except for that one ESC, everything is printable.
>>
>> I'd describe "printable except for ESC" as binary. And some sequences
>> use other non-printable characters like ASCII BEL (Ctl-G) (perhaps not
>> VT100 standard, but for example commands to change fonts and colors for
>> xterm).
>
>But ESC is related text; it's a character described in ASCII used for
>signaling in the middle of text, which is what it's doing here.

So are most of the other ASCII codes less than 0x20. Including
file and record delimiters and shift-in/shift-out.

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
> On 01/02/2024 00:49, Keith Thompson wrote:
>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>> On 31/01/2024 16:33, Keith Thompson wrote:
>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>>> On 31/01/2024 07:18, Tim Rentsch wrote:
>>>> [...]
>>>>>> It is FAR more cumbersome to accomplish what this command
>>>>>> is doing without sending binary data through standard out.
>>>>>> Anyone who doesn't understand this doesn't understand Unix.
>>>>>>
>>>>> Yes. I don't do that sort of thing.
>>>> You don't. Others do. What was your point again?
>>>>
>>> There's kind of an implication that it's something I ought to be doing.
>> Nope.
>> You seemed to be implying that it's something nobody else ought to be
>> doing. If that's not what you meant, we can just drop this.
>>
> No. It's something which has no value in and of itself,

Demonstrably wrong.

> so you should
> only do it or learn much about it if faced with a practical need to
> set up or debug and troubleshoot such systems. Unlike some other types
> of computer programming. But I don't want to imply that there is never
> a need for it or that you're a lesser person if you do have a need for
> it. But it's nothing to be proud of either. It's like knowing how to
> use MS Word if you are doing a PhD. Often you would and we even had
> formal courses run by the university on Word for long documents. But
> it doesn't make you a scientist.
>
> But as you say, I think I've made the point. And you weren't the
> person to imply that I ought to know more about Unix pipelines than I
> do.

So your point is that you want to be condescending about something you
don't understand very well.

Got it.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

On 31/01/2024 23:36, Ben Bacarisse wrote:
> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>
>> On 30/01/2024 07:27, Tim Rentsch wrote:
>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>
>>>> On 29/01/2024 20:10, Tim Rentsch wrote:
>>>>
>>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>>>
>>>>> [...]
>>>>>
>>>>>> I've never used standard output for binary data.
>>>>>> [...] it strikes me as a poor design decision.
>>>>>
>>>>> How so?
>>>>
>>>> Because the output can't be inspected by humans, and because it might
>>>> have unusual effects if passed though systems designed to handle
>>>> human-readable text.
>
> Maybe you are not used to a system where it's trivial to inspect such
> data. When "some_prog" produces data that are not compatible with the
> current terminal settings, "some_prog | hd" shows a hex dump instead.
> The need to do this does not make "some_prog" poorly designed. It may
> simply mean that the output is /intended/ for further processing.
>
>> For instance in some systems designed to receive
>>>> ASCII text, there is no distinction between the nul byte and "waiting
>>>> for next data byte". Obviously this will cause difficuties if the data
>>>> is binary.
>>>> Also many binary formats can't easily be extended, so you can pass one
>>>> image and that's all. While it is possible to devise a text format
>>>> which is similar, in practice text formats usually have enough
>>>> redundancy to be easily extended.
>>>>
>>>> So it's harder to correct errors, more prone to errors, and harder to
>>>> extend.
>>> Your reasoning is all gobbledygook. Your comments reflect only
>>> limitations in your thinking, not any essential truth about using
>>> standard out for binary data.
>>>
>> I must admit that it's nothing I have ever done or considered doing.
>>
>> However standard output is designed for text and not binary ouput.
>
> What is your evidence? stdout was just designed for output (as far as I
> can tell) and, anyway, what is the distinction you are making between
> binary and text? iconv --from ACSII --to EBCDIC-UK will produce
> something that is "logically" text on stdout, but it might look like
> binary to you.
>
> An example where it's really useful not to care: I have a suite of tools
> for doing toy cryptanalysis. Some apply various transformations and/or
> filters to byte streams and others collect and output (on stderr)
> various statistics. Plugging them together in various pipelines is very
> handy when investigating an encrypted text. The output is almost always
> "binary" in the sense that there would be not point in looking at on a
> terminal.
>
> According to you, these tools are poorly designed. I don't think so.
> How would you design them? Endless input and output file names to be
> juggled and tidied up afterwards?
>
I'd write a monolithic program.
Load the encryoted text into memory, and then pass it to subroutines to
do the various analyses.
You can of course process it, and then pass the processed output to
other programs. And that does have a point if the program which is
acceoting the processed outout is doing something which has no necessary
connection to cryptanalysis. So for example a program to produce a pie
chart from a list of letter frequencies. But if it's transforming the
encrypted text in intricate and specialised ways, then analysing the
transformed text in other specialised and intricate ways, then firstly
you've probably introduced coupling and dependency between the two
programs, and secondly you're probably at some point going to want to
modify the second program in the pipeline to look at the raw data.

So whilst of course you can get this to work, it's not very robust. If
additionally the data is binary or text which can't be inspected by a
human, then you can't easily construct test inputs to get the individual
programs working and debugged independently of the others.

But you are right that creating temporary binary files isn't ideal
either. The files tend to hang about and then people wonder what they
mean and if they are an essential component of something. In reality
when getting a video game together, you need a lot of temporary binary
files - you can't have raw sources, pipelines, and a final executable.
And our Azure build pipeline works that way too. It doesn't pipe C
source to the compilers, it gathers the source, does the pre-processing,
and creates temporary files which the compilers are invoked upon. Then
it deletes them. It is indeed endless input and output file names to be
juggled and tidied up afterward.

--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm