On 31/01/2024 23:34, Keith Thompson wrote:
> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>> On 31/01/2024 20:14, Keith Thompson wrote:
> [...]
>>> Terminal control sequences (almost always based on VT100 these days)
>>> are typically not printable, but tend to avoid null characters, which
>>> means you can very probably use printf to print them (assuming you're
>>> on a POSIX-like system).
>>> [...]
>>>
>> In ASCII, 0 means NUL, or "ignore". So an ASCII sequence may contain
>> any number of embedded zero bytes, which the receiver ignores. That's
>> because for technical reasons some communications channels have to
>> send data every cycle, and if there is no data, they will send a
>> signal indistinguishable from all bits zero.
>
> Not particularly relevant. A quick experiment with xterm indicates that
> embedding null bytes in a control sequence prevents it from being
> recognized. There may be some standards that require embedded zero
> bytes to be ignored, but xterm doesn't any such standard. Similarly, if
> you embed null bytes in text written to a file, the result is corrupted
> text file.
>
The standard is ASCII. (American standard for computer information
interchange). Byte zero is NUL, which means "ignore".
--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On 01/02/2024 01:36, Keith Thompson wrote:
> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>> On 01/02/2024 00:49, Keith Thompson wrote:
>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>> On 31/01/2024 16:33, Keith Thompson wrote:
>>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>>>> On 31/01/2024 07:18, Tim Rentsch wrote:
>>>>> [...]
>>>>>>> It is FAR more cumbersome to accomplish what this command
>>>>>>> is doing without sending binary data through standard out.
>>>>>>> Anyone who doesn't understand this doesn't understand Unix.
>>>>>>>
>>>>>> Yes. I don't do that sort of thing.
>>>>> You don't. Others do. What was your point again?
>>>>>
>>>> There's kind of an implication that it's something I ought to be doing.
>>> Nope.
>>> You seemed to be implying that it's something nobody else ought to be
>>> doing. If that's not what you meant, we can just drop this.
>>>
>> No. It's something which has no value in and of itself,
>
> Demonstrably wrong.
>
You're not going to learn anything useful about anything by learning
abut Unix pipelines. Except how to set up a Unix pipeline itself.

That's what you don't seem to have taken on board. It's not
"condescending" to point that out. That's not true of many other topics
in programming.

--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
> On 31/01/2024 23:34, Keith Thompson wrote:
>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>> On 31/01/2024 20:14, Keith Thompson wrote:
>> [...]
>>>> Terminal control sequences (almost always based on VT100 these days)
>>>> are typically not printable, but tend to avoid null characters, which
>>>> means you can very probably use printf to print them (assuming you're
>>>> on a POSIX-like system).
>>>> [...]
>>>>
>>> In ASCII, 0 means NUL, or "ignore". So an ASCII sequence may contain
>>> any number of embedded zero bytes, which the receiver ignores. That's
>>> because for technical reasons some communications channels have to
>>> send data every cycle, and if there is no data, they will send a
>>> signal indistinguishable from all bits zero.
>> Not particularly relevant. A quick experiment with xterm indicates
>> that
>> embedding null bytes in a control sequence prevents it from being
>> recognized. There may be some standards that require embedded zero
>> bytes to be ignored, but xterm doesn't any such standard. Similarly, if
>> you embed null bytes in text written to a file, the result is corrupted
>> text file.
>>
> The standard is ASCII. (American standard for computer information
> interchange). Byte zero is NUL, which means "ignore".

"Ignore" is good advice.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
[...]
> You're not going to learn anything useful about anything by learning
> abut Unix pipelines. Except how to set up a Unix pipeline itself.
>
> That's what you don't seem to have taken on board. It's not
> "condescending" to point that out. That's not true of many other
> topics in programming.

Nonsense, and not worth my time to refute.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

On 31/01/2024 23:36, Ben Bacarisse wrote:
> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>
>> On 30/01/2024 07:27, Tim Rentsch wrote:
>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>
>>>> On 29/01/2024 20:10, Tim Rentsch wrote:
>>>>
>>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>>>
>>>>> [...]
>>>>>
>>>>>> I've never used standard output for binary data.
>>>>>> [...] it strikes me as a poor design decision.
>>>>>
>>>>> How so?
>>>>
>>>> Because the output can't be inspected by humans, and because it might
>>>> have unusual effects if passed though systems designed to handle
>>>> human-readable text.
>
> Maybe you are not used to a system where it's trivial to inspect such
> data. When "some_prog" produces data that are not compatible with the
> current terminal settings, "some_prog | hd" shows a hex dump instead.
> The need to do this does not make "some_prog" poorly designed. It may
> simply mean that the output is /intended/ for further processing.
>
Well almost by definition binary output is intended for further
processing. Binary audio files must ultimately be converted to analogue
if anyone is to listen to them, for example.
I had to check how to do a hex dump on the system I'm typing this on.
The name of the hex dumper is xxd instead of hd, but otherwise it works
the same way and will accept piped data. But the fact I had to look it
up tells you that I've never actually used it. The two problems with hex
dumps are that you've got to do mental arithmetic to convert 8 bit hex
values into 16 or 32 bit fields, and that once you get a variable length
field, it's virtually impossible to keep track of and match up the
following fields. So in reality what I do when troubleshooting binary
data is to write a scratch program, or, more often because the trouble
is in the existing parser, put diagnosics in an existing parser to print
out a few fields and inspect them that way. Of course to check that
audio or image data is right you have to listen to it or view it - you
can't tell from looking at the individual samples.

--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On 1/31/24 14:43, Richard Harnden wrote:
> On 31/01/2024 13:47, Janis Papanagnou wrote:
>> On 30.01.2024 20:39, Richard Harnden wrote:
>>>
>>> Nobody uses printf to output binary data. fwrite(3) would be common, as
>>> would write(2).
>>
>> Right. I'm using the OS'es write(2), but also printf with ANSI escapes,
>> e.g. sprintf (buf, "\033[%d;%dH", ...
>
> I meant 'binary' as in has \0s
>
> It seems to work fine with ESC's and utf8 (and i abuse it thus often)
> ... but, from what James said, that is not actually guarenteed.

You're right. I forgot to point it out, but it's clear that null
characters also invalidate that guarantee.

The simplest way in which that guarantee could fail would be that any of
the prohibited characters would simply be dropped, either while writing
in text mode or when reading them back in text mode. However, that would
be pretty arbitrary.

A more subtle alternative is that some or all of the prohibited
characters are used by the operating system to control how the other
contents of text files are interpreted when reading the file. Examples:
backspace characters could be removed, along with the immediately
preceding characters. Carriage return characters could be removed along
with the entire preceding line. Vertical tabs and form feeds could be
replaced with an appropriate number of newline characters. I don't know
if any real operating system does any of those things with text files,
but if it did, that would not prevent a fully conforming implementation
of C on that platform, thanks to the clause I cited.
However, I also know of one real-life example, though an obscure one:
text files are stored in fixed-size blocks, with each line starting with
a count of the blocks it occupies. Newline characters are converted in
spaces that pad out the end of the last block - the net result is that
spaces immediately preceding a newline would be indistinguishable from
the padding, and would therefore get dropped when reading the text file
back in. The existence of such systems is precisely why "no new-line
character is immediately preceded by space characters" is one of the
specifications for text files.

In that case, the I/O routines have two options: they can remove those
characters when writing the text to prevent them from being
misinterpreted, which would be how the text read in fails to match the
text that was written. The other alternative is let those characters
through, and let them be misinterpreted when read back, which could
produce arbitrarily bizarre consequences (NOT limited to the examples I
gave).

A similar fixed-size block scheme with null characters padding out the
end of the last block is the reason why the standard says that a binary
"stream may, however, have an implementation-defined number of null
characters appended to the end of the stream." when read back in.

On 1/31/24 15:14, Keith Thompson wrote:
> Richard Harnden <richard.nospam@gmail.invalid> writes:
>> On 31/01/2024 13:47, Janis Papanagnou wrote:
>>> On 30.01.2024 20:39, Richard Harnden wrote:
>>>>
>>>> Nobody uses printf to output binary data. fwrite(3) would be common, as
>>>> would write(2).
>>> Right. I'm using the OS'es write(2), but also printf with ANSI
>>> escapes,
>>> e.g. sprintf (buf, "\033[%d;%dH", ...
>>
>> I meant 'binary' as in has \0s
>
> I don't think that's what "binary" means.

The standard defines text and binary streams. I think it would be
reasonable to call a file binary if it violates any of the requirements
for a text stream to read back the same way it was written out. If so,
null and ESC characters both qualify.

On 31/01/2024 14:35, Janis Papanagnou wrote:
> On 31.01.2024 14:05, David Brown wrote:
>>
>> But it is correct that English has become the main language for
>> international communication, and is therefore critical for anything that
>> involves cross-border communication, or where there are significant
>> numbers of foreign workers. That includes academic work. Different
>> parts of Europe previously used German or Russian for this,
>
> Don't forget the importance of French! - The whole postal and
> telecommunication sectors were (and probably still are) massively
> influenced by France.

The French would never let the Anglophiles forget the importance of
French :-)

>
> (You're always writing so much text, so I'll skip it and avoid
> more comments.)
>
> Just two (unrelated) notes concerning statements I've seen
> somewhere in the thread (maybe here as well)...
>
> First; the EU publishes in all languages of the member states,
> for example. (There's no single lingua franca.)

Weirdly, while Norway is not in the EU but Sweden and Denmark are, they
publish (for some things at least) in Norwegian but not in Swedish or
Danish. The Danes and the Swedes don't like each other's languages, but
are happy enough with Norwegian so they use that to save a little time
and money.

>
> And the second note; we have to distinguish the language of the
> programming language's keywords, the comments in the source
> code, and the language(s) used for user-interaction.
>

Absolutely. That is a critical distinction.

> I don't know whether there's some native language that use
> non-English keywords, but I'd suppose so, since in the past
> I've seen some using preprocessors for a "native language"
> source code. So while not typical, probably a demand at some
> places. (Elsethread I mentioned the German TR440 commands,
> but a [primitive] command language, as opposed to, say, the
> Unix shell, I don't consider much as a language.)
>

Apparently there are a few - including, of course, one in French. But I
have not heard of any that are relevant in modern times.

> The comments' languages varies, in my experience. Sometimes
> there's coding standards (that demand the native language, or
> that demand English), sometimes it's not defined. Myself I'm
> reluctant to switch between languages and stay with English.
> But there were also other cases with longer descriptions on
> a conceptual basis; if you come from a native language's
> perspective it can be better to stay with the language of the
> specification instead of introducing sources of misunderstanding.
>
> The user interface, finally, is of course as specified, and can
> be anything, or even multi-lingual.
>

That's my experience too.

On 31/01/2024 19:35, Keith Thompson wrote:
> David Brown <david.brown@hesbynett.no> writes:
>> On 31/01/2024 15:46, Janis Papanagnou wrote:
>>> On 31.01.2024 15:09, David Brown wrote:
>>>>
>>>> I would expect that the majority of uses of "cat" are with just one
>>>> file,
>>> And of course just because of ignorance; the majority of (but not
>>> all)
>>> uses with just one file are UUOCs.
>>
>> I regularly see it as more symmetrical and clearer to push data left
>> to right. So I might write "cat infile | grep foo | sort > outfile".
>> Of course I could use "<" redirection, but somehow it seems more
>> natural to me to have this flow. I'll use "<" for simpler cases.
>>
>> But perhaps this is just my habit, and makes little sense to other people.
>
> You can also use:
>
> < infile grep foo | sort > outfile
>
> Redirections don't have to be written after a command.
>

I did not know you could write it that way - thanks for another
off-topic, but useful, tip.

On 31/01/2024 20:11, Janis Papanagnou wrote:

> On Linux you got only a ksh clone, first basically based on ksh88
> (but I anyway never used it but downloaded the original AT&T ksh93).
> Meanwhile, I think, you should get a ksh93 on Linux per default.
> But I suggest to use the ksh93u+m (the version maintained by Martijn
> Dekker), instead; it has a lot errors fixed.
>
> A lot has changed between ksh88 and ksh93 already, and the current
> ksh93 version developed even further. (Its man page shows details.)
>

The most common standard shell on Linux is bash. But alternatives
include busybox (popular on small systems), dash (Ubuntu uses this,
IIRC), and others such as csh, zsh, fish, ksh, sash, yash, and lots of
ones for distributed systems and other special cases. That was just
from a very quick peek on my system.

On 01.02.2024 11:45, David Brown wrote:
> On 31/01/2024 20:11, Janis Papanagnou wrote:
>
>> On Linux you got only a ksh clone, first basically based on ksh88
>> (but I anyway never used it but downloaded the original AT&T ksh93).
>> Meanwhile, I think, you should get a ksh93 on Linux per default.
>> But I suggest to use the ksh93u+m (the version maintained by Martijn
>> Dekker), instead; it has a lot errors fixed.
>>
>> A lot has changed between ksh88 and ksh93 already, and the current
>> ksh93 version developed even further. (Its man page shows details.)
>>
>
> The most common standard shell on Linux is bash. [...]

Sure. But don't confuse a "common 'standard' on Linux" with a
[POSIX] standard. (Ksh and Bash widely conform to POSIX. It's
getting interesting, though, when extensions are getting used.)

If you were confused by "On Linux you got only a ksh clone,",
you have to read it in the context of this subthread; it was
about Kornshell, the POSIX standard , and the relation between
those two. If you wanted Kornshell on Linux, in early days,
you've got some clone. (I compiled the original AT&T sources
for my initial Linux environment myself. I think later they
provided also Linux binaries, but I'm not sure I recall that
correctly.) At some point we got the original AT&T Kornshell
with Linux standard distributions.

Janis

On 31.01.2024 20:28, Scott Lurndal wrote:
> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>> On 31.01.2024 18:20, Scott Lurndal wrote:
>
> <snip commentary about POSIX adopting most of ksh88>
>
>> On Linux you got only a ksh clone, first basically based on ksh88
>> (but I anyway never used it but downloaded the original AT&T ksh93).
>> Meanwhile, I think, you should get a ksh93 on Linux per default.
>> But I suggest to use the ksh93u+m (the version maintained by Martijn
>> Dekker), instead; it has a lot errors fixed.
>
> The AT&T ksh93 has been distributed for a long time. My 2012
> install of Fedora Core 20, OSX, and my recent Ubuntu install use:
>
> $ Version AJM 93u+ 2012-08-01

This is what I regularly see. - Also on my system. But I replaced it
with Version AJM 93u+m/1.0.8 2024-01-01.

(I suppose you use bash, but if you're using ksh I suggest to switch
to "ksh93u+m".)

>
> It's been decades since pdksh/mksh were defaults.

This would just be about one decade. (But it's fruitless to discuss
whether it's a year earlier or later.)

Initially - don't exactly know when I first used Linux, but it must
have been in the late 1990's - you had to download the original Ksh
from kornshell.com (source and/or Linux binaries; ISTR that the
Linux binaries came late, initially they supported the contemporary
commercial Unix platforms).

Janis

On 31.01.2024 21:16, Dan Cross wrote:
> In article <upe60i$1latn$1@dont-email.me>,
> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
>> [snip]
>> On Linux you got only a ksh clone, first basically based on ksh88
>> (but I anyway never used it but downloaded the original AT&T ksh93).
>> Meanwhile, I think, you should get a ksh93 on Linux per default.
>> [snip]
>
> I'm not sure I understand this. Linux is just the kernel; to
> get a useful operating system, you need a number of other
> components such as utilities, system libraries, a load, and
> other tools. A number of such distributions are based on GNU
> tools; at least one is based on BSD tools. So writing, "On
> Linux you got only a ksh clone..." doesn't make much sense to me
> and seems like a category error.

Yes, you are absolutely right. - This was sloppy speech on my part.
Sorry for the confusion.

>
> Certainly, one can install `ksh93` in a Linux distribution.

Meanwhile it seems to be standard in various (or all? can't tell)
distros. - I thought at least that should be clear from the text
you quoted above; see the "meanwhile [...] a ksh93 [...]" part.

Janis

On 31.01.2024 22:20, Dan Cross wrote:
>
> That said, when one's talking about whether ksh93 is shipped
> with "Linux" or not, it's surely important to talk about what
> distribution one is referring to!

Yes. Though it's probably not interesting what any individual
runs. More important is how the support for shells by distros
evolve. Here we can collect experiences. (Or someone who has,
for whatever reason, knowledge about all existing distros and
can provide a more accurate picture.)

Currently I'm using Ubuntu, but I had used also other distros
in former times. So I could not give a coherent picture from
my own installations (or the systems I worked with). Even less
I could (not in former times, not now) give a coherent picture
of what tool/version was or is "generally" part of the distros.

My observation, concerning the topic in this subthread, was that
"You don't always get what you want." or only with difficulties.

Janis

On 01/02/2024 12:12, Janis Papanagnou wrote:
> On 31.01.2024 20:28, Scott Lurndal wrote:
>> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>>> On 31.01.2024 18:20, Scott Lurndal wrote:
>>
>> <snip commentary about POSIX adopting most of ksh88>
>>
>>> On Linux you got only a ksh clone, first basically based on ksh88
>>> (but I anyway never used it but downloaded the original AT&T ksh93).
>>> Meanwhile, I think, you should get a ksh93 on Linux per default.
>>> But I suggest to use the ksh93u+m (the version maintained by Martijn
>>> Dekker), instead; it has a lot errors fixed.
>>
>> The AT&T ksh93 has been distributed for a long time. My 2012
>> install of Fedora Core 20, OSX, and my recent Ubuntu install use:
>>
>> $ Version AJM 93u+ 2012-08-01
>
> This is what I regularly see. - Also on my system. But I replaced it
> with Version AJM 93u+m/1.0.8 2024-01-01.
>
> (I suppose you use bash, but if you're using ksh I suggest to switch
> to "ksh93u+m".)
>
>>
>> It's been decades since pdksh/mksh were defaults.

It's funny how this newsgroup often veers away from theoretical
discussions of the C standard, by turning into stackoverflow.

Could you get any more specific!

I thought even mention of a particular C compiler was off-topic. And yes
I can see the OT tag, this stuff still looks totally out of place.

(I've no idea what it is about. For all I know, this is the usenet
equivalent of Mornington Crescent (see Wikipedia), and they're making it
up.)

On 01.02.2024 06:24, Malcolm McLean wrote:
> On 31/01/2024 23:36, Ben Bacarisse wrote:
>>
>> Maybe you are not used to a system where it's trivial to inspect such
>> data. When "some_prog" produces data that are not compatible with the
>> current terminal settings, "some_prog | hd" shows a hex dump instead.
>> The need to do this does not make "some_prog" poorly designed. It may
>> simply mean that the output is /intended/ for further processing.
>>
> Well almost by definition binary output is intended for further
> processing. Binary audio files must ultimately be converted to analogue
> if anyone is to listen to them, for example.

Well, not necessarily. Let's leave the typical use case for a moment...

It might also be analyzed and converted to a digitally represented
formula, say some TeX code, or e.g. like the formal syntax that the
lilypond program uses.

> I had to check how to do a hex dump on the system I'm typing this on.
> The name of the hex dumper is xxd instead of hd, but otherwise it works
> the same way and will accept piped data. But the fact I had to look it
> up tells you that I've never actually used it.

Well, there's always the old Unix standard tool, 'od'.

I use that without thinking or looking it up, since it was ever there,
despite I only rarely use it.

And you observed correctly that nowadays there's typically even more
than one tool available. (And Bart will probably write his own tool. :-)

> The two problems with hex
> dumps are that you've got to do mental arithmetic to convert 8 bit hex
> values into 16 or 32 bit fields,

Hmm.. - have you inspected the man pages of the tools?

At least for 'od' I know it's easy per option...
od -c file # characters (or escapes and octals)
od -t x1 file # hex octets
od -t x2 file # words (two octets)
od -c -t x1 file # characters and octets

> and that once you get a variable length
> field, it's virtually impossible to keep track of and match up the
> following fields.

An inherent property of a binary. In that case you need data specific
applications.

> So in reality what I do when troubleshooting binary
> data is to write a scratch program, or, more often because the trouble
> is in the existing parser, put diagnosics in an existing parser to print
> out a few fields and inspect them that way.

That's fine.

> Of course to check that
> audio or image data is right you have to listen to it or view it - you
> can't tell from looking at the individual samples.

Depends on the sort of check, and the solution approach. (See my
lilypond example.)

Janis

Michael S <already5chosen@yahoo.com> writes:

> On Tue, 30 Jan 2024 23:18:21 -0800
> Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:

[..on sending binary to standard out..]

>> Simple example (disclaimer: not tested):
>>
>> ssh foo 'cd blah ; tar -cf - . | gzip -c' | \
>> (mkdir foo.blah ; cd foo.blah ; gunzip -c | tar -xf -)
>>
>> Of the five main programs in this command, four are using
>> standard out to send binary data:
>>
>> tar -cf - .
>> gzip -c
>> ssh foo [...]
>> gunzip -c
>>
>> The tar -xf - at the end reads binary data on standard in
>> but doesn't output any (or anything else for that matter).
>>
>> It is FAR more cumbersome to accomplish what this command
>> is doing without sending binary data through standard out.
>
> If I am not mistaken, tar, gzip and gunzip do not write binary
> data to standard output by default. [...]

What I think you mean is that these programs don't send their
primary processing output to standard out unless specifically
directed to do so. Well sure. I don't think that takes away from
the point that it is useful to use standard out for a binary output
stream.

>> Anyone who doesn't understand this doesn't understand Unix.
>
> Frankly, Unix redirection racket looks like something hacked
> together rather than designed as result of the solid thinking
> process. As long as there were only standard input and output it
> was sort of logical. But when they figured out that it is
> insufficient, they had chosen a quick hack instead of
> constructing a solution that wouldn't offend engineering senses
> of any non-preconditioned observer.

First I think you are being too harsh on the people who originally
came up the pipe/redirection mechanism. Considering the historical
context it was a big step forward, and a good match to available
processing resources at the time.

That said, no one is claiming that the single pipe/redirection
mechanism is the be-all and end-all, or that it solves all the
world's problems. But it does do a good job of solving a
significant subclass of the world's problems, and in that context
provides a good motivating example for using standard out for
binary data as well as textual data.

In article <upg23b$22ef0$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
>On 31.01.2024 21:16, Dan Cross wrote:
>> In article <upe60i$1latn$1@dont-email.me>,
>> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
>>> [snip]
>>> On Linux you got only a ksh clone, first basically based on ksh88
>>> (but I anyway never used it but downloaded the original AT&T ksh93).
>>> Meanwhile, I think, you should get a ksh93 on Linux per default.
>>> [snip]
>>
>> I'm not sure I understand this. Linux is just the kernel; to
>> get a useful operating system, you need a number of other
>> components such as utilities, system libraries, a load, and
>> other tools. A number of such distributions are based on GNU
>> tools; at least one is based on BSD tools. So writing, "On
>> Linux you got only a ksh clone..." doesn't make much sense to me
>> and seems like a category error.
>
>Yes, you are absolutely right. - This was sloppy speech on my part.
>Sorry for the confusion.
>
>> Certainly, one can install `ksh93` in a Linux distribution.
>
>Meanwhile it seems to be standard in various (or all? can't tell)
>distros. - I thought at least that should be clear from the text
>you quoted above; see the "meanwhile [...] a ksh93 [...]" part.

Oh I see now; that was aspirational. I suppose the challenge
there would be convincing the various distro maintainers that
shipping ksh93 by default adds value beyond what they get from
whatever shell(s) they are currently shipping.

- Dan C.

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:

> [...]

I would greatly appreciate it if for future such [OT] threads
that they be started in a newsgroup where they are more
relevant to the subjects usually discussed there. Thank you.

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:

> On 31/01/2024 07:18, Tim Rentsch wrote:
>
>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>
>>> On 30/01/2024 07:27, Tim Rentsch wrote:
>>>
>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>>
>>>>> On 29/01/2024 20:10, Tim Rentsch wrote:
>>>>>
>>>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>>>>
>>>>>> [...]
>>>>>>
>>>>>>> I've never used standard output for binary data.
>>>>>>> [...] it strikes me as a poor design decision.
>>>>>>
>>>>>> How so?
>>>>>
>>>>> Because the output can't be inspected by humans, and because it might
>>>>> have unusual effects if passed though systems designed to handle
>>>>> human-readable text. For instance in some systems designed to receive
>>>>> ASCII text, there is no distinction between the nul byte and "waiting
>>>>> for next data byte". Obviously this will cause difficuties if the data
>>>>> is binary.
>>>>> Also many binary formats can't easily be extended, so you can pass one
>>>>> image and that's all. While it is possible to devise a text format
>>>>> which is similar, in practice text formats usually have enough
>>>>> redundancy to be easily extended.
>>>>>
>>>>> So it's harder to correct errors, more prone to errors, and harder to
>>>>> extend.
>>>>
>>>> Your reasoning is all gobbledygook. Your comments reflect only
>>>> limitations in your thinking, not any essential truth about using
>>>> standard out for binary data.
>>>
>>> I must admit that it's nothing I have ever done or considered doing.
>>>
>>> [...]
>>
>> Simple example (disclaimer: not tested):
>>
>> ssh foo 'cd blah ; tar -cf - . | gzip -c' | \
>> (mkdir foo.blah ; cd foo.blah ; gunzip -c | tar -xf -)
>>
>> Of the five main programs in this command, four are using
>> standard out to send binary data:
>>
>> tar -cf - .
>> gzip -c
>> ssh foo [...]
>> gunzip -c
>>
>> The tar -xf - at the end reads binary data on standard in
>> but doesn't output any (or anything else for that matter).
>>
>> It is FAR more cumbersome to accomplish what this command
>> is doing without sending binary data through standard out.
>> Anyone who doesn't understand this doesn't understand Unix.
>
> Yes. I don't do that sort of thing.
> While I have used Unix, it is as a platform for interactive programs
> which work on graohics, or a general C compilation environment. I
> don;t build pipeliens to do that sort of data processing. If I had to
> download a tar file I'd either use a graphical tool or type serveal
> commands into the shell, each launching single executable,
> interactively.
>
> The reason is that I'd only run the command once, and it's so likely
> that there will be either a syntax misunderstanding or a typing error
> that I'd have to test to ensure that it was right. And by the time
> you've done that any time saved by typing only one commandline is
> lost. Of course if you are writing scripts then that doesn't
> apply. But now it's effectively a programming language, and, from the
> example code, a very poorly designed one which is cryptic and fussy
> and liable to be hard to maintain. So it's better to use a language
> like Perl to achieve the same thing, and I did have a few Perl scripts
> handy for repetitive jobs of that nature in my Unix days.
>
> You admit this with "not tested". Says it all. '"Understandig Unix" is
> an intellectually useless achievement. You might have to do it if you
> have to use the system and debug and trouble shoot. But it's nothing
> to be proud about.

You're an idiot. As usual trying to have a useful discussion
with you has turned out to be a complete waste of time.

On 01/02/2024 13:02, Janis Papanagnou wrote:
> On 01.02.2024 06:24, Malcolm McLean wrote:
>> On 31/01/2024 23:36, Ben Bacarisse wrote:
>>>
>>> Maybe you are not used to a system where it's trivial to inspect such
>>> data. When "some_prog" produces data that are not compatible with the
>>> current terminal settings, "some_prog | hd" shows a hex dump instead.
>>> The need to do this does not make "some_prog" poorly designed. It may
>>> simply mean that the output is /intended/ for further processing.
>>>
>> Well almost by definition binary output is intended for further
>> processing. Binary audio files must ultimately be converted to analogue
>> if anyone is to listen to them, for example.
>
> Well, not necessarily. Let's leave the typical use case for a moment...
>
> It might also be analyzed and converted to a digitally represented
> formula, say some TeX code, or e.g. like the formal syntax that the
> lilypond program uses.
>
And ultimately converted to a non binary form. A list of 1s and 0s is
seldom any use to the final consumer of the data.
>

>> The two problems with hex
>> dumps are that you've got to do mental arithmetic to convert 8 bit hex
>> values into 16 or 32 bit fields,
>
> Hmm.. - have you inspected the man pages of the tools?
>
I just ran "man xxd". The man page contains this statement.

The tool's weirdness matches its creator's brain. Use entirely at your
own risk. Copy files. Trace it. Become a wizard.
> At least for 'od' I know it's easy per option...
> od -c file # characters (or escapes and octals)
> od -t x1 file # hex octets
> od -t x2 file # words (two octets)
> od -c -t x1 file # characters and octets
>
So a JPEG file starts with
FF D8
FF E0
hi lo (length of the FF E0 segment)

So we want the output

FF D8 FF E0 [1000] to check that the segment markers are correct and FF
E0 segment is genuinely a thousand bytes (or whatever it is). This isn't
easy to achieve with a hex dump utility.

--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On 01/02/2024 01:21, bart wrote:
> On 31/01/2024 23:36, Ben Bacarisse wrote:
>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>
>>> On 30/01/2024 07:27, Tim Rentsch wrote:
>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>>
>>>>> On 29/01/2024 20:10, Tim Rentsch wrote:
>>>>>
>>>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>>>>
>>>>>> [...]
>>>>>>
>>>>>>> I've never used standard output for binary data.
>>>>>>> [...] it strikes me as a poor design decision.
>>>>>>
>>>>>> How so?
>>>>>
>>>>> Because the output can't be inspected by humans, and because it might
>>>>> have unusual effects if passed though systems designed to handle
>>>>> human-readable text.
>>
>> Maybe you are not used to a system where it's trivial to inspect such
>> data.  When "some_prog" produces data that are not compatible with the
>> current terminal settings, "some_prog | hd" shows a hex dump instead.
>> The need to do this does not make "some_prog" poorly designed.  It may
>> simply mean that the output is /intended/ for further processing.
>>
>>> For instance in some systems designed to receive
>>>>> ASCII text, there is no distinction between the nul byte and "waiting
>>>>> for next data byte".  Obviously this will cause difficuties if the
>>>>> data
>>>>> is binary.
>>>>> Also many binary formats can't easily be extended, so you can pass one
>>>>> image and that's all.  While it is possible to devise a text format
>>>>> which is similar, in practice text formats usually have enough
>>>>> redundancy to be easily extended.
>>>>>
>>>>> So it's harder to correct errors, more prone to errors, and harder to
>>>>> extend.
>>>> Your reasoning is all gobbledygook.  Your comments reflect only
>>>> limitations in your thinking, not any essential truth about using
>>>> standard out for binary data.
>>>>
>>> I must admit that it's nothing I have ever done or considered doing.
>>>
>>> However standard output is designed for text and not binary ouput.
>>
>> What is your evidence?  stdout was just designed for output (as far as I
>> can tell) and, anyway, what is the distinction you are making between
>> binary and text?  iconv --from ACSII --to EBCDIC-UK will produce
>> something that is "logically" text on stdout, but it might look like
>> binary to you.
>>
>> An example where it's really useful not to care: I have a suite of tools
>> for doing toy cryptanalysis.  Some apply various transformations and/or
>> filters to byte streams and others collect and output (on stderr)
>> various statistics.  Plugging them together in various pipelines is very
>> handy when investigating an encrypted text.  The output is almost always
>> "binary" in the sense that there would be not point in looking at on a
>> terminal.
>>
>> According to you, these tools are poorly designed.  I don't think so.
>> How would you design them?  Endless input and output file names to be
>> juggled and tidied up afterwards?
>
> I think they're poorly designed too.

Then you are wrong too. (Although I think you've been doing a slightly
better job than Malcolm of explaining /why/ you think they way you do.)

Basically, neither you nor Malcolm are familiar with these tools. You
don't use them significantly - and when you do, it is for simple things
and you feel they are over-complicated for the tasks /you/ need. And
maybe they /are/ over-complicated for your particular needs - but they
are useful to a large number of people in a large number of ways. They
are not poorly designed - they are simply not designed for /your/ needs
and preferences.

This of course works in all directions. Some people prefer gui tools
and think command-line tools are hard to use. Some people prefer
command line tools and think gui tools are slow and inefficient. Some
people like combining multiple small, flexible tools that each handle
part of the task - others prefer monolithic tools that try to do
everything. Each possibility has its advantages, and disadvantages, its
proponents and detractors. And if you want to be a happy and efficient
developer, you stay open to using any of the solutions - learn at least
a bit about them all, and use what suits you and your needs best at the
time.

But do not mistake "I don't like this" or "I think this is hard" with
"it's poorly designed".

>
> From the POV of interactive console programs, they /are/ poor. But the
> mistake is thinking that they are actual programs or commands, when
> really they are just filters. They are not designed to be standalone
> commands.
>

Of course they are programs. Some programs can be used as filters -
that doesn't mean they are not programs!

> Even 'cat', if I type it by itself, just sits there. (I wonder what use
> it has in a sequence like ... | cat | ...; what does it add to the data?)
>

Nothing, in this case.

And if I open - say - MS Word, then close it again, I have also done
nothing. Does that mean MS Word is a useless program that doesn't
really do anything?

> AFAICS, this stuff mainly works inside scripts. Or do people here spend
> all day manually piping stuff between programs?

This stuff works in scripts /and/ ad-hoc from the command line. People
use both, whatever is convenient at the time. I don't spend "all day"
piping stuff, but I guess I perhaps use pipes a dozen or so times a day
- the variance here is so big it's hard to give a sensible average.
Many cases are piping the output of one command into "less" and/or
"grep". Other common targets for pipes (for me) are head, tail (or
"tail -f"), "hexdump -C", "wc -l".

>
> As for alternatives, I don't know. There are any number of ways this
> could be done. But if everyone has become inured to this piping
> business, they will not be receptive to anything different.

No one has claimed pipes or *nix stream redirection is a "perfect"
system. No one has suggested they think things could not have been done
in different ways. Just as with C, you are mistaking "we know how this
works, we use it, we are happy enough with it that we can work with it,
we know why it is this way" with "we think this is perfect and nothing
else will come close". Discussions with you would be so much more
fruitful if you didn't have such an absurd "you're with us or against
us" attitude. Opinions are not binary.

Those of us who understand *nix shell usage, pipes and indirection, and
are familiar with *nix common command line tools, and are therefore able
to give qualified opinions based on facts and experience (i.e., not you,
and not Malcolm), seem to find them useful. I'm sure, however, we can
all think of potential improvements.

But like C, the advantages of familiarity and common implementations
greatly outweigh the disadvantages. Maybe "ls" would have been better
if it had been spelt "list". Perhaps "rm" should have had different
defaults, or "sed" could have been less cryptic. The fact that "ls",
"rm", and other fundamentals works exactly the same way on every *nix
system made in the last 40 years, gaining new features if needed without
breaking compatibility, is a /massive/ advantage.

You claim others are not open to considering something different, which
may potentially be "better" (for some values of "better"). You are
wrong. Most people, however, understand that momentum is important. It
can mean clearly inferior systems become standard - such as Windows or
the x86 ISA, which are technically massively inferior to alternatives,
but are successful due to momentum.

>
> Here however are some ideas:
>
> * Have versions of these tools for use as filters with no UI, just
>   a default input and output, and versions for interactive use with
>   helpful prompts. Or even just a sensibly named output! Instead of
>   every program writing a.out.
>

Sometimes a "front end" with a nice UI /is/ useful. So people write
front ends with nice UI's - text-based or gui. Typically these are
great for common tasks, and are cumbersome or useless for rarer or more
advanced stuff. And that's fine - use whatever works best for you at
the time. If you think "ssh" is complicated from the command line, use
"putty" - but that won't handle all the uses that some people need.

On 01.02.2024 14:22, Tim Rentsch wrote:
> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>
>> [...]
>
> I would greatly appreciate it if for future such [OT] threads
> that they be started in a newsgroup where they are more
> relevant to the subjects usually discussed there. Thank you.

I changed to this specific subject when the subthread already was
OT and I additionally marked it as [OT] to make it possible for
readers who are not interested in that digression to just skip it.

Label and subject! - Thanks for your attention and understanding.

This thread that originally had the subject iso646.h has hundreds
of posts not related to iso646.h - ...in case you've not noticed.
Why don't you comment on these untagged digressions in the first
place if you feel so comfortable in a Usenet control freak role?
(Unfortunately I read them because they had NOT changed subject
or marked them as [OT]. You probably missed them because of the
topical subject despite having off-topic content?)
Looking forward to see you replaying to all the OT posts that are
not marked as [OT] or carry an appropriate subject.
No, not really! But you would certainly be The One here who would
do so.
Hint: The Netiquette is a big source for more reasons to complain
(line lengths, etc.). Good luck.

Janis

bart <bc@freeuk.com> writes:
>On 01/02/2024 00:47, Scott Lurndal wrote:
>> bart <bc@freeuk.com> writes:
>>> On 31/01/2024 23:36, Ben Bacarisse wrote:
>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>
>>>> According to you, these tools are poorly designed. I don't think so.
>>>> How would you design them? Endless input and output file names to be
>>>> juggled and tidied up afterwards?
>>>
>>> I think they're poorly designed too.
>>
>> Of course you do. They're not bart programs.
>>
>>>
>>> From the POV of interactive console programs, they /are/ poor.
>>
>> You don't provide any reason why - do elucidate!
>
>They only do one thing, like you can't first do A, then B. They don't
>give any prompts. They often apparently do nothing (so you can't tell if
>they're busy, waiting for input, or hanging). There is no dialog.

Those are features, not defects.

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>On 31/01/2024 23:36, Ben Bacarisse wrote:
>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>
>>> On 30/01/2024 07:27, Tim Rentsch wrote:
>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>>
>>>>> On 29/01/2024 20:10, Tim Rentsch wrote:
>>>>>
>>>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>>>>
>>>>>> [...]
>>>>>>
>>>>>>> I've never used standard output for binary data.
>>>>>>> [...] it strikes me as a poor design decision.
>>>>>>
>>>>>> How so?
>>>>>
>>>>> Because the output can't be inspected by humans, and because it might
>>>>> have unusual effects if passed though systems designed to handle
>>>>> human-readable text.
>>
>> Maybe you are not used to a system where it's trivial to inspect such
>> data. When "some_prog" produces data that are not compatible with the
>> current terminal settings, "some_prog | hd" shows a hex dump instead.
>> The need to do this does not make "some_prog" poorly designed. It may
>> simply mean that the output is /intended/ for further processing.
>>
>>> For instance in some systems designed to receive
>>>>> ASCII text, there is no distinction between the nul byte and "waiting
>>>>> for next data byte". Obviously this will cause difficuties if the data
>>>>> is binary.
>>>>> Also many binary formats can't easily be extended, so you can pass one
>>>>> image and that's all. While it is possible to devise a text format
>>>>> which is similar, in practice text formats usually have enough
>>>>> redundancy to be easily extended.
>>>>>
>>>>> So it's harder to correct errors, more prone to errors, and harder to
>>>>> extend.
>>>> Your reasoning is all gobbledygook. Your comments reflect only
>>>> limitations in your thinking, not any essential truth about using
>>>> standard out for binary data.
>>>>
>>> I must admit that it's nothing I have ever done or considered doing.
>>>
>>> However standard output is designed for text and not binary ouput.
>>
>> What is your evidence? stdout was just designed for output (as far as I
>> can tell) and, anyway, what is the distinction you are making between
>> binary and text? iconv --from ACSII --to EBCDIC-UK will produce
>> something that is "logically" text on stdout, but it might look like
>> binary to you.
>>
>> An example where it's really useful not to care: I have a suite of tools
>> for doing toy cryptanalysis. Some apply various transformations and/or
>> filters to byte streams and others collect and output (on stderr)
>> various statistics. Plugging them together in various pipelines is very
>> handy when investigating an encrypted text. The output is almost always
>> "binary" in the sense that there would be not point in looking at on a
>> terminal.
>>
>> According to you, these tools are poorly designed. I don't think so.
>> How would you design them? Endless input and output file names to be
>> juggled and tidied up afterwards?
>>
>I'd write a monolithic program.

Even a monolithic program is decomposed into subroutines (or malcolm functions).

A pipeline is the same concept at a higher level.

>Load the encryoted text into memory, and then pass it to subroutines to
>do the various analyses.

So your program is arbitrarily large and needs to be recompiled
to add new subroutines. Advantage to the pipeline, again.