On Wed, 31 Jan 2024 23:25:25 +0000, Malcolm McLean wrote:

> In ASCII, 0 means NUL, or "ignore".

Fun fact: one of the names for hex 7F was “rubout”. On seven-track paper
tape, if you made a mistake typing your program, intead of throwing away
the tape and starting again, you could go back and punch out all the holes
at that position to produce a “rubout” character. The meaning was “ignore
this character”.

On 31.01.2024 19:24, David Brown wrote:
> On 31/01/2024 19:01, Janis Papanagnou wrote:
>
>> All I can say is that the Unix shell was a reliable companion
>> wherever we had to automate tasks on Unix systems or on Cygwin
>> enhanced Windows.
>
> Automation is certainly easier with good scripting - whatever the
> language or shell.

Sure. And shell was always available as part of standard Unix.
That was not (not always) true for other languages, like Perl
(for example).

Janis

On 31.01.2024 19:20, David Brown wrote:
> On 31/01/2024 16:25, Janis Papanagnou wrote:
>> On 31.01.2024 15:21, David Brown wrote:
>>> On 31/01/2024 09:36, Malcolm McLean wrote:
>>>> [ I snipped a couple of "I actually don't know/need it" things ]
>>>>
>>>> But now it's effectively a programming language, and, from the example
>>>> code, a very poorly designed one which is cryptic and fussy and liable
>>>> to be hard to maintain. So it's better to use a language like Perl to
>>>> achieve the same thing, and I did have a few Perl scripts handy for
>>>> repetitive jobs of that nature in my Unix days.
>>>
>>> That gave me a laugh! You think bash is cryptic, fussy and poorly
>>> designed, and choose /Perl/ as the alternative :-)
>>
>> I don't think it's that clear a joke. The Unix shell is extremely
>> error prone to program, and you should not let a newbie write shell
>> programs without careful supervision. ("newbie" [in shell context]
>> = less than 10 years of practical experience. - Am I exaggerating?
>> Maybe. But not much.)
>
> I'm not a great fan of shell programming - anything advanced, and I tend
> to reach for Python. But I think that is a matter of familiarity and
> practice. But if you consider bash programming as difficult to get
> right, I'll not argue.

Not specifically bash programming, the same is true for ksh, etc.;
it's the underlying shell design that has a lot of pitfalls. And
it's not only about familiarity with the tool - of course being
familiar with the concepts is necessary. But there's still enough
pits where even years long programmers stumble into. (I'm saying
that as someone who did 35+ years ksh programming, I gave courses,
defined shell coding standards, followed 20+ years the problems
that users had in comp.unix.shell, and even saw experienced shell
book authors (and I'm not even mentioning bloggers), to fail in
some instances of the language.)

But of course, with knowledge and discipline, you can write also
fine shell programs.

Despite the shell inherent issues I like it because I can solve
some types of tasks reliably in Unix context.

>
> Perl is famously known as a "write-only" language. Sure, it is possible
> to write good, clear, maintainable Perl code - but few people do that.

I've programmed just a few times in Perl, mostly only extending
existing programs. But a friend of mine is leading a Perl user
group in our city; his programs (despite some cryptic elements)
are still quite legible.

>
> Thus the idea that finding bash cryptic or difficult and using Perl
> instead is the joke.

Well, in shell it's all that '1>&2' and '${f##*/}' and whatnot
stuff that can only be called cryptic. I wouldn't count regexps
because that's the base in any proper scripting language. What
remains for Perl to be cryptic? The variable type prefixes are
the most prominent punctuation elements that pop into my head.
If one wants better legible yet simple scripts he could resort
to Awk; but its focus is different from shell. (Perl supports
both, but is not everywhere available.)

Janis

Lawrence D'Oliveiro <ldo@nz.invalid> writes:
>On Wed, 31 Jan 2024 14:45:49 +0100, David Brown wrote:
>
>> On 31/01/2024 07:07, Lawrence D'Oliveiro wrote:
>>
>>> On Tue, 30 Jan 2024 17:25:31 +0100, David Brown wrote:
>>>
>>>> Mixing binary data with formatted text data is very unlikely to be
>>>> useful.
>>>
>>> PDF does exactly that. To the point where the spec suggests putting
>>> some random unprintable bytes up front, to distract format sniffers
>>> from thinking they’re looking at a text file.
>>
>> PDF files start with the "magic" indicator "%PDF", which is enough for
>> many programs to identify them correctly.
>
>Sure, if you were looking for PDF files specifically.
>
>But consider the more generic case of file-transfer tools that try to
>automatically convert between line-endings for text files on different
>platforms: if they mistook a PDF file for text, they could screw it up
>royally.

Just another reason not to use the system with two-byte line endings.

Not a problem on unix.

On 01.02.2024 23:56, bart wrote:
> On 01/02/2024 22:02, David Brown wrote:
>> On 01/02/2024 18:06, bart wrote:
>
>>> You can't send output A to stdout, then B to stdout, and certainly
>>> can't interleave messages to the console on stdout, as that would
>>> then be all mixed up with the possibly binary data, and if
>>> redirected, you won't see it.
>>
>> $ cat A
>> one
>> two
>> three
>>
>> $ cat B
>>
>> cat
>> dog
>> cow
>>
>> $ (cat A; cat B) | wc -l
>> 6
>>
>> That's the output of two commands, "cat A" and "cat B", each going to
>> their stdout, and they are concatenated into a single pipe going to
>> the "wc -l" command to count the lines.

I see you're trying to teach shell basics to Bart; interesting.

>
> I see you don't get it. This is the equivalent of a program which is

What is meant by "This"? (I cannot find code or description.)

> supposed to do this:
>
> print A to TTY
> print B to LPT
> print C to TTY
> print D to LPT

Is that the task? Are A-D program names? Any other requirements
or restrictions you impose?

cat A
cat B | lpr # of course you don't need cat here: lpr B
cat C
cat D | lpr # of course you don't need cat here: lpr D

You want stdout collected? Do it as David suggested, e.g.

{ cat A ; lpr B ; cat C ; lpr D ;} | processor-for-A-and-C

Or you want to split A, B, C, or D to different channels or tools?
Then use 'tee', and/or use redirection to files, and/or to processes.

>
> but instead is written as this:
>
> print A to TTY
> print B to TTY
> print C to TTY
> print D to TTY
>
> and you are expected to redirect all TTY output to LPT.
>
> At least, on LPT, B and D can each start with a separate title page; on
> stdout directed to a file, it will be all mixed up.

See above. - Any other task? - Or did you mean something else?

(It's hard to believe that there's something possible in the DOS
cmd/bat/bart world that wouldn't be possible in Unix shell - besides
blue screens of course.)

Janis

Lawrence D'Oliveiro <ldo@nz.invalid> writes:
>On Wed, 31 Jan 2024 23:25:25 +0000, Malcolm McLean wrote:
>
>> In ASCII, 0 means NUL, or "ignore".
>
>Fun fact: one of the names for hex 7F was “rubout”.

Additional fun fact. Rubout was the legend on the keycap on
the ASR-33 used to rub out the prior character (the A in ASR
means it has the reader/punch). On paper tape, it means ignore
the prior character.

> bart <bc@freeuk.com> writes:
> [...]
>> I've just realised why it is that your filter programs don't show
>> prompts or any kinds of messages: because those are sent to stdout,
>> and therefore will screw up any data that is being sent there as the
>> primary output.

Erm, no. Filter programs just don't need a prompt.

And if you have a program that want to provide a prompt, and provide
data on standard output, you usually wouldn't use stdout for that;
use stderr or /dev/tty, depending on the intention of the tool.

And even if you intend to use a prompt on stdout (what really sounds
as a weak idea) you can program a test whether your file descriptor is
attached to the tty if you like.

To demonstrate that, here's some ksh code...

$ ( exec 0>&- ; [[ -t 0 ]] ; echo $? ) # stdin closed
1 # error
$ ( [[ -t 0 ]] ; echo $? ) # stdin attached to tty
0 # okay
$ ls x | ( [[ -t 0 ]] ; echo $? ) # stdin attached to pipe
1 # error

Janis

On 02.02.2024 03:12, Scott Lurndal wrote:
> Lawrence D'Oliveiro <ldo@nz.invalid> writes:
>>
>> But consider the more generic case of file-transfer tools that try to
>> automatically convert between line-endings for text files on different
>> platforms: if they mistook a PDF file for text, they could screw it up
>> royally.
>
> Just another reason not to use the system with two-byte line endings.

That cannot always be avoided.

>
> Not a problem on unix.

There are several situations where it matters to consider CR/LF or when
some OS setting may handle these line terminators. Even if you're only
staying in your Unix universe. The "funniest" thing is if you process
files that have been edited by different people on different platforms.
(I know that I am not the first one who has written a CR-LF-CRLF tool
to check and fix (in some consistent way) the line endings of files.)

Janis

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>On 02.02.2024 03:12, Scott Lurndal wrote:
>> Lawrence D'Oliveiro <ldo@nz.invalid> writes:
>>>
>>> But consider the more generic case of file-transfer tools that try to
>>> automatically convert between line-endings for text files on different
>>> platforms: if they mistook a PDF file for text, they could screw it up
>>> royally.
>>
>> Just another reason not to use the system with two-byte line endings.
>
>That cannot always be avoided.
>
>>
>> Not a problem on unix.
>
>There are several situations where it matters to consider CR/LF or when
>some OS setting may handle these line terminators. Even if you're only
>staying in your Unix universe. The "funniest" thing is if you process
>files that have been edited by different people on different platforms.
>(I know that I am not the first one who has written a CR-LF-CRLF tool
>to check and fix (in some consistent way) the line endings of files.)

vim can load and save with either line ending, switching if the
user wishes.

scott@slp53.sl.home (Scott Lurndal) writes:
> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
[...]
>>There are several situations where it matters to consider CR/LF or when
>>some OS setting may handle these line terminators. Even if you're only
>>staying in your Unix universe. The "funniest" thing is if you process
>>files that have been edited by different people on different platforms.
>>(I know that I am not the first one who has written a CR-LF-CRLF tool
>>to check and fix (in some consistent way) the line endings of files.)
>
> vim can load and save with either line ending, switching if the
> user wishes.

Yes, but if a file has a mix of LF and CRLF line endings, it shows an
annoying "^M" on the end of each CRLF line. If all lines are LF, or all
lines are CRLF, it doesn't.

There's probably a way to change that, but when I've run into the
problem in the past I've been able to modify the files.

(Vague relevance: All C compilers I've tried don't complain about mixed
LF and CRLF line endings.)

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

On Thu, 1 Feb 2024 12:50:36 +0000, bart wrote:

> For all I know, this is the usenet equivalent of Mornington Crescent ...

You can’t get far in C without knowing POSIX. C on Windows is, let’s face
it, a crippled language.

On Thu, 1 Feb 2024 11:45:47 +0100, David Brown wrote:

> .. dash (Ubuntu uses this, IIRC) ...

On Debian and derivatives (including Ubuntu), dash is the go-to for a
minimal, strictly POSIX-compliant shell.

On Thu, 01 Feb 2024 05:22:54 -0800, Tim Rentsch wrote:

> I would greatly appreciate it if for future such [OT] threads that they
> be started in a newsgroup where they are more relevant to the subjects
> usually discussed there. Thank you.

Friendly tip: all the good USENET news readers have facilities to allow
you to ignore messages you don’t want to see. Use them.

On Wed, 31 Jan 2024 12:43:32 +0200, Michael S wrote:

> Frankly, Unix redirection racket looks like something hacked together
> rather than designed as result of the solid thinking process.
> As long as there were only standard input and output it was sort of
> logical. But when they figured out that it is insufficient, they had
> chosen a quick hack instead of constructing a solution that wouldn't
> offend engineering senses of any non-preconditioned observer.

One thing with *nix-type OSes versus others is that shells are not
specially-privileged system components in any sense, they are just
ordinary user programs like any other.

If you have some clever ideas for an alternative command language that
doesn’t offend your sensibilities, feel free to code it up and show us how
it’s done.

On Fri, 02 Feb 2024 02:15:05 GMT, Scott Lurndal wrote:

> Lawrence D'Oliveiro <ldo@nz.invalid> writes:
>
>>On Wed, 31 Jan 2024 23:25:25 +0000, Malcolm McLean wrote:
>>
>>Fun fact: one of the names for hex 7F was “rubout”.
>
> Additional fun fact. Rubout was the legend on the keycap on the ASR-33
> used to rub out the prior character (the A in ASR means it has the
> reader/punch). On paper tape, it means ignore the prior character.

No, you had to overpunch the character to be ignored. Did that key
automatically backspace the tape for you, or did you have to do it
manually?

On Mon, 29 Jan 2024 09:52:45 +0100, David Brown wrote:

> (Facebook uses Haskell for their filters and other data handling.

I’m sure that may work OK for a filter expressed as a pure function. How
does it work for one where the filter has internal state which is changed
as a result of prior input, and which alters the production of subsequent
output?

On 02/02/2024 01:48, Janis Papanagnou wrote:
> On 31.01.2024 19:20, David Brown wrote:
>> On 31/01/2024 16:25, Janis Papanagnou wrote:
>>> On 31.01.2024 15:21, David Brown wrote:
>>>> On 31/01/2024 09:36, Malcolm McLean wrote:
>>>>> [ I snipped a couple of "I actually don't know/need it" things ]
>>>>>
>>>>> But now it's effectively a programming language, and, from the example
>>>>> code, a very poorly designed one which is cryptic and fussy and liable
>>>>> to be hard to maintain. So it's better to use a language like Perl to
>>>>> achieve the same thing, and I did have a few Perl scripts handy for
>>>>> repetitive jobs of that nature in my Unix days.
>>>>
>>>> That gave me a laugh! You think bash is cryptic, fussy and poorly
>>>> designed, and choose /Perl/ as the alternative :-)
>>>
>>> I don't think it's that clear a joke. The Unix shell is extremely
>>> error prone to program, and you should not let a newbie write shell
>>> programs without careful supervision. ("newbie" [in shell context]
>>> = less than 10 years of practical experience. - Am I exaggerating?
>>> Maybe. But not much.)
>>
>> I'm not a great fan of shell programming - anything advanced, and I tend
>> to reach for Python. But I think that is a matter of familiarity and
>> practice. But if you consider bash programming as difficult to get
>> right, I'll not argue.
>
> Not specifically bash programming, the same is true for ksh, etc.;
> it's the underlying shell design that has a lot of pitfalls. And
> it's not only about familiarity with the tool - of course being
> familiar with the concepts is necessary. But there's still enough
> pits where even years long programmers stumble into. (I'm saying
> that as someone who did 35+ years ksh programming, I gave courses,
> defined shell coding standards, followed 20+ years the problems
> that users had in comp.unix.shell, and even saw experienced shell
> book authors (and I'm not even mentioning bloggers), to fail in
> some instances of the language.)
>
> But of course, with knowledge and discipline, you can write also
> fine shell programs.
>
> Despite the shell inherent issues I like it because I can solve
> some types of tasks reliably in Unix context.
>
>>
>> Perl is famously known as a "write-only" language. Sure, it is possible
>> to write good, clear, maintainable Perl code - but few people do that.
>
> I've programmed just a few times in Perl, mostly only extending
> existing programs. But a friend of mine is leading a Perl user
> group in our city; his programs (despite some cryptic elements)
> are still quite legible.
>
>>
>> Thus the idea that finding bash cryptic or difficult and using Perl
>> instead is the joke.
>
> Well, in shell it's all that '1>&2' and '${f##*/}' and whatnot
> stuff that can only be called cryptic. I wouldn't count regexps
> because that's the base in any proper scripting language. What
> remains for Perl to be cryptic? The variable type prefixes are
> the most prominent punctuation elements that pop into my head.
> If one wants better legible yet simple scripts he could resort
> to Awk; but its focus is different from shell. (Perl supports
> both, but is not everywhere available.)
>
In Perl you have an implict variable called $_. Some Perl statements
will operate on $_ without it actually being specified, and you then
have to reference $_ exoictitly to obtain the result. It's highly
confusing for anyone used to a conventional language with only one type
of named varibales. And that's one of the main decisions which makes
Perl hard to read.

However often you can write slightly less idiomatic Perl code which
doesn't make use of this feature, and then it's clearer. Or you can lay
the code out so that all the places where $_ are used in the same way
are together and make it a bit easier to work out what is going on.
There are thing you can do and Perl doesn't have to look like a
confusing mess.

--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On 02/02/2024 04:57, Lawrence D'Oliveiro wrote:
> On Thu, 1 Feb 2024 12:50:36 +0000, bart wrote:
>
>> For all I know, this is the usenet equivalent of Mornington Crescent ...
>
> You can’t get far in C without knowing POSIX. C on Windows is, let’s face
> it, a crippled language.
>
My program Crossword Designer is a Windows program and is written
entirely in C.
--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On 02/02/2024 06:13, Lawrence D'Oliveiro wrote:
> On Mon, 29 Jan 2024 09:52:45 +0100, David Brown wrote:
>
>> (Facebook uses Haskell for their filters and other data handling.
>
> I’m sure that may work OK for a filter expressed as a pure function. How
> does it work for one where the filter has internal state which is changed
> as a result of prior input, and which alters the production of subsequent
> output?

I have no idea. You could ask in a Haskell group, but that is well
beyond my knowledge and well outside this group's topicality. I can
only tell you it is not only possible in Haskell, but practical and
viewed by a big, rich and experienced company as the best language for
the task. As for /how/ it works, you'll have to look elsewhere.

On 02/02/2024 05:57, Lawrence D'Oliveiro wrote:
> On Thu, 1 Feb 2024 12:50:36 +0000, bart wrote:
>
>> For all I know, this is the usenet equivalent of Mornington Crescent ...
>
> You can’t get far in C without knowing POSIX. C on Windows is, let’s face
> it, a crippled language.

I've been a professional programmer for thirty years, and while I have
used a number of languages, most of my work has been in C. POSIX is
totally irrelevant to my programming, as is C on Windows.

It is true that C is often used on POSIX targets. It is true that
programming on POSIX systems is often done on C. It is /not/ true that
they are synonymous.

POSIX has been essential for the standardisation of *nix systems, and
gives a good common base that is beyond the C standard but not as
specific as an OS. However, it is in itself pretty irrelevant to most C
programmers. Even if you limit yourself to C programmers targetting
*nix systems, they target Linux, or particular libraries and frameworks
(such as GTK or SDL), or standard C, or gcc+glibc. POSIX makes no
difference - if the functions and features they want are in the
libraries they will be using, they will use them.

/Some/ C programmers write code for POSIX, with an aim to portability
across any POSIX system, but it is surely a very small minority.

(But most C programmers probably have access to the POSIX-extended
printf that started this.)

On 01/02/2024 23:56, bart wrote:
> On 01/02/2024 22:02, David Brown wrote:

>> I'm not sure we are getting anywhere with you trying to invent more
>> and more complex situations in an attempt to find something that can't
>> be done from a Linux bash shell.
>
> They're remarkably simple situations!

As I said, you can try to invent more and more unrealistic situations to
try to "prove" that bash is useless and Unix is flawed and its designers
were incompetent, along with every other programmer and developer
throughout time, while Bart from Usenet has the perfect solution for
everything.

If you are happy with everything you have made yourself, and think
everything else is unusable, incompetently made, and probably designed
specifically to annoy you personally, why bother with any of it? Why
are you here, complaining about everything? What do you gain from
frothing at the mouth like this, other than high blood pressure?

Ask about C issues. Tell us about C programs. You can even ask about
off-topic stuff - many of us have tried to give you lots of information
about things you are ignorant of. But /please/ stop whining.

On 01/02/2024 23:38, Malcolm McLean wrote:
> I'm sure you're capable of going through the exercise and then you might
> gain a bit of insight on how to design such software systems. And, no,
> arguing that you'd go for a monolithic program doesn't necessarily mean
> that you are a "quivering jelly" at the thought of writing several
> simpler ones. And in fact to start you off I actually mentioned a few
> advantages of the pipeline approach.

I am perfectly aware of the advantages and disadvantages of monolithic
approaches.

I am also perfectly aware that you won't read that previous sentence,
understand it, or consider it before making up your next pointless straw
man or making up another lecture on something you know nothing about
while the rest of us do.

>
> There are advantages and drawbacks to both. But I can't force you to
> think about what those might be if you won't, and from experience just
> telling you provokes your natural contentiousness and isn't very effective.
>

On 02/02/2024 04:57, Lawrence D'Oliveiro wrote:
> On Thu, 1 Feb 2024 12:50:36 +0000, bart wrote:
>
>> For all I know, this is the usenet equivalent of Mornington Crescent ...
>
> You can’t get far in C without knowing POSIX. C on Windows is, let’s face
> it, a crippled language.

It sounds like you're the one who would be crippled without your beloved
POSIX.

There has of course been a huge amount of software written in C on Windows.

On 01.02.2024 16:50, Malcolm McLean wrote:
> On 01/02/2024 15:07, Janis Papanagnou wrote:
>>
>> I don't know binary format details about jpg, so I cannot help you here.
>>
> JPEG is an extremely common binary file format and JPEG files will be
> found on most general purpose computers.

Really?!

> All you need to know for the purposes of the discussion is that the
> first four bytes are segment identifiers and must have the values I

All I wanted to know is what you intended to do. The intended
task.

> gave, whilst bytes five and six are a big endian 16 bit number that
> represents a segment length, and that potentially any of those values
> could be unexpected and you might want to inspect them.

And from subsequent posts I assume you want to test the values
to determine the file type.

> So how would you achieve that in a convenient and non-error prone way?

If I'd be interested in the file type of the data I'd do what
has already been suggested by others, use the 'file' command.
It's the purpose of that command to try to determine the file
type by objective means, or, if that is not unambiguously
possible, by heuristic means.

In case I want to do a more thorough data analysis of such
binary code it applies what I've written upthread about the
need for knowledge about the binary data structures (and use
the tools I mentioned).

Janis

On 01.02.2024 21:09, Keith Thompson wrote:
> [...] I would use semicolons instead:
>
> $ if file /tmp/garage.jpg | grep JPEG > /dev/null ; then echo "it is a jpeg" ; fi
> it is a jpeg
>
> (I might also use "grep -q" rather than redirecting to /dev/null.)

And probably also avoid multi-line code (to prevent the ^J confusion)

file /tmp/garage.jpg | grep -q JPEG && echo "it is a jpeg"

Janis