On Friday, July 9, 2021 at 5:18:36 PM UTC-7, Keith Thompson wrote:

> #include <stdio.h>
> int main(void) {
> int prev = 0;
> int c;
> while ((c = getchar()) != EOF) {
> if (! (prev == '\r' && c == '\n')) {
> putchar(c);
> }
> prev = c;
> }
> }

It works very well, thanks.

Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
> On 10.07.2021 13:02, Spiros Bousbouras wrote:
>> By the way , for this kind of thing the first thing to cross my mind would be
>> to use C because for this problem it actually provides the clearest code.
>
> And likely also the fastest, which would matter if - as I would expect
> with health data - a lot of data has to be handled.
>
>> So
>> something like Keith's code in <87k0lz6my1.fsf@nosuchdomain.example.com>
>> would be my preference.
>
> For small data sets a quick awk prototype solution might be preferable,
> and also if there's no C development environment available in the OP's
> working context.

I thought of using C because I found it clearer to think of the input
and output as character-oriented, not line-oriented. awk processing is
line-oriented.

On the other hand, there's more than one way to describe the problem.

One way, which led to my C solution, was:

Copy input *characters* to output, except that every occurrence of
CRLF is replaced by CR.

An equivalent way to describe it is:

Copy input *lines* to output, except that any line ending in CR is
printed without a trailing newline (think "echo -n" vs. "echo").

(This assumes a "line" is terminated by a single LF character.)

The input is line-oriented. The output is not.

That led me to this awk solution, which is similar to the one Kaz
posted, except that mine is a bit more verbose. It might be easier for
someone who's not an awk expert to follow. (My awk is a bit rusty).

#!/usr/bin/awk -f

{
if (/\r$/) {
printf("%s", $0);
}
else {
print
}
}

Awk's input is performed a line at a time, discarding the terminating
newline character. The "print" statement adds a newline character (and
prints $0, the input line, if you don't give it an argument). "printf"
prints exactly what you tell it to, printing a newline only if you
specify it. (`awk '{print}'` copies input to output *except* that if
the input doesn't end in a newline, it will add one.)

And here's a Perl one-liner solution:

perl -pe 'chomp if /\r$/'

Perl, like awk, is line-oriented, but it doesn't discard newlines on
input. "chomp" deletes a trailing newline. "-p" tells Perl to do an
awk-like loop copying input lines to output. An equivalent Perl script
that doesn't use special command-line options:

#!/usr/bin/perl

# The following two lines don't matter in this case, but are good Perl
# practice in general.
use strict;
use warnings;

while (<>) {
chomp if /\r$/;
print;
}

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips
void Void(void) { Void(); } /* The recursive call of the void */

On 11.07.2021 01:55, Keith Thompson wrote:
>
> I thought of using C because I found it clearer to think of the input
> and output as character-oriented, not line-oriented.

Definitely.

> awk processing is line-oriented.

Unless you use GNU Awk with a "stream processing" setup.

>
> On the other hand, there's more than one way to describe the problem.

Sure.

Janis

Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
> On 11.07.2021 01:55, Keith Thompson wrote:
>> I thought of using C because I found it clearer to think of the input
>> and output as character-oriented, not line-oriented.
>
> Definitely.
>
>> awk processing is line-oriented.
>
> Unless you use GNU Awk with a "stream processing" setup.

I don't see anything about that in the gawk manual. Do you have a
reference?

For myself, if a problem isn't strictly line-oriented I generally treat
that as a sign that I should use something other than awk, but I'm sure
it's more powerful than what I'm aware of.

(There's a comp.lang.awk newsgroup, so perhaps this isn't quite topical.)

>> On the other hand, there's more than one way to describe the problem.
>
> Sure.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips
void Void(void) { Void(); } /* The recursive call of the void */

On 11.07.2021 05:41, Keith Thompson wrote:
> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>> On 11.07.2021 01:55, Keith Thompson wrote:
>>> I thought of using C because I found it clearer to think of the input
>>> and output as character-oriented, not line-oriented.
>>
>> Definitely.
>>
>>> awk processing is line-oriented.
>>
>> Unless you use GNU Awk with a "stream processing" setup.
>
> I don't see anything about that in the gawk manual. Do you have a
> reference?

I don't recall whether it was mentioned in Arnold's Awk book and/or
also discussed in comp.lang.awk, and myself I rarely used it and it
was decades ago. A quick test (from memory) shows how to activate
that feature, by setting RS to NUL. The following script

awk 'BEGIN{RS="\0"} END{print NR}'

will print 1 (for non-empty files), meaning that it has processed one
record.

> For myself, if a problem isn't strictly line-oriented I generally treat
> that as a sign that I should use something other than awk, but I'm sure
> it's more powerful than what I'm aware of.

Actually you can expect that GNU Awk supports quite some features that
the Awk standard doesn't, and quite useful features, like allowing RS
to be a regular expression (just for a prominent and important example).

Given GNU Awk's widespread availability (on many platforms), and being
open source, and being still actively supported and developed, makes it
effectively the quasi standard tool for awk.

Mind that here in comp.unix.shell there's regularly code posted that
contains bash-specifcs, often without this fact being mentioned. I
think it should always be mentioned when non-standard constructs are
(necessarily or unnecessarily) used, so folks can decide whether the
solution is suited or not, and the same (IMO to a lesser degree,
because of being a quasi-standard) is true for GNU Awk.

> (There's a comp.lang.awk newsgroup, so perhaps this isn't quite topical.)

Awk solutions are fine here. For specific Awk language or tool specific
discussions the awk newsgroup might be better suited. Though I don't
see that we went into any gory awk details here with our discussion.

Janis

In article <scehh5$76d$1@news-1.m-online.net>,
Janis Papanagnou <janis_papanagnou@hotmail.com> wrote:
....
>>> Unless you use GNU Awk with a "stream processing" setup.
>>
>> I don't see anything about that in the gawk manual. Do you have a
>> reference?
>
>I don't recall whether it was mentioned in Arnold's Awk book and/or
>also discussed in comp.lang.awk, and myself I rarely used it and it
>was decades ago.

I don't think "stream processing" is a GAWK "thing". I think you are
mis-remembering that (to put it kindly).

From what you say below, I am going to guess that what you are attempting
to evoke is the idea of reading in a whole file at once, and then
processing through the buffer. Sort of like what you might do in a
low-level language like C.

>A quick test (from memory) shows how to activate
>that feature, by setting RS to NUL. The following script
>
> awk 'BEGIN{RS="\0"} END{print NR}'

This doesn't do what you think it does. In fact, setting RS to a NUL
character is perfectly legitimate and will do exactly that. This can be
useful when processing the /proc/*/environ files in Linux. E.g.,

gawk 'BEGIN { RS = "\0" } { print NR,$0 }' /proc/self/environ | less

>will print 1 (for non-empty files), meaning that it has processed one
>record.

Only if there are no actual nulls in the file.

Now, what I *think* you are going for is that if you set RS="^$", then it
*is* guaranteed to never match. This has been discussed in the newsgroup
(comp.lang.awk), and has been codified as the following include file
(readfile.awk), which I have in my "awksrc" directory (accessed via the
AWKPATH environment variable) on all of my systems:

--- Cut Here ---
# readfile.awk --- read an entire file at once
# # Original idea by Denis Shirokov, cosmogen@gmail.com, April 2013
#

function readfile(file, tmp, save_rs)
{ save_rs = RS
RS = "^$"
getline tmp < file
close(file)
RS = save_rs

return tmp
} --- Cut Here ---

This could be used to solve OP's problem (assuming we ever actually figure
out what OP's problem is), assuming the file isn't too large.

P.S. I posted on this thread a few days ago. To date, mine is the only
actual solution to OP's problem as originally described. Everything else
(i.e., all the other posts on this thread) has been noise.

--
BigBusiness types (aka, Republicans/Conservatives/Independents/Liberatarians/whatevers)
don't hate big government. They *love* big government as a means for them to get
rich, sucking off the public teat. What they don't like is *democracy* - you know,
like people actually having the right to vote and stuff like that.

On 11.07.2021 13:55, Kenny McCormack wrote:
> In article <scehh5$76d$1@news-1.m-online.net>,
> Janis Papanagnou <janis_papanagnou@hotmail.com> wrote:
> ...
>>>> Unless you use GNU Awk with a "stream processing" setup.
>>>
>>> I don't see anything about that in the gawk manual. Do you have a
>>> reference?
>>
>> I don't recall whether it was mentioned in Arnold's Awk book and/or
>> also discussed in comp.lang.awk, and myself I rarely used it and it
>> was decades ago.
>
> I don't think "stream processing" is a GAWK "thing". I think you are
> mis-remembering that (to put it kindly).

It is, strictly, not stream processing and that's why I wrote "stream
processing" (in quotes). But I don't think I am misremembering because
it serves the intended purpose, which was to regard the various line
separator characters in the text file as part of the data (as opposed
to being record/line separators).

>
> From what you say below, I am going to guess that what you are attempting
> to evoke is the idea of reading in a whole file at once, and then
> processing through the buffer.

Effectively, yes.

> Sort of like what you might do in a low-level language like C.

Not quite. In C I usually do real stream processing, not filling the
whole file in the record buffer (in an array).

>
>> A quick test (from memory) shows how to activate
>> that feature, by setting RS to NUL. The following script
>>
>> awk 'BEGIN{RS="\0"} END{print NR}'
>
> This doesn't do what you think it does.

It always did what I thought it would do. (What do you think I have
thought it would do?)

> In fact, setting RS to a NUL
> character is perfectly legitimate and will do exactly that. This can be
> useful when processing the /proc/*/environ files in Linux. E.g.,
>
> gawk 'BEGIN { RS = "\0" } { print NR,$0 }' /proc/self/environ | less
>
>> will print 1 (for non-empty files), meaning that it has processed one
>> record.
>
> Only if there are no actual nulls in the file.

We have been considering text files in this thread (not binary). Myself
I have never considered awk to be a binary file processor, but if one
wants to use it that way I'm of course fine with it.

>
> Now, what I *think* you are going for is that if you set RS="^$", then it
> *is* guaranteed to never match.

This is an alternative (and works for your binary file case as well).

Janis

> [...]

gazelle@shell.xmission.com (Kenny McCormack) writes:

> P.S. I posted on this thread a few days ago. To date, mine is the only
> actual solution to OP's problem as originally described. Everything else
> (i.e., all the other posts on this thread) has been noise.

Including the three the OP said worked well? Odd. I thought your
suggestion was intended as a joke (at the OP's expense) but maybe you
were being serious.

--
Ben.

On 11.07.2021 05:41, Keith Thompson wrote:
>> Unless you use GNU Awk with a [sort of] "stream processing" setup.
>
> I don't see anything about that in the gawk manual. Do you have a
> reference?

It's indeed mentioned in Arnold Robbin's Book and also in the GNU Awk
manual online (see chapter "Record Splitting with gawk" for "\0").
https://www.gnu.org/software/gawk/manual/gawk.html

The current manual version mentions also "^$" in the context of a
getline based readfile function
https://www.gnu.org/software/gawk/manual/gawk.html#Readfile-Function

And finally there's mention of an extension library function to read
an entire file at once
https://www.gnu.org/software/gawk/manual/gawk.html#Extension-Sample-Readfile

Janis

In article <sch1vd$u3m$1@news-1.m-online.net>,
Janis Papanagnou <janis_papanagnou@hotmail.com> wrote:
>On 11.07.2021 05:41, Keith Thompson wrote:
>>> Unless you use GNU Awk with a [sort of] "stream processing" setup.
>>
>> I don't see anything about that in the gawk manual. Do you have a
>> reference?
>
>It's indeed mentioned in Arnold Robbin's Book and also in the GNU Awk
>manual online (see chapter "Record Splitting with gawk" for "\0").
>https://www.gnu.org/software/gawk/manual/gawk.html

As I've already explained to you, there is absolutely nothing special or
unique about setting RS = "\0".

It works entirely as expected. I even gave you an example of using it to
parse Linux proc "environ" files. You can also use it to parse the
"cmdline" file. There are other files in /proc as well that are delimited
by null characters.

--
If you ask a Trumper who is to blame for the debacle of Jan 6, they will almost certainly say
something about Antifa/BLM/something/whatever. This shows just how screwed up they are; they can't
even get their narrative straight. What they *should* say is "Eugene Goodman". If not for him, the plot
would probably have succeeded, so he (Eugene) is clearly to blame for the failure.

On 12.07.2021 12:23, Kenny McCormack wrote:
> In article <sch1vd$u3m$1@news-1.m-online.net>,
> Janis Papanagnou <janis_papanagnou@hotmail.com> wrote:
>> On 11.07.2021 05:41, Keith Thompson wrote:
>>>> Unless you use GNU Awk with a [sort of] "stream processing" setup.
>>>
>>> I don't see anything about that in the gawk manual. Do you have a
>>> reference?
>>
>> It's indeed mentioned in Arnold Robbin's Book and also in the GNU Awk
>> manual online (see chapter "Record Splitting with gawk" for "\0").
>> https://www.gnu.org/software/gawk/manual/gawk.html
>
> As I've already explained to you, there is absolutely nothing special or
> unique about setting RS = "\0".

Depends on your mental image of "special". Special is that it's not
portably working; it's even explained in the GNU Awk manual IIRC,
since you cannot expect awk's to not fail when using C strings as an
Awk string implementation. In GNU Awk, OTOH, it is _technically_ not
special, but it had been mentioned in the past that it can be used
to achieve what we're talking about here in this thread. That's all.
Is that so hard to accept?

I'm not sure what your problem is. Are you trying to teach me how to
breathe? Do you feel to be misunderstood by me?

>
> It works entirely as expected.

Yes. I also wrote upthread "It always did what I thought it would do."
(in response to your "This doesn't do what you think it does." hubris).

> I even gave you an example of using it to
> parse Linux proc "environ" files. You can also use it to parse the
> "cmdline" file. There are other files in /proc as well that are delimited
> by null characters.

So what? I know all that. Repeating your trivial statements is nothing
but unnecessary noise (to use your habitual style of word choices).

Janis

On 7/11/2021 5:35 AM, Janis Papanagnou wrote:
<snip>
> The following script
>
> awk 'BEGIN{RS="\0"} END{print NR}'
>
> will print 1 (for non-empty files), meaning that it has processed one
> record.

Assuming gawk, use RS="^$" instead of RS="\0". An input file could
contain NULs and still be processed by gawk (but not some other awks as
it's then not a valid POSIX text file), but "^$" only matches an empty
file and therefore cannot match any string in a non-empty file.

Ed.

In article <schean$k7q$1@dont-email.me>,
Ed Morton <mortonspam@gmail.com> wrote:
>On 7/11/2021 5:35 AM, Janis Papanagnou wrote:
><snip>
>> The following script
>>
>> awk 'BEGIN{RS="\0"} END{print NR}'
>>
>> will print 1 (for non-empty files), meaning that it has processed one
>> record.
>
>Assuming gawk, use RS="^$" instead of RS="\0". An input file could
>contain NULs and still be processed by gawk (but not some other awks as
>it's then not a valid POSIX text file), but "^$" only matches an empty
>file and therefore cannot match any string in a non-empty file.

Like a stopped clock...

--
People who want to share their religious views with you
almost never want you to share yours with them. -- Dave Barry

Hi Ed,

since you killfiled a prominent member you obviously haven't seen all
replies.

On 12.07.2021 14:59, Ed Morton wrote:
> On 7/11/2021 5:35 AM, Janis Papanagnou wrote:
> <snip>
>> The following script
>>
>> awk 'BEGIN{RS="\0"} END{print NR}'
>>
>> will print 1 (for non-empty files), meaning that it has processed one
>> record.
>
> Assuming gawk, use RS="^$" instead of RS="\0".

That has already been suggested...

> An input file could
> contain NULs and still be processed by gawk (but not some other awks as
> it's then not a valid POSIX text file), but "^$" only matches an empty
> file and therefore cannot match any string in a non-empty file.

....and I replied that we're considering text files here (no NULs).

Seen from a wider perspective all the options are arguable and depend
on the constraints and tools used.

This has actually all been already disputed in this newsgroup 8 years
ago (and the discussion was spanning many months).

To quote one of my years old replies (just as an example, but it may
also answer another posters vigorous attack) from that old thread:

A '\0' character had never been a reliable separator in case of
binary files. And WRT text files using RS=SUBSEP would be even
better than the suggested RS="\0" in cases when your programs
shall run on other (and older) awks as well.

There are yet more aspects to consider. At that time you posted (also
just for another example) to use "\n$" in certain application contexts.
Another poster suggested an equivalent of RS=".*" (IIRC). And so on.
In short; what fits best depends.

The "^$" is very appealing because you can use it for binary files as
well (although I'm using awk as _text_-processor). There's a few things
I don't like much with it, though. Similar to "\0" it is non-portable.
It's more cryptic, no one seems to be perfectly sure how it works - at
least that was the expression I've got from the discussion 8 years ago,
and also now I immediately start a test case to see whether it matches
an empty line (just to have some confidence). Because of that fact I
think it should explicitly be documented and supported by a rationale
that explains the _mechanism_. (Of course there could also just be a
statement that it's sort of a special pattern that works "magically",
so that no [technical] explanation needs to be formulated.) The GNU
Awk manual has the topic incoherently spread across three chapters.
It would certainly be helpful to have a more coherent picture and a
guidance or suggestion (with all the caveats, e.g. about what happens
with a RS="^$" statement in other awks). There have also performance
issues been addressed by Arnold in the past, I think that the @load
option is the fastest because it bypasses the regexp processing,
which would aid the user to make an informed choice what to use when.

Until we have such a "directive" I fear we'll repeat our discussions
every couple years again, and they seem to not be quickly terminated
discussions on every re-iteration. ;-)

Janis

>
> Ed.

Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:

> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>> On 11.07.2021 01:55, Keith Thompson wrote:
>>> I thought of using C because I found it clearer to think of the input
>>> and output as character-oriented, not line-oriented.
>>
>> Definitely.
>>
>>> awk processing is line-oriented.
>>
>> Unless you use GNU Awk with a "stream processing" setup.
>
> I don't see anything about that in the gawk manual. Do you have a
> reference?
>
> For myself, if a problem isn't strictly line-oriented I generally treat
> that as a sign that I should use something other than awk, but I'm sure
> it's more powerful than what I'm aware of.

You can write something like your C solution using gawk if you set
RS="().". The ()s are needed to make RS longer than one character, but
having done that, every character is then available, one by one, in RT.

Something like (IIRC)

awk 'BEGIN{RS="()."} p!="\r" || RT!="\n" {printf RT} {p=RT}'

That's how I'd interpret doing "stream processing" in GAWK.

--
Ben.

On 7/12/2021 9:25 AM, Janis Papanagnou wrote:
<snip>
> The "^$" is very appealing because you can use it for binary files as
> well (although I'm using awk as _text_-processor). There's a few things
> I don't like much with it, though. Similar to "\0" it is non-portable.
> It's more cryptic, no one seems to be perfectly sure how it works

I don't know why it wouldn't just work like any other RS. gawk looks for
where a string matching that regexp occurs in the file and uses that to
identify the end of a record. If no such string exists in the file the
whole file is stored in $0. Since "^" means "start-of-string" and "$"
means "end-of-string" the only possible way that "^$" could match a
string in a file would be if there was nothing between the start and end
of the file, i.e. the file is empty. Using RS='^$' for any file is no
different than using RS='foo' when foo doesn't exist in the file.

Ed.

In article <schmss$lpb$1@dont-email.me>,
Ed Morton <mortonspam@gmail.com> wrote:
>On 7/12/2021 9:25 AM, Janis Papanagnou wrote:
><snip>
>> The "^$" is very appealing because you can use it for binary files as
>> well (although I'm using awk as _text_-processor). There's a few things
>> I don't like much with it, though. Similar to "\0" it is non-portable.
>> It's more cryptic, no one seems to be perfectly sure how it works
>
>I don't know why it wouldn't just work like any other RS. gawk looks for
>where a string matching that regexp occurs in the file and uses that to
>identify the end of a record. If no such string exists in the file the
>whole file is stored in $0. Since "^" means "start-of-string" and "$"
>means "end-of-string" the only possible way that "^$" could match a
>string in a file would be if there was nothing between the start and end
>of the file, i.e. the file is empty. Using RS='^$' for any file is no
>different than using RS='foo' when foo doesn't exist in the file.

This is not your "stopped clock" moment, I'm afraid.

--
The randomly chosen signature file that would have appeared here is more than 4
lines long. As such, it violates one or more Usenet RFCs. In order to remain
in compliance with said RFCs, the actual sig can be found at the following URL:
http://user.xmission.com/~gazelle/Sigs/Security

On 12.07.2021 17:25, Ed Morton wrote:
> On 7/12/2021 9:25 AM, Janis Papanagnou wrote:
>> It's more cryptic, no one seems to be perfectly sure how it works
>
> I don't know why it wouldn't just work like any other RS.

The responses in that other thread left a different impression to me,
that it wasn't obvious or as clear as it should be.

> gawk looks for
> where a string matching that regexp occurs in the file and uses that to
> identify the end of a record. If no such string exists in the file the
> whole file is stored in $0.

If all we want is a pattern that is principally non-existing wouldn't
it be clearer to use something like "$^" (i.e. "^$" reversed), which
is a meta-character sequence that does obviously not make any sense.

If there's a meta-character sequence that may make sense in a way that
you have to think about what constitutes a string in a context where
we are just in the process to define the parsing unit string by setting
RS opens room for confusion. YMMV.

It seems to me that an explanation of "$^" (one that cannot exist per
definition of the meaning of '^' and '$') is clearer than the version
that requires a more verbose explanation like the one here (that had
in this or similar form also already been given 8 years ago, IIRC):

> Since "^" means "start-of-string" and "$"
> means "end-of-string" the only possible way that "^$" could match a
> string in a file would be if there was nothing between the start and end
> of the file, i.e. the file is empty. Using RS='^$' for any file is no
> different than using RS='foo' when foo doesn't exist in the file.
>
> Ed.

Janis

On 7/12/2021 9:25 AM, Janis Papanagnou wrote:
> Hi Ed,
>
> since you killfiled a prominent member you obviously haven't seen all
> replies.

The only person I have killfiled is Kenny and that's well-deserved. If
he's yammering away somewhere in this thread I'm sure that, as always,
it'll be wrong and/or redundant and/or infantile name-calling so I know
I'm not missing anything there.

Actually I may still have that other well-know netkook Alan Conor aka
Tom Newton killfiled too but I haven't seen him post anything in years
so I doubt if he's who you're referring to.

Ed.

On 7/12/2021 3:49 PM, Janis Papanagnou wrote:
> On 12.07.2021 17:25, Ed Morton wrote:
>> On 7/12/2021 9:25 AM, Janis Papanagnou wrote:
>>> It's more cryptic, no one seems to be perfectly sure how it works
>>
>> I don't know why it wouldn't just work like any other RS.
>
> The responses in that other thread left a different impression to me,
> that it wasn't obvious or as clear as it should be.

I don't recall the previous conversation on it you're referring to but
it seems very clear and simple to me.

>> gawk looks for
>> where a string matching that regexp occurs in the file and uses that to
>> identify the end of a record. If no such string exists in the file the
>> whole file is stored in $0.
>
> If all we want is a pattern that is principally non-existing wouldn't
> it be clearer to use something like "$^" (i.e. "^$" reversed), which
> is a meta-character sequence that does obviously not make any sense.

`$^` means the literal chars `$` then `^` since `$` is only an anchor
metachar at the end of a regexp or subexpression and `^` only at the
beginning. Look:

$ echo 'a$^b' | grep '$^'
a$^b

See
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_03_08
for more info.

Regards,

Ed.

On 12.07.2021 22:52, Ed Morton wrote:
> On 7/12/2021 9:25 AM, Janis Papanagnou wrote:
>> Hi Ed,
>>
>> since you killfiled a prominent member you obviously haven't seen all
>> replies.
>
> The only person I have killfiled is Kenny and [...]

And, IIRC, he's the one that suggested to use "^$".
(Or was it that he suggested to not use "\0"?)
Anyway...

On 2021-07-12, Janis Papanagnou <janis_papanagnou@hotmail.com> wrote:
> On 12.07.2021 17:25, Ed Morton wrote:
>> On 7/12/2021 9:25 AM, Janis Papanagnou wrote:
>>> It's more cryptic, no one seems to be perfectly sure how it works
>>
>> I don't know why it wouldn't just work like any other RS.
>
> The responses in that other thread left a different impression to me,
> that it wasn't obvious or as clear as it should be.
>
>> gawk looks for
>> where a string matching that regexp occurs in the file and uses that to
>> identify the end of a record. If no such string exists in the file the
>> whole file is stored in $0.
>
> If all we want is a pattern that is principally non-existing wouldn't
> it be clearer to use something like "$^" (i.e. "^$" reversed), which
> is a meta-character sequence that does obviously not make any sense.

Isn't $^ something that occurs at the end of every line? (End of this
line, beginning of the next)

Actually, I also have problems with ^$ since that would seem to mean and
empty line, which is certainly possible (LFLF) would seem to have an
empty line in it. But clearly I would have to know EXACTLU how awk
determines the start and end of a line.

>
> If there's a meta-character sequence that may make sense in a way that
> you have to think about what constitutes a string in a context where
> we are just in the process to define the parsing unit string by setting
> RS opens room for confusion. YMMV.
>
> It seems to me that an explanation of "$^" (one that cannot exist per
> definition of the meaning of '^' and '$') is clearer than the version
> that requires a more verbose explanation like the one here (that had
> in this or similar form also already been given 8 years ago, IIRC):
>
>> Since "^" means "start-of-string" and "$"
>> means "end-of-string" the only possible way that "^$" could match a
>> string in a file would be if there was nothing between the start and end
>> of the file, i.e. the file is empty. Using RS='^$' for any file is no
>> different than using RS='foo' when foo doesn't exist in the file.
>>
>> Ed.
>
> Janis
>

On 12.07.2021 22:59, Ed Morton wrote:
> On 7/12/2021 3:49 PM, Janis Papanagnou wrote:
>> On 12.07.2021 17:25, Ed Morton wrote:
>>> On 7/12/2021 9:25 AM, Janis Papanagnou wrote:
>>>> It's more cryptic, no one seems to be perfectly sure how it works
>>>
>>> I don't know why it wouldn't just work like any other RS.
>>
>> The responses in that other thread left a different impression to me,
>> that it wasn't obvious or as clear as it should be.
>
> I don't recall the previous conversation on it you're referring to but
> it seems very clear and simple to me.
>
>>> gawk looks for
>>> where a string matching that regexp occurs in the file and uses that to
>>> identify the end of a record. If no such string exists in the file the
>>> whole file is stored in $0.
>>
>> If all we want is a pattern that is principally non-existing wouldn't
>> it be clearer to use something like "$^" (i.e. "^$" reversed), which
>> is a meta-character sequence that does obviously not make any sense.
>
> `$^` means the literal chars `$` then `^` since `$` is only an anchor
> metachar at the end of a regexp or subexpression and `^` only at the
> beginning. Look:
>
> $ echo 'a$^b' | grep '$^'
> a$^b

We were talking about GNU Awk, don't we?

Consequently I have tested GNU Awk:

$ echo $'a$^b\na$^b' | awk 'BEGIN{RS="$^"}{print NR, $0}'
1 a$^b
a$^b

Janis

>
> See
> https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_03_08
> for more info.
>
> Regards,
>
> Ed.

On 12.07.2021 23:06, William Unruh wrote:
> On 2021-07-12, Janis Papanagnou <janis_papanagnou@hotmail.com> wrote:
>> On 12.07.2021 17:25, Ed Morton wrote:
>>> On 7/12/2021 9:25 AM, Janis Papanagnou wrote:
>>>> It's more cryptic, no one seems to be perfectly sure how it works
>>>
>>> I don't know why it wouldn't just work like any other RS.
>>
>> The responses in that other thread left a different impression to me,
>> that it wasn't obvious or as clear as it should be.
>>
>>> gawk looks for
>>> where a string matching that regexp occurs in the file and uses that to
>>> identify the end of a record. If no such string exists in the file the
>>> whole file is stored in $0.
>>
>> If all we want is a pattern that is principally non-existing wouldn't
>> it be clearer to use something like "$^" (i.e. "^$" reversed), which
>> is a meta-character sequence that does obviously not make any sense.
>
> Isn't $^ something that occurs at the end of every line? (End of this
> line, beginning of the next)

Well, on close look it may have a similar interpretation problem (but
probably to a lesser degree?).

>
> Actually, I also have problems with ^$ since that would seem to mean and
> empty line, which is certainly possible (LFLF) would seem to have an
> empty line in it. But clearly I would have to know EXACTLU how awk
> determines the start and end of a line.

I think it boils down to have a good documentation and/or guidance in
the manual. (Then these threads could be significantly shortened. ;-)

Janis

On Mon, 12 Jul 2021 16:25:30 +0200
Janis Papanagnou <janis_papanagnou@hotmail.com> wrote:

[On AWK dark corners.]
> The GNU
> Awk manual has the topic incoherently spread across three chapters.
> It would certainly be helpful to have a more coherent picture and a
> guidance or suggestion (with all the caveats, e.g. about what happens
> with a RS="^$" statement in other awks). There have also performance
> issues been addressed by Arnold in the past, I think that the @load
> option is the fastest because it bypasses the regexp processing,
> which would aid the user to make an informed choice what to use when.
>
> Until we have such a "directive" I fear we'll repeat our discussions
> every couple years again, and they seem to not be quickly terminated
> discussions on every re-iteration. ;-)

The only person who can provide authoritative answers is Arnold Robbins. I
think he reads comp.lang.awk but I'm not sure if he reads comp.unix.shell
so you should have crossposted to comp.lang.awk (which I've done). As I'm
typing this , I can see that several more posts have been made in the thread
discussing esoteric issues regarding AWK .It would have served everyone best
if these also appeared on comp.lang.awk .