On 7/12/2021 4:10 PM, Janis Papanagnou wrote:
> On 12.07.2021 22:59, Ed Morton wrote:
>> On 7/12/2021 3:49 PM, Janis Papanagnou wrote:
>>> On 12.07.2021 17:25, Ed Morton wrote:
>>>> On 7/12/2021 9:25 AM, Janis Papanagnou wrote:
>>>>> It's more cryptic, no one seems to be perfectly sure how it works
>>>>
>>>> I don't know why it wouldn't just work like any other RS.
>>>
>>> The responses in that other thread left a different impression to me,
>>> that it wasn't obvious or as clear as it should be.
>>
>> I don't recall the previous conversation on it you're referring to but
>> it seems very clear and simple to me.
>>
>>>> gawk looks for
>>>> where a string matching that regexp occurs in the file and uses that to
>>>> identify the end of a record. If no such string exists in the file the
>>>> whole file is stored in $0.
>>>
>>> If all we want is a pattern that is principally non-existing wouldn't
>>> it be clearer to use something like "$^" (i.e. "^$" reversed), which
>>> is a meta-character sequence that does obviously not make any sense.
>>
>> `$^` means the literal chars `$` then `^` since `$` is only an anchor
>> metachar at the end of a regexp or subexpression and `^` only at the
>> beginning. Look:
>>
>> $ echo 'a$^b' | grep '$^'
>> a$^b
>
> We were talking about GNU Awk, don't we?

Yeah, but there's no magic about GNU awks regexp handling. My mistake
was assuming BREs and EREs were the same in regard to anchors.

>
> Consequently I have tested GNU Awk:
>
> $ echo $'a$^b\na$^b' | awk 'BEGIN{RS="$^"}{print NR, $0}'
> 1 a$^b
> a$^b

I stand corrected, apparently that's a difference between BREs and EREs:

$ echo 'a$^b' | grep '$^'
a$^b

$ echo 'a$^b' | grep -E '$^'
$

$ echo 'a$^b' | sed 's/$^/X/'
aXb

$ echo 'a$^b' | sed -E 's/$^/X/'
a$^b

See
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_04_09
which I thought was saying you could use any character before the `^`
and it wouldn't match which was supported by this test:

$ printf 'ax^b\nax^b\n' | awk 'BEGIN{RS="x^"}{print NR, $0}'
1 ax^b
ax^b

but then I can't explain this which is apparently just ignoring the RS
setting:

$ printf 'a.^b\na.^b\n' | awk 'BEGIN{RS=".^"}{print NR, $0}'
1 a.^b
2 a.^b

Regards,

Ed.

On 7/12/2021 4:06 PM, William Unruh wrote:
> On 2021-07-12, Janis Papanagnou <janis_papanagnou@hotmail.com> wrote:
>> On 12.07.2021 17:25, Ed Morton wrote:
>>> On 7/12/2021 9:25 AM, Janis Papanagnou wrote:
>>>> It's more cryptic, no one seems to be perfectly sure how it works
>>>
>>> I don't know why it wouldn't just work like any other RS.
>>
>> The responses in that other thread left a different impression to me,
>> that it wasn't obvious or as clear as it should be.
>>
>>> gawk looks for
>>> where a string matching that regexp occurs in the file and uses that to
>>> identify the end of a record. If no such string exists in the file the
>>> whole file is stored in $0.
>>
>> If all we want is a pattern that is principally non-existing wouldn't
>> it be clearer to use something like "$^" (i.e. "^$" reversed), which
>> is a meta-character sequence that does obviously not make any sense.
>
> Isn't $^ something that occurs at the end of every line? (End of this
> line, beginning of the next)

You're mixing up string and lines and records. `^` means start of a
string and `$` means the end of string.

In a line-oriented tool like grep or sed (without the GNU -z option and
without using a hold space), the string in question is the line that was
just read into memory and so `^` and `$` can be used to find the
start/end of the line because the line in question is the whole string.

In a record-oriented tool like awk, when used with the default RS of
`\n`, the `^` and `$` can be used the same way as in sed or grep, but
when used with a different RS the record can contain newlines so the `^`
and `$` do not match the start and end of lines any more, they match the
start and end of the record. If you use an RS that can't exist in the
input then the whole input file is the record and so `^` matches the
start of the input file while `$` matches the end of the input file.

So no, `$^` does not occur anywhere in any input if we're assuming `$`
and `^` to be anchor metachars in that expression (which they apparently
are when using an ERE such as awk uses).

>
> Actually, I also have problems with ^$ since that would seem to mean and
> empty line,

No, it's an empty string which could be a line given some RS values or
it could be an empty record that's part of a file given other RS values
or it could be an empty file given yet other RS values.

which is certainly possible (LFLF) would seem to have an
> empty line in it. But clearly I would have to know EXACTLU how awk
> determines the start and end of a line.

A line starts with the character following `^` or the previous `\n`. A
line ends with the character before `$` or the next `\n`.

Ed.

On 7/12/2021 5:06 PM, Ed Morton wrote:
> On 7/12/2021 4:06 PM, William Unruh wrote:
>> On 2021-07-12, Janis Papanagnou <janis_papanagnou@hotmail.com> wrote:
>>> On 12.07.2021 17:25, Ed Morton wrote:
>>>> On 7/12/2021 9:25 AM, Janis Papanagnou wrote:
>>>>> It's more cryptic, no one seems to be perfectly sure how it works
>>>>
>>>> I don't know why it wouldn't just work like any other RS.
>>>
>>> The responses in that other thread left a different impression to me,
>>> that it wasn't obvious or as clear as it should be.
>>>
>>>> gawk looks for
>>>> where a string matching that regexp occurs in the file and uses that to
>>>> identify the end of a record. If no such string exists in the file the
>>>> whole file is stored in $0.
>>>
>>> If all we want is a pattern that is principally non-existing wouldn't
>>> it be clearer to use something like "$^" (i.e. "^$" reversed), which
>>> is a meta-character sequence that does obviously not make any sense.
>>
>> Isn't $^ something that occurs at the end of every line? (End of this
>> line, beginning of the next)
>
> You're mixing up string and lines and records. `^` means start of a
> string and `$` means the end of string.
>
> In a line-oriented tool like grep or sed (without the GNU -z option and
> without using a hold space), the string in question is the line that was
> just read into memory and so `^` and `$` can be used to find the
> start/end of the line because the line in question is the whole string.
>
> In a record-oriented tool like awk, when used with the default RS of
> `\n`, the `^` and `$` can be used the same way as in sed or grep, but
> when used with a different RS the record can contain newlines so the `^`
> and `$` do not match the start and end of lines any more, they match the
> start and end of the record. If you use an RS that can't exist in the
> input then the whole input file is the record and so `^` matches the
> start of the input file while `$` matches the end of the input file.
>
> So no, `$^` does not occur anywhere in any input if we're assuming `$`
> and `^` to be anchor metachars in that expression (which they apparently
> are when using an ERE such as awk uses).
>
>>
>> Actually, I also have problems with ^$ since that would seem to mean and
>> empty line,
>
> No, it's an empty string which could be a line given some RS values or
> it could be an empty record that's part of a file given other RS values
> or it could be an empty file given yet other RS values.
>
> which is certainly possible (LFLF) would seem to have an
>> empty line in it. But clearly I would have to know EXACTLU how awk
>> determines the start and end of a line.
>
> A line starts with the character following `^` or the previous `\n`. A
> line ends with the character before `$` or the next `\n`.
>
>     Ed.

Here's some examples showing what's matched between `^` and `$` when the
string in memory (the current awk record) is a line, a multi-line
paragraph, and a whole file, all based on the value of `RS`:

The sample input (courtesy of Robert Burns "Tam O'Shanter" written/set
near my home town):

$ cat file
Ah, gentle dames! it gars me greet,
To think how mony counsels sweet,

How mony lengthen'd, sage advices,
The husband frae the wife despises!

Read 1 line at a time:

$ awk 'match($0,/^.*$/) { print "<" substr($0,RSTART,RLENGTH) ">"
}' file
<Ah, gentle dames! it gars me greet,>
<To think how mony counsels sweet,>
<>
<How mony lengthen'd, sage advices,>
<The husband frae the wife despises!>

Read 1 paragraph at a time:

$ awk -v RS='' 'match($0,/^.*$/) { print "<"
substr($0,RSTART,RLENGTH) ">" }' file
<Ah, gentle dames! it gars me greet,
To think how mony counsels sweet,>
<How mony lengthen'd, sage advices,
The husband frae the wife despises!>

Read the whole file at once (RS='^$` could be `RS='anything nonexistent'`):

$ awk -v RS='^$' 'match($0,/^.*$/) { print "<"
substr($0,RSTART,RLENGTH) ">" }' file
<Ah, gentle dames! it gars me greet,
To think how mony counsels sweet,

How mony lengthen'd, sage advices,
The husband frae the wife despises!
>

As you can see `^` and `$` always match the start and end of the record
that awk is currently processing, whether that's a line, or a paragraph,
or a whole file. The only time when `^` and `$` also identify the
start/end of a line is when the whole record is a single line.

To find lines in the multi-line paragraph case, you need to test for
`^|\n` at the start of the lines (`^` to find the start of the first
line, `\n` to find subsequent) and/or `\n|$` at the end of the lines
(`$` to find the end of the last line, `\n` to find previous), depending
on what you want to do with it, and you also need to account for the
fact that if you find `\n` it'll be part of the matching string, unlike
`^` or `$`, e.g.

$ awk -v RS='' 'match($0,/^[^\n]*/) { print "<"
substr($0,RSTART,RLENGTH) ">" }' file
<Ah, gentle dames! it gars me greet,>
<How mony lengthen'd, sage advices,>

$ awk -v RS='' 'match($0,/\n[^\n]*/) { print "<"
substr($0,RSTART+1,RLENGTH-1) ">" }' file
<To think how mony counsels sweet,>
<The husband frae the wife despises!>

Regards,

Ed.

On 12.07.2021 23:55, Ed Morton wrote:
>>
>> $ echo $'a$^b\na$^b' | awk 'BEGIN{RS="$^"}{print NR, $0}'
>> 1 a$^b
>> a$^b
>
> I stand corrected, apparently that's a difference between BREs and EREs:
>
> $ echo 'a$^b' | grep '$^'
> a$^b
>
> $ echo 'a$^b' | grep -E '$^'
> $
>
> $ echo 'a$^b' | sed 's/$^/X/'
> aXb
>
> $ echo 'a$^b' | sed -E 's/$^/X/'
> a$^b
>
> See
> https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_04_09
> which I thought was saying you could use any character before the `^`
> and it wouldn't match which was supported by this test:
>
> $ printf 'ax^b\nax^b\n' | awk 'BEGIN{RS="x^"}{print NR, $0}'
> 1 ax^b
> ax^b
>
> but then I can't explain this which is apparently just ignoring the RS
> setting:
>
> $ printf 'a.^b\na.^b\n' | awk 'BEGIN{RS=".^"}{print NR, $0}'
> 1 a.^b
> 2 a.^b

$ printf 'a.^b\na.^b\n' | awk 'BEGIN{RS="[.]^"}{print NR, $0}'
1 a.^b
a.^b

It seems ".^" had not been "ignored" as RS but interpreted as string?

Janis

>
> Regards,
>
> Ed.

On 13.07.2021 08:05, Janis Papanagnou wrote:
> On 12.07.2021 23:55, Ed Morton wrote:
>>
>> but then I can't explain this which is apparently just ignoring the RS
>> setting:
>>
>> $ printf 'a.^b\na.^b\n' | awk 'BEGIN{RS=".^"}{print NR, $0}'
>> 1 a.^b
>> 2 a.^b
>
> $ printf 'a.^b\na.^b\n' | awk 'BEGIN{RS="[.]^"}{print NR, $0}'
> 1 a.^b
> a.^b
>
> It seems ".^" had not been "ignored" as RS but interpreted as string?

....which still wouldn't explain the outcome of your test case, though.

Hmm..

> Janis
>
>>
>> Regards,
>>
>> Ed.
>

On 7/13/2021 1:39 AM, Janis Papanagnou wrote:
> On 13.07.2021 08:05, Janis Papanagnou wrote:
>> On 12.07.2021 23:55, Ed Morton wrote:
>>>
>>> but then I can't explain this which is apparently just ignoring the RS
>>> setting:
>>>
>>> $ printf 'a.^b\na.^b\n' | awk 'BEGIN{RS=".^"}{print NR, $0}'
>>> 1 a.^b
>>> 2 a.^b
>>
>> $ printf 'a.^b\na.^b\n' | awk 'BEGIN{RS="[.]^"}{print NR, $0}'
>> 1 a.^b
>> a.^b
>>
>> It seems ".^" had not been "ignored" as RS but interpreted as string?
>
> ...which still wouldn't explain the outcome of your test case, though.
>
> Hmm..
>

I talked to Arnold and he thinks it's a bug which he's investigating.
See https://lists.gnu.org/archive/html/bug-gawk/2021-07/msg00026.html.

Ed.