I have an email attachment ATT000001.xml with content like below.
Note that there are two consecutive lines <?xml version="1.0"?>.
How could I use sed or awk to delete one of them?

Note that the two <?xml version="1.0"?> lines causes "strict parse error" when I renamed ATT000001.xml as xxx.xls and tried to open it.
Normally there is only one such <?xml version="1.0"?> line generated in such email attachment; but occasionally two were generated, and gave me trouble opening the attachment.

TIA

--
Content-Disposition: attachment; filename="xxx.xls";
<?xml version="1.0"?>
<?xml version="1.0"?>
<ss:Workbook xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet">
<ss:Styles> <ss:Style ss:ID="1"> <ss:Font ss:Bold="1"/></ss:Style>
<ss:Style ss:ID="s65"> <ss:Alignment ss:Vertical="Top" ss:WrapText="1"/></ss:Style> </ss:Styles>
<ss:Worksheet ss:Name="ESI report"><ss:Table>
<ss:Column ss:Width="100"/>
<ss:Column ss:Width="60"/>
<ss:Column ss:Width="60"/>
<ss:Column ss:Width="70"/>
<ss:Column ss:Width="100"/>
<ss:Column ss:Width="100"/>
<ss:Column ss:Width="140"/>
<ss:Column ss:Width="640"/>
[snip]
--

On 5/26/2021 12:01 AM, Harry wrote:
> I have an email attachment ATT000001.xml with content like below.
> Note that there are two consecutive lines <?xml version="1.0"?>.
> How could I use sed or awk to delete one of them?

awk '!( ($0=="<?xml version=\"1.0\"?>") && seen[$0]++ )' file

Ed.

On Wednesday, May 26, 2021 at 5:05:50 AM UTC-7, Ed Morton wrote:

> awk '!( ($0=="<?xml version=\"1.0\"?>") && seen[$0]++ )' file

Great! It works!

On Tue, 25 May 2021 22:01:28 -0700, Harry wrote:

> I have an email attachment ATT000001.xml with content like below.
> Note that there are two consecutive lines <?xml version="1.0"?>.
> How could I use sed or awk to delete one of them?

Ed Morton supplied an awk solution.

Here's my alternative non-sed, non-awk, solution:
tail -n+2 ATT000001.xml

HTH
--
Lew Pitcher
"In Skills, We Trust"

On 5/26/2021 10:20 AM, Lew Pitcher wrote:
> On Tue, 25 May 2021 22:01:28 -0700, Harry wrote:
>
>> I have an email attachment ATT000001.xml with content like below.
>> Note that there are two consecutive lines <?xml version="1.0"?>.
>> How could I use sed or awk to delete one of them?
>
> Ed Morton supplied an awk solution.
>
> Here's my alternative non-sed, non-awk, solution:
> tail -n+2 ATT000001.xml
>

That would delete the line

Content-Disposition: attachment; filename="xxx.xls";

at the top of the OPs sample input shown in the question (I'm assuming
everything below the "--" in the question is what the OP is working with
as input) and/or delete the one and only

<?xml version="1.0"?>

in those cases where it isn't duplicated (the OP said "occasionally...").

Ed.

On 2021-05-26, Harry <harryooopotter@hotmail.com> wrote:
> I have an email attachment ATT000001.xml with content like below.
> Note that there are two consecutive lines <?xml version="1.0"?>.
> How could I use sed or awk to delete one of them?
>
> Note that the two <?xml version="1.0"?> lines causes "strict parse error" when I renamed ATT000001.xml as xxx.xls and tried to open it.
> Normally there is only one such <?xml version="1.0"?> line generated in such email attachment; but occasionally two were generated, and gave me trouble opening the attachment.
>
> TIA
>
> --
> Content-Disposition: attachment; filename="xxx.xls";
>
><?xml version="1.0"?>
><?xml version="1.0"?>
><ss:Workbook xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet">
><ss:Styles> <ss:Style ss:ID="1"> <ss:Font ss:Bold="1"/></ss:Style>
><ss:Style ss:ID="s65"> <ss:Alignment ss:Vertical="Top" ss:WrapText="1"/></ss:Style> </ss:Styles>
> <ss:Worksheet ss:Name="ESI report"><ss:Table>
> <ss:Column ss:Width="100"/>
> <ss:Column ss:Width="60"/>
> <ss:Column ss:Width="60"/>
> <ss:Column ss:Width="70"/>
> <ss:Column ss:Width="100"/>
> <ss:Column ss:Width="100"/>
> <ss:Column ss:Width="140"/>
> <ss:Column ss:Width="640"/>
> [snip]
> --

txr -c '@(bind x "<?xml version=\"1.0\"?>")
@(repeat)
@ (cases)
@x
@x
@ (bind line x)
@ (or)
@line
@ (end)
@ (do (put-line line))
@(end)' < input > output

Explanation:

@(bind x "<?xml version=\"1.0\"?>")

Bind variable x to the string we are looking for.

@(repeat)

Scan through the lines of input, doing multi-line matches at each line.

@ (cases)

Match multiple cases separated by @(or)

@x
@x

Match two consecutive lines, each of which is identical to x.

@ (bind line x)

If the lines matched bind the line variable to x. This always succeeds.

@ (or)
@line

Otherwise, if the previous case didn't match, just match the current
line, binding it to the line variable.

@ (end)

End of cases construct. We now have a line variable.

@ (do (put-line line))

Output the line variable as a line.

@(end)

End of repeat construct.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal

>>>>> "Harry" == Harry <harryooopotter@hotmail.com> writes:

Harry> I have an email attachment ATT000001.xml with content like below.
Harry> Note that there are two consecutive lines <?xml version="1.0"?>.
Harry> How could I use sed or awk to delete one of them?

Surprised not to see a sed solution yet. sed has a great finite state
machine built in. I've done some truly horrific things with that.

I stared at the sed manual for a few minutes, but can't rememeber enough
of my hackery, but it would basically be 3 states, before seeing the
first one (print it and repeat), seeing the first one (print it, go
to next state), then print anything that's not it.

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Dart/Flutter consulting, Technical writing, Comedy, etc. etc.
Still trying to think of something clever for the fourth line of this .sig

On 2021-05-26, Randal L. Schwartz <merlyn@stonehenge.com> wrote:
>>>>>> "Harry" == Harry <harryooopotter@hotmail.com> writes:
>
>Harry> I have an email attachment ATT000001.xml with content like below.
>Harry> Note that there are two consecutive lines <?xml version="1.0"?>.
>Harry> How could I use sed or awk to delete one of them?
>
> Surprised not to see a sed solution yet. sed has a great finite state
> machine built in. I've done some truly horrific things with that.
>
> I stared at the sed manual for a few minutes, but can't rememeber enough
> of my hackery, but it would basically be 3 states, before seeing the
> first one (print it and repeat), seeing the first one (print it, go
> to next state), then print anything that's not it.

He said that he wanted to eliminate one of two consecutive lines. Not
that line repeated anywhere in the file as I read it. Thus you have to
reset the state if the next line is not the the repeat. This is also
true of the awk suggestions.

>

On 26.05.2021 07:01, Harry wrote:
> I have an email attachment ATT000001.xml with content like below.
> Note that there are two consecutive lines <?xml version="1.0"?>.
> How could I use sed or awk to delete one of them?

One way could be

awk -v p='<?xml version="1.0"?>' '$0!=p, $0==p'

(You may stop reading here.)

First some introductory comments referring to other posts...

Well, there are many ways to skin a cat. We can also make some
assumptions - because it's possible to do so or because we are
guessing - to keep the solutions terse (as Ed did and William
pointed out), or to extend the range of faulty data to be able
to handle by a solution (like the code shown).

The first hint, though, would be to get the process that creates
such wrong data fixed! (Instead of letting arbitrary amounts of
people work around the issue individually.)

We've already seen an awk solution, with a hint in a followup
about a limit it inherently has. But we can also rewrite that
algorithm, since it's so simple to do with awk[*]. We can also
try memorizing how that would be done more cryptically in sed,
were another poster obviously had difficulties to recall.

Just for a variant I want to provide another awk approach[**]

awk -v p='<?xml version="1.0"?>' '$0!=p, $0==p'

which has the property to keep only one of multiple (2+) and
only consecutive lines matching the passed pattern.[***]

Janis

[*] I abstain from posting an update of that awk code.

[**] Probably a bit cryptic as well, though I simply derived it
from another algorithm I suggested a couple years ago to squeeze
multiple empty lines that separate text blocks to a single empty
line: awk 'NF,!NF'

[***] It has an own (here also irrelevant) restriction if the N
pattern(s) appear as the first line(s) of the files.

> Note that the two <?xml version="1.0"?> lines causes "strict parse
> error" when I renamed ATT000001.xml as xxx.xls and tried to open it.
> Normally there is only one such <?xml version="1.0"?> line generated
> in such email attachment; but occasionally two were generated, and gave
> me trouble opening the attachment.
>
> TIA
>
> --
> Content-Disposition: attachment; filename="xxx.xls";
>
> <?xml version="1.0"?>
> <?xml version="1.0"?>
> <ss:Workbook xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet">
> <ss:Styles> <ss:Style ss:ID="1"> <ss:Font ss:Bold="1"/></ss:Style>
> <ss:Style ss:ID="s65"> <ss:Alignment ss:Vertical="Top" ss:WrapText="1"/></ss:Style> </ss:Styles>
> <ss:Worksheet ss:Name="ESI report"><ss:Table>
> <ss:Column ss:Width="100"/>
> <ss:Column ss:Width="60"/>
> <ss:Column ss:Width="60"/>
> <ss:Column ss:Width="70"/>
> <ss:Column ss:Width="100"/>
> <ss:Column ss:Width="100"/>
> <ss:Column ss:Width="140"/>
> <ss:Column ss:Width="640"/>
> [snip]
> --
>

>>>>> "William" == William Unruh <unruh@invalid.ca> writes:

William> On 2021-05-26, Randal L. Schwartz <merlyn@stonehenge.com> wrote:

William> He said that he wanted to eliminate one of two consecutive lines. Not
William> that line repeated anywhere in the file as I read it. Thus you have to
William> reset the state if the next line is not the the repeat. This is also
William> true of the awk suggestions.

Still sounds like a fun sed challenge. One of the things I remember
creating was a backslash-at-end-of-line folding action.

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Dart/Flutter consulting, Technical writing, Comedy, etc. etc.
Still trying to think of something clever for the fourth line of this .sig

On Wednesday, May 26, 2021 at 1:12:09 PM UTC-7, Janis Papanagnou wrote:

> Just for a variant I want to provide another awk approach[**]
>
> awk -v p='<?xml version="1.0"?>' '$0!=p, $0==p'
>
> which has the property to keep only one of multiple (2+) and
> only consecutive lines matching the passed pattern.[***]
>
> Janis

Nice solution too.
I tried with my file having two set of 2 consecutive <?xml version="1.0"?> lines,
and this solution eliminates the two duplicated lines (one for each set).

In my particular situation, there won't be two set of consecutive <?xml version="1.0"?> lines.

Just make the story short, the programmer was writing two consecutive <?xml version="1.0"?> lines in his program, and said that sendmail would eat up one of them (as a sendmail usual behaviour).
And occasionally, this "usual sendmail behaviour" did not occur, making the particular excel file
unable to open. I am just on the receiving end of the attachment, which should have only one <?xml version="1.0"?> line per attachment.
But in case there are multiple attachments in the sendmail file, your solution would get rid of the multiple duplicates :-)

On Wednesday, May 26, 2021 at 1:12:09 PM UTC-7, Janis Papanagnou wrote:
> On 26.05.2021 07:01, Harry wrote:
> > I have an email attachment ATT000001.xml with content like below.
> > Note that there are two consecutive lines <?xml version="1.0"?>.
> > How could I use sed or awk to delete one of them?
> One way could be
>
> awk -v p='<?xml version="1.0"?>' '$0!=p, $0==p'

Nice solution too.
I tried with my file having two set of 2 consecutive <?xml version="1.0"?> lines,
and this solution eliminates the two duplicated lines (one for each set).

[Edited]
I deleted what I said on "sendmail usual behaviour" ....
Actually the programmer wrote "smtp stripping first line so extra line."

On 2021-05-26, Janis Papanagnou <janis_papanagnou@hotmail.com> wrote:
> On 26.05.2021 07:01, Harry wrote:
....
> Just for a variant I want to provide another awk approach[**]
>
> awk -v p='<?xml version="1.0"?>' '$0!=p, $0==p'
>
> which has the property to keep only one of multiple (2+) and
> only consecutive lines matching the passed pattern.[***]
....
>
> [**] Probably a bit cryptic as well, though I simply derived it
> from another algorithm I suggested a couple years ago to squeeze
> multiple empty lines that separate text blocks to a single empty
> line: awk 'NF,!NF'

This reminds me of a lecture at a Seminar given by one of the profs
here.

"I gave a seminar 14 years ago on this topic, and I assume you all
remember that."

Yes, it is crytptic. Could you please expand?

On Wed, 26 May 2021 23:12:33 -0000 (UTC)
William Unruh <unruh@invalid.ca> wrote:
> On 2021-05-26, Janis Papanagnou <janis_papanagnou@hotmail.com> wrote:
> > On 26.05.2021 07:01, Harry wrote:
> ...
> > Just for a variant I want to provide another awk approach[**]
> >
> > awk -v p='<?xml version="1.0"?>' '$0!=p, $0==p'
> >
> > which has the property to keep only one of multiple (2+) and
> > only consecutive lines matching the passed pattern.[***]
> ...
> >
> > [**] Probably a bit cryptic as well, though I simply derived it
> > from another algorithm I suggested a couple years ago to squeeze
> > multiple empty lines that separate text blocks to a single empty
> > line: awk 'NF,!NF'
>
> This reminds me of a lecture at a Seminar given by one of the profs
> here.
>
> "I gave a seminar 14 years ago on this topic, and I assume you all
> remember that."
>
> Yes, it is crytptic. Could you please expand?

Are you familiar with range patterns in AWK ?

On 27.05.2021 01:12, William Unruh wrote:
> On 2021-05-26, Janis Papanagnou <janis_papanagnou@hotmail.com> wrote:
>> On 26.05.2021 07:01, Harry wrote:
> ...
>> Just for a variant I want to provide another awk approach[**]
>>
>> awk -v p='<?xml version="1.0"?>' '$0!=p, $0==p'
>>
>> which has the property to keep only one of multiple (2+) and
>> only consecutive lines matching the passed pattern.[***]
> ...
>>
>> [**] Probably a bit cryptic as well, though I simply derived it
>> from another algorithm I suggested a couple years ago to squeeze
>> multiple empty lines that separate text blocks to a single empty
>> line: awk 'NF,!NF'
>
> This reminds me of a lecture at a Seminar given by one of the profs
> here.
>
> "I gave a seminar 14 years ago on this topic, and I assume you all
> remember that."

Sorry, it was meant as: "If you understand the simpler form it's easy
to derive the a little bit more complex form." (i.e. an explanation
how I came to that code pattern).

> Yes, it is crytptic. Could you please expand?

Sure. A condition x,y is a range pattern, it executes the action block
(here non is given, so the default print $0 becomes executed),
starting from a record where x is true until (and including) a record
where y is true.

NF,!NF thus means (since NF means a blank line), start from any line
with visible input until (and including) a line that is blank. All
subsequent blank lines will not the printed since the next matching
pattern again start at a non-blank line.

The transposition to the new code is now easier to understand, I think.
$0!=p it starts printing with lines that don't match the pattern, and
continues until the first occurrence of the pattern ($0==p). The next
block starts again on the line that does not match the patter, thus it
skips additional patterns directly after the first one found.

Janis

On 27.05.2021 02:38, Janis Papanagnou wrote:
> On 27.05.2021 01:12, William Unruh wrote:
>> On 2021-05-26, Janis Papanagnou <janis_papanagnou@hotmail.com> wrote:
>>> On 26.05.2021 07:01, Harry wrote:
>> ...
>>> Just for a variant I want to provide another awk approach[**]
>>>
>>> awk -v p='<?xml version="1.0"?>' '$0!=p, $0==p'
>>>
>>> which has the property to keep only one of multiple (2+) and
>>> only consecutive lines matching the passed pattern.[***]
>> ...
>>>
>>> [**] Probably a bit cryptic as well, though I simply derived it
>>> from another algorithm I suggested a couple years ago to squeeze
>>> multiple empty lines that separate text blocks to a single empty
>>> line: awk 'NF,!NF'
>>
>> This reminds me of a lecture at a Seminar given by one of the profs
>> here.
>>
>> "I gave a seminar 14 years ago on this topic, and I assume you all
>> remember that."
>
> Sorry, it was meant as: "If you understand the simpler form it's easy
> to derive the a little bit more complex form." (i.e. an explanation
> how I came to that code pattern).
>
>> Yes, it is crytptic. Could you please expand?
>
> Sure. A condition x,y is a range pattern, it executes the action block
> (here non is given, so the default print $0 becomes executed),
> starting from a record where x is true until (and including) a record
> where y is true.
>
> NF,!NF thus means (since NF means a blank line), start from any line

Should have been: (since NF means a non-blank line)

> with visible input until (and including) a line that is blank. All
> subsequent blank lines will not the printed since the next matching
> pattern again start at a non-blank line.
>
> The transposition to the new code is now easier to understand, I think.
> $0!=p it starts printing with lines that don't match the pattern, and
> continues until the first occurrence of the pattern ($0==p). The next
> block starts again on the line that does not match the patter, thus it
> skips additional patterns directly after the first one found.
>
> Janis
>

In article <s8mpqn$7m0$2@news-1.m-online.net>,
Janis Papanagnou <janis_papanagnou@hotmail.com> wrote:
....
>> NF,!NF thus means (since NF means a blank line), start from any line
>
>Should have been: (since NF means a non-blank line)

Of course, a true nitpicker will point out that a line consisting only of
(a non-zero number of) spaces and/or tabs is not blank but will have NF==0.

If this concerns you, you could use "length" instead of "NF":

length,!length

Note: Gawk (the only AWK anyone who doesn't have access to TAWK should ever
use) supports this form of "length"; not sure which others do.

--
The randomly chosen signature file that would have appeared here is more than 4
lines long. As such, it violates one or more Usenet RFCs. In order to remain
in compliance with said RFCs, the actual sig can be found at the following URL:
http://user.xmission.com/~gazelle/Sigs/Cancer

On 2021-05-27, Janis Papanagnou <janis_papanagnou@hotmail.com> wrote:
> On 27.05.2021 01:12, William Unruh wrote:
>> Yes, it is crytptic. Could you please expand?
....
>
> Sure. A condition x,y is a range pattern, it executes the action block
> (here non is given, so the default print $0 becomes executed),
> starting from a record where x is true until (and including) a record
> where y is true.

OK, makes sense. I had not come across that .
Thanks for the explanation.

On 27.05.2021 02:56, Kenny McCormack wrote:
> In article <s8mpqn$7m0$2@news-1.m-online.net>,
> Janis Papanagnou <janis_papanagnou@hotmail.com> wrote:
> ...
>>> NF,!NF thus means (since NF means a blank line), start from any line
>>
>> Should have been: (since NF means a non-blank line)
>
> Of course, a true nitpicker will point out that a line consisting only of
> (a non-zero number of) spaces and/or tabs is not blank but will have NF==0.

I deliberately used the words blank/non-blank to distinguish it from
empty/non-empty. Any blank/tab-character makes the line non-empty but
it still appears to be "blank", thus visually separating text blocks.
In the given case NF,!NF any accidentally typed blank or tab should
be considered like an empty line. (There may be other cases where it
might be desired to assume a different semantics, but (to me) in case
of blank lines this seemed more appropriate.)

>
> If this concerns you, you could use "length" instead of "NF":
>
> length,!length

This would not fit with the semantics I had in mind. Application cases,
interpretations, and opinions may differ, of course.

Janis

> Note: Gawk (the only AWK anyone who doesn't have access to TAWK should ever
> use) supports this form of "length"; not sure which others do.
>

On 26.05.2021 at 23:12, William Unruh scribbled:

> This reminds me of a lecture at a Seminar given by one of the profs
> here.
>
> "I gave a seminar 14 years ago on this topic, and I assume you all
> remember that."

I had one that was even worse, in my freshman year of Applied
Information Technology. He was a military officer with the rank of
Major (and later Colonel) who was teaching math at civilian colleges.

"I'm not going to explain what's on the next 30 pages of your
course because you've all seen that stuff in high school last year."

Later on I discreetly told him that my graduation from high school was
11 years ago, and that I had in the meantime been to college in a
completely unrelated direction, then put in my conscription duty in the
military and then had several also unrelated jobs before deciding
to go back to college in order to study IT.

He didn't know what to say. Oopsie.

--
With respect,
= Aragorn =