Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  newsreader  groups  login

Message-ID:  

Pie are not square. Pie are round. Cornbread are square.


devel / comp.unix.shell / sed buffering

SubjectAuthor
* sed bufferingJavier
+* Re: sed bufferingJanis Papanagnou
|`* Re: sed bufferingJavier
| `* Re: sed bufferingJanis Papanagnou
|  `* Re: sed bufferingRandal L. Schwartz
|   `- Re: sed bufferingJanis Papanagnou
+- Re: sed bufferingChristian Weisgerber
`- Re: sed bufferingKaz Kylheku

1
sed buffering

<OuOdnb0s3MQt_yv9nZ2dnUU78QnNnZ2d@brightview.co.uk>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=3891&group=comp.unix.shell#3891

 copy link   Newsgroups: comp.unix.shell
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!border2.nntp.ams1.giganews.com!nntp.giganews.com!buffer2.nntp.ams1.giganews.com!nntp.brightview.co.uk!news.brightview.co.uk.POSTED!not-for-mail
NNTP-Posting-Date: Tue, 01 Jun 2021 11:39:12 -0500
From: inva...@invalid.invalid (Javier)
Subject: sed buffering
Newsgroups: comp.unix.shell
Message-ID: <OuOdnb0s3MQt_yv9nZ2dnUU78QnNnZ2d@brightview.co.uk>
Date: Tue, 01 Jun 2021 11:39:12 -0500
Lines: 21
X-Usenet-Provider: http://www.giganews.com
X-Trace: sv3-2NIQZmKcCsanwfR5yyKZYEAa+Le7R7oJEX0WHP1cAkdrMubeJx1b/wBpgQeI08BxYIVcrK9uyZ78Bs5!wvnDpSCZZyyp6m99hz2XPgpxqqdtrMChriCJ9YHSviinNnw4k6jRDptMvodpWHFQ9XNWbmR6bSUy!0b7qicVX4fZojGRtFs1KlNNXQw==
X-Abuse-and-DMCA-Info: Please be sure to forward a copy of ALL headers
X-Abuse-and-DMCA-Info: Otherwise we will be unable to process your complaint properly
X-Postfilter: 1.3.40
X-Original-Bytes: 1555
 by: Javier - Tue, 1 Jun 2021 16:39 UTC

I'm trying to see dns queries in real time, and doing some filtering
of tcpdump output. tcpdump has the option -U for bufffering by packets.
sed has also the option -u for minimal buffering, but that doesnt' seem to help.
It looks like the problem is gnu/sed (the system is linux).

dns_server=192.168.1.1
sudo tcpdump -U -i eth0 -l -vvv dst host ${dns_server} and dst port 53 \
| sed -u 's/^.*\+ A+\? //g' | sed -u 's/. (.*$//g'

The gnu/sed man page says:

-u, --unbuffered

load minimal amounts of data from the input files and flush the
output buffers more often

The man page is not very clear what it is minimal and often output buffers are flushed.

Does anybody have any suggestion?

Re: sed buffering

<s962j1$k3n$1@news-1.m-online.net>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=3892&group=comp.unix.shell#3892

 copy link   Newsgroups: comp.unix.shell
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!news.mixmin.net!news2.arglkargh.de!news.karotte.org!news.space.net!news.m-online.net!.POSTED!not-for-mail
From: janis_pa...@hotmail.com (Janis Papanagnou)
Newsgroups: comp.unix.shell
Subject: Re: sed buffering
Date: Tue, 1 Jun 2021 21:42:25 +0200
Organization: (posted via) M-net Telekommunikations GmbH
Lines: 40
Message-ID: <s962j1$k3n$1@news-1.m-online.net>
References: <OuOdnb0s3MQt_yv9nZ2dnUU78QnNnZ2d@brightview.co.uk>
NNTP-Posting-Host: 2001:a61:252a:da01:c8f2:210b:abd0:4ebc
Mime-Version: 1.0
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Trace: news-1.m-online.net 1622576545 20599 2001:a61:252a:da01:c8f2:210b:abd0:4ebc (1 Jun 2021 19:42:25 GMT)
X-Complaints-To: news@news-1.m-online.net
NNTP-Posting-Date: Tue, 1 Jun 2021 19:42:25 +0000 (UTC)
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
Thunderbird/45.8.0
X-Enigmail-Draft-Status: N1110
In-Reply-To: <OuOdnb0s3MQt_yv9nZ2dnUU78QnNnZ2d@brightview.co.uk>
 by: Janis Papanagnou - Tue, 1 Jun 2021 19:42 UTC

On 01.06.2021 18:39, Javier wrote:
> I'm trying to see dns queries in real time, and doing some filtering
> of tcpdump output. tcpdump has the option -U for bufffering by packets.
> sed has also the option -u for minimal buffering, but that doesnt' seem to help.

What evidence do you have for the impression you've got?

> It looks like the problem is gnu/sed (the system is linux).
>
> dns_server=192.168.1.1
> sudo tcpdump -U -i eth0 -l -vvv dst host ${dns_server} and dst port 53 \
> | sed -u 's/^.*\+ A+\? //g' | sed -u 's/. (.*$//g'
>

Seems to work for me.

>
> The gnu/sed man page says:
>
> -u, --unbuffered
>
> load minimal amounts of data from the input files and flush the
> output buffers more often
>
> The man page is not very clear what it is minimal and often output buffers are flushed.

Both, "unbuffered" and "minimal amounts", indicate that output is
unbuffered. (The "more often" is probably just an informal hint.)

> Does anybody have any suggestion?

In case your commands would not do the unbuffering I'd resort to
a (any) pty command to guarantee it (but I doubt that unbuffering
isn't implemented in sed if it's documented).

BTW, line-buffering would suffice in case of line-oriented text
from tcpdump processed by sed.

Janis

Re: sed buffering

<slrnsbd4ep.1jvj.naddy@lorvorc.mips.inka.de>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=3893&group=comp.unix.shell#3893

 copy link   Newsgroups: comp.unix.shell
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!news.szaf.org!inka.de!mips.inka.de!.POSTED.localhost!not-for-mail
From: nad...@mips.inka.de (Christian Weisgerber)
Newsgroups: comp.unix.shell
Subject: Re: sed buffering
Date: Tue, 1 Jun 2021 20:00:25 -0000 (UTC)
Message-ID: <slrnsbd4ep.1jvj.naddy@lorvorc.mips.inka.de>
References: <OuOdnb0s3MQt_yv9nZ2dnUU78QnNnZ2d@brightview.co.uk>
Injection-Date: Tue, 1 Jun 2021 20:00:25 -0000 (UTC)
Injection-Info: lorvorc.mips.inka.de; posting-host="localhost:::1";
logging-data="53236"; mail-complaints-to="usenet@mips.inka.de"
User-Agent: slrn/1.0.3 (FreeBSD)
 by: Christian Weisgerber - Tue, 1 Jun 2021 20:00 UTC

On 2021-06-01, Javier <invalid@invalid.invalid> wrote:

> I'm trying to see dns queries in real time, and doing some filtering
> of tcpdump output. tcpdump has the option -U for bufffering by packets.
> sed has also the option -u for minimal buffering, but that doesnt' seem to help.
> It looks like the problem is gnu/sed (the system is linux).
>
> dns_server=192.168.1.1
> sudo tcpdump -U -i eth0 -l -vvv dst host ${dns_server} and dst port 53 \
> | sed -u 's/^.*\+ A+\? //g' | sed -u 's/. (.*$//g'

Before doing real-time analysis, I would capture some packets with
-w, then replay them with -r to check that my regular expressions
work as expected.

That first sed pattern looks like you are trying to use extended
regular expressions. However, sed defaults to basic regular
expressions.

--
Christian "naddy" Weisgerber naddy@mips.inka.de

Re: sed buffering

<20210601152342.370@kylheku.com>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=3894&group=comp.unix.shell#3894

 copy link   Newsgroups: comp.unix.shell
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: 563-365-...@kylheku.com (Kaz Kylheku)
Newsgroups: comp.unix.shell
Subject: Re: sed buffering
Date: Tue, 1 Jun 2021 22:24:34 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 4
Message-ID: <20210601152342.370@kylheku.com>
References: <OuOdnb0s3MQt_yv9nZ2dnUU78QnNnZ2d@brightview.co.uk>
Injection-Date: Tue, 1 Jun 2021 22:24:34 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="2045a1992f1e67117c001bb861b90c73";
logging-data="468"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19j9OzutUlayvIizvIsBkPDwb8VdEfXHsI="
User-Agent: slrn/1.0.3 (Linux)
Cancel-Lock: sha1:JN7dfl1dW0zHCeDdxE0rCq6PBhU=
 by: Kaz Kylheku - Tue, 1 Jun 2021 22:24 UTC

On 2021-06-01, Javier <invalid@invalid.invalid> wrote:
> I'm trying to see dns queries in real time, and doing some filtering

"I'm bed-ridden and suffering; oops, I mean sed-ridden and buffering."

Re: sed buffering

<VoOdnantmKARfSv9nZ2dnUU78TmdnZ2d@brightview.co.uk>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=3895&group=comp.unix.shell#3895

 copy link   Newsgroups: comp.unix.shell
Path: i2pn2.org!i2pn.org!usenet.goja.nl.eu.org!2.eu.feeder.erje.net!feeder.erje.net!border1.nntp.ams1.giganews.com!nntp.giganews.com!buffer1.nntp.ams1.giganews.com!nntp.brightview.co.uk!news.brightview.co.uk.POSTED!not-for-mail
NNTP-Posting-Date: Tue, 01 Jun 2021 20:36:12 -0500
From: inva...@invalid.invalid (Javier)
Subject: Re: sed buffering
Newsgroups: comp.unix.shell
References: <OuOdnb0s3MQt_yv9nZ2dnUU78QnNnZ2d@brightview.co.uk> <s962j1$k3n$1@news-1.m-online.net>
Message-ID: <VoOdnantmKARfSv9nZ2dnUU78TmdnZ2d@brightview.co.uk>
Date: Tue, 01 Jun 2021 20:36:12 -0500
Lines: 10
X-Usenet-Provider: http://www.giganews.com
X-Trace: sv3-530rrpVeg1TjedbNGw0UhSU67KIY98pzHpjZb6MbHgc5ldKwyzMJRnordmF9SMzFx+5ZAXQ7l5Egjny!nG3Q44JzUyYA0GcfEwp8rl6FrelmK2Yy+tMJ3zWmRic3dMXJLJZTKGamVaKoCvBgSQZtxINhGKjm!DGUzzAQdheYKQVQreNP9je7DAg==
X-Abuse-and-DMCA-Info: Please be sure to forward a copy of ALL headers
X-Abuse-and-DMCA-Info: Otherwise we will be unable to process your complaint properly
X-Postfilter: 1.3.40
X-Original-Bytes: 1172
 by: Javier - Wed, 2 Jun 2021 01:36 UTC

I have fixed it now. I think the problem was the regexp, not the -u option of sed.
This code works:

dns_server=...
net_iface=...
sudo tcpdump -U -i ${net_iface} -l -vvv dst host ${dns_server} and dst port 53 \
| sed -u -e 's/^.*[?] //g' | sed -u -e 's/. (.*$//g'

Re: sed buffering

<s97gcb$1cr$1@news-1.m-online.net>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=3896&group=comp.unix.shell#3896

 copy link   Newsgroups: comp.unix.shell
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!news.mixmin.net!news2.arglkargh.de!news.karotte.org!news.space.net!news.m-online.net!.POSTED!not-for-mail
From: janis_pa...@hotmail.com (Janis Papanagnou)
Newsgroups: comp.unix.shell
Subject: Re: sed buffering
Date: Wed, 2 Jun 2021 10:43:55 +0200
Organization: (posted via) M-net Telekommunikations GmbH
Lines: 21
Message-ID: <s97gcb$1cr$1@news-1.m-online.net>
References: <OuOdnb0s3MQt_yv9nZ2dnUU78QnNnZ2d@brightview.co.uk>
<s962j1$k3n$1@news-1.m-online.net>
<VoOdnantmKARfSv9nZ2dnUU78TmdnZ2d@brightview.co.uk>
NNTP-Posting-Host: 2001:a61:252a:da01:349d:30f1:c589:5773
Mime-Version: 1.0
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Trace: news-1.m-online.net 1622623435 1435 2001:a61:252a:da01:349d:30f1:c589:5773 (2 Jun 2021 08:43:55 GMT)
X-Complaints-To: news@news-1.m-online.net
NNTP-Posting-Date: Wed, 2 Jun 2021 08:43:55 +0000 (UTC)
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
Thunderbird/45.8.0
In-Reply-To: <VoOdnantmKARfSv9nZ2dnUU78TmdnZ2d@brightview.co.uk>
 by: Janis Papanagnou - Wed, 2 Jun 2021 08:43 UTC

On 02.06.2021 03:36, Javier wrote:
> I have fixed it now. I think the problem was the regexp, not the -u option of sed.

That's fine.

Meanwhile I also had a look into the 'sed' sources and it seems to
operate 'unbuffered' (not line-buffered) if the respective command
line option is set.

Janis

> This code works:
>
> dns_server=...
> net_iface=...
> sudo tcpdump -U -i ${net_iface} -l -vvv dst host ${dns_server} and dst port 53 \
> | sed -u -e 's/^.*[?] //g' | sed -u -e 's/. (.*$//g'
>
>
>

Re: sed buffering

<86eedk1g49.fsf@red.stonehenge.com>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=3897&group=comp.unix.shell#3897

 copy link   Newsgroups: comp.unix.shell
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!newsreader4.netcologne.de!news.netcologne.de!peer03.ams1!peer.ams1.xlned.com!news.xlned.com!peer01.iad!feed-me.highwinds-media.com!news.highwinds-media.com!fx47.iad.POSTED!not-for-mail
From: mer...@stonehenge.com (Randal L. Schwartz)
Newsgroups: comp.unix.shell
Subject: Re: sed buffering
References: <OuOdnb0s3MQt_yv9nZ2dnUU78QnNnZ2d@brightview.co.uk>
<s962j1$k3n$1@news-1.m-online.net>
<VoOdnantmKARfSv9nZ2dnUU78TmdnZ2d@brightview.co.uk>
<s97gcb$1cr$1@news-1.m-online.net>
x-mayan-date: Long count = 13.0.8.10.5; tzolkin = 8 Chicchan; haab = 3 Zotz
Message-ID: <86eedk1g49.fsf@red.stonehenge.com>
Organization: Stonehenge Consulting Services; Portland, Oregon, USA
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (berkeley-unix)
Cancel-Lock: sha1:y1O12qCzfuUannse5Q+BNiMgcSo=
MIME-Version: 1.0
Content-Type: text/plain
Lines: 14
X-Complaints-To: abuse@blocknews.net
NNTP-Posting-Date: Wed, 02 Jun 2021 15:03:06 UTC
Date: Wed, 02 Jun 2021 07:43:50 -0700
X-Received-Bytes: 1572
 by: Randal L. Schwartz - Wed, 2 Jun 2021 14:43 UTC

>>>>> "Janis" == Janis Papanagnou <janis_papanagnou@hotmail.com> writes:

Janis> Meanwhile I also had a look into the 'sed' sources and it seems to
Janis> operate 'unbuffered' (not line-buffered) if the respective command
Janis> line option is set.

But *which* sed, and it's sad that we have to say that now, thanks to
the GNU project. :(

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Dart/Flutter consulting, Technical writing, Comedy, etc. etc.
Still trying to think of something clever for the fourth line of this .sig

Re: sed buffering

<s98891$850$1@news-1.m-online.net>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=3898&group=comp.unix.shell#3898

 copy link   Newsgroups: comp.unix.shell
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!4.us.feeder.erje.net!feeder.erje.net!news2.arglkargh.de!news.karotte.org!news.space.net!news.m-online.net!.POSTED!not-for-mail
From: janis_pa...@hotmail.com (Janis Papanagnou)
Newsgroups: comp.unix.shell
Subject: Re: sed buffering
Date: Wed, 2 Jun 2021 17:31:45 +0200
Organization: (posted via) M-net Telekommunikations GmbH
Lines: 17
Message-ID: <s98891$850$1@news-1.m-online.net>
References: <OuOdnb0s3MQt_yv9nZ2dnUU78QnNnZ2d@brightview.co.uk>
<s962j1$k3n$1@news-1.m-online.net>
<VoOdnantmKARfSv9nZ2dnUU78TmdnZ2d@brightview.co.uk>
<s97gcb$1cr$1@news-1.m-online.net> <86eedk1g49.fsf@red.stonehenge.com>
NNTP-Posting-Host: 2001:a61:252a:da01:349d:30f1:c589:5773
Mime-Version: 1.0
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Trace: news-1.m-online.net 1622647906 8352 2001:a61:252a:da01:349d:30f1:c589:5773 (2 Jun 2021 15:31:46 GMT)
X-Complaints-To: news@news-1.m-online.net
NNTP-Posting-Date: Wed, 2 Jun 2021 15:31:46 +0000 (UTC)
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
Thunderbird/45.8.0
X-Enigmail-Draft-Status: N1110
In-Reply-To: <86eedk1g49.fsf@red.stonehenge.com>
 by: Janis Papanagnou - Wed, 2 Jun 2021 15:31 UTC

On 02.06.2021 16:43, Randal L. Schwartz wrote:
>>>>>> "Janis" == Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>
> Janis> Meanwhile I also had a look into the 'sed' sources and it seems to
> Janis> operate 'unbuffered' (not line-buffered) if the respective command
> Janis> line option is set.
>
> But *which* sed, and it's sad that we have to say that now, thanks to
> the GNU project. :(

The OP said GNU/sed, so I looked into the current git version I found.
Can you elaborate on your comment about issues with the "GNU project"?
Is it because of the changelog being from Sept 2014? Yet '-unbuffered'
is already implemented, so I assume that part is still valid.

Janis

1
server_pubkey.txt

rocksolid light 0.9.7
clearnet tor