Message-ID:

"The chain which can be yanked is not the eternal chain." -- G. Fitch

Re: tail

<mailman.227.1650752234.20749.python-list@python.org>

https://www.novabbs.com/devel/article-flat.php?id=18007&group=comp.lang.python#18007

copy link Newsgroups: comp.lang.python

Path: i2pn2.org!i2pn.org!news.uzoreto.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: cs...@cskk.id.au (Cameron Simpson)
Newsgroups: comp.lang.python
Subject: Re: tail
Date: Sun, 24 Apr 2022 08:09:44 +1000
Lines: 27
Message-ID: <mailman.227.1650752234.20749.python-list@python.org>
References: <CAPTjJmrJacamKq1V5T8FECkm4jURdYQgj0VsC+JK5Db0NoFaww@mail.gmail.com>
<YmR5KHiegnFT3Zt+@cskk.homeip.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Trace: news.uni-berlin.de SITKeD739bT9H1JG7X49LAwixgmUpuW8tm1F5OHi0ldw==
Return-Path: <cameron@cskk.id.au>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=none reason="no signature";
dkim-adsp=none (unprotected policy); dkim-atps=neutral
X-Spam-Status: OK 0.002
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; '2022': 0.05; 'sun,': 0.07;
'angelico': 0.09; 'byte': 0.09; 'hard.': 0.09; 'text.': 0.09;
'yes.': 0.09; 'cheers,': 0.11; 'memory': 0.15; 'approach.': 0.16;
'cameron': 0.16; 'from:addr:cs': 0.16; 'from:addr:cskk.id.au':
0.16; 'from:name:cameron simpson': 0.16; 'locate': 0.16; 'message-
id:@cskk.homeip.net': 0.16; 'mostly,': 0.16; 'received:10.10':
0.16; 'reuse': 0.16; 'simpson': 0.16; 'wrote:': 0.16; 'instead':
0.17; 'to:addr:python-list': 0.20; 'function': 0.27; 'chris':
0.28; 'it,': 0.29; 'header:User-Agent:1': 0.30; 'whole': 0.30;
'approach': 0.31; 'think': 0.32; 'but': 0.32; 'header:In-Reply-
To:1': 0.34; 'using': 0.37; "it's": 0.37; 'file': 0.38; 'single':
0.39; 'use': 0.39; 'seeking': 0.40; 'both': 0.40; 'follow': 0.62;
'simply': 0.63; 'your': 0.64; 'received:userid': 0.66; 'back':
0.67; 'header:Received:6': 0.67; 'time,': 0.67; 'per': 0.68;
'sulla': 0.84
X-RG-Spam: Unknown
X-RazorGate-Vade: gggruggvucftvghtrhhoucdtuddrgedvfedrtdejgddtjecutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfupfevtfgpvffgnffuvffttedpqfgfvfenuceurghilhhouhhtmecugedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujfgurhepfffhvffukfggtggujggffhesthdtredttdervdenucfhrhhomhepvegrmhgvrhhonhcuufhimhhpshhonhcuoegtshestghskhhkrdhiugdrrghuqeenucggtffrrghtthgvrhhnpeekkeejueduhfefveeikeeftdekueeggeduhfevtdfflefhheethfeujeehuefgvdenucffohhmrghinhepphihthhhohhnrdhorhhgnecukfhppedurddugeegrddvfedtrdduieejnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehhvghlohepsghorhhgrdgtshhkkhdrhhhomhgvihhprdhnvghtpdhinhgvthepuddrudeggedrvdeftddrudeijedpmhgrihhlfhhrohhmpegtrghmvghrohhnsegtshhkkhdrihgurdgruhdpnhgspghrtghpthhtohepuddprhgtphhtthhopehphihthhhonhdqlhhishhtsehphihthhhonhdrohhrgh
X-RazorGate-Vade-Verdict: clean 0
X-RazorGate-Vade-Classification: clean
X-RG-VS-CLASS: clean
X-Authentication-Info: Submitted using ID cskk@bigpond.com
Mail-Followup-To: Python List <python-list@python.org>
Content-Disposition: inline
In-Reply-To: <CAPTjJmrJacamKq1V5T8FECkm4jURdYQgj0VsC+JK5Db0NoFaww@mail.gmail.com>
User-Agent: Mutt/2.2.1 (2022-02-19)
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <YmR5KHiegnFT3Zt+@cskk.homeip.net>
X-Mailman-Original-References: <CAPTjJmrJacamKq1V5T8FECkm4jURdYQgj0VsC+JK5Db0NoFaww@mail.gmail.com>

by: Cameron Simpson - Sat, 23 Apr 2022 22:09 UTC

On 24Apr2022 07:15, Chris Angelico <rosuav@gmail.com> wrote:
>On Sun, 24 Apr 2022 at 07:13, Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
>> Emh, why chunks? My function simply reads byte per byte and compares
>> it to b"\n". When it find it, it stops and do a readline():
[...]
>> This is only for one line and in utf8, but it can be generalised.

For some encodings that generalisation might be hard. But mostly, yes.

>Ah. Well, then, THAT is why it's inefficient: you're seeking back one
>single byte at a time, then reading forwards. That is NOT going to
>play nicely with file systems or buffers.

An approach I think you both may have missed: mmap the file and use
mmap.rfind(b'\n') to locate line delimiters.
https://docs.python.org/3/library/mmap.html#mmap.mmap.rfind

Avoids sucking the whole file into memory in the usualy sense, instead
the file is paged in as needed. Far more efficient that a seek/read
single byte approach.

If the file's growing you can do this to start with, then do a normal
file open from your end point to follow accruing text. (Or reuse the
descriptor you sues for the mmap, but using s.read().)

Cheers,
Cameron Simpson <cs@cskk.id.au>

"The chain which can be yanked is not the eternal chain." -- G. Fitch

devel / comp.lang.python / Re: tail