Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  newsreader  groups  login

Message-ID:  

Egotist: A person of low taste, more interested in himself than in me. -- Ambrose Bierce


devel / comp.lang.python / Re: tail

SubjectAuthor
o Re: tailChris Angelico

1
Re: tail

<mailman.229.1650752507.20749.python-list@python.org>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=18009&group=comp.lang.python#18009

 copy link   Newsgroups: comp.lang.python
Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: ros...@gmail.com (Chris Angelico)
Newsgroups: comp.lang.python
Subject: Re: tail
Date: Sun, 24 Apr 2022 08:21:34 +1000
Lines: 29
Message-ID: <mailman.229.1650752507.20749.python-list@python.org>
References: <CAPTjJmrJacamKq1V5T8FECkm4jURdYQgj0VsC+JK5Db0NoFaww@mail.gmail.com>
<YmR5KHiegnFT3Zt+@cskk.homeip.net>
<CAPTjJmo7oNKuuiQPa_t3X9Sws3veDuX4660We5ZKFPeajC9+Ng@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
X-Trace: news.uni-berlin.de KE2iGtqeCbvAD6AfxmIOPgGMgr84qgyzZRKFM6JRDmUg==
Return-Path: <rosuav@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=gmail.com header.i=@gmail.com header.b=mpCYXOSF;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.004
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; '2022': 0.05; 'sun,': 0.07;
'angelico': 0.09; 'byte': 0.09; 'hard.': 0.09; 'yes.': 0.09;
'cameron': 0.16; 'chrisa': 0.16; 'chunking,': 0.16; 'elaborate':
0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16;
'locate': 0.16; 'mostly,': 0.16; 'predicting': 0.16; 'simpson':
0.16; 'vague': 0.16; 'wrote:': 0.16; 'to:addr:python-list': 0.20;
"i've": 0.22; 'idea': 0.24; 'function': 0.27; 'chris': 0.28;
'it,': 0.29; 'approach': 0.31; 'think': 0.32; 'zero': 0.32;
'message-id:@mail.gmail.com': 0.32; 'but': 0.32; "i'm": 0.33;
"didn't": 0.34; 'header:In-Reply-To:1': 0.34;
'received:google.com': 0.34; 'from:addr:gmail.com': 0.35; "it's":
0.37; 'received:209.85': 0.37; 'file': 0.38; 'received:209': 0.39;
'use': 0.39; 'seeking': 0.40; 'both': 0.40; 'something': 0.40;
'simply': 0.63; 'back': 0.67; 'time,': 0.67; 'per': 0.68;
'performance': 0.71; 'job,': 0.84; 'sulla': 0.84
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to;
bh=MnV2r/p3d1o/gT5DgknuZl+Bymid36RUdzkpJenR0Ig=;
b=mpCYXOSFMzzUqv77kOXDA2Z3Ih5nW1iZsIJdRUfCNIhkNgYeIqsvkaMzQRxIS9u+xK
CXiB2KtVfwfWJMrQYBIPfX5d9xCeVqLXEJev2JLoLcBzFgA0neNIGyOx7CyCZEVqzs9p
xqGUn4X3nkY8HtVA7MzZpzmAHxKaA60wgG1Djfyz8/gZp9B/ckTu5uimmanEZKpGMByP
4sTqQkeZuXb6X38ypTTQ2QSWskANRuKZo3OzfyD/+Qe5n0C0mI43/7VZorhutWiN7lGj
uNoSJPZTTI+j8xM9NwATqAGfWOPrRrptO6H+xZZhBurO9oUfA7SSkV3SLCQbg2/jMoqm
pXGg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to;
bh=MnV2r/p3d1o/gT5DgknuZl+Bymid36RUdzkpJenR0Ig=;
b=OEiULLHKy/iKqQ7uCUIXRAg2FlKgtcCR19BL0VkP7c86+L4o3J5ZVOJD2mYSNZFl4+
1vQy/4He/ysQfihDNyAB0Qb8QybYL8tsZCTvq5uG6UTpCqdEA5NKeRGN2W7Skq3ouugA
0xcz4cCFgnhW19q1noLwp7caSLQRPImM8IwGbn1qVQ0PENjbAOJHxJUF+KjlXhQ6fHl1
ERDj2gqUSJMLsjosSnUyJDr+rhuDpNjAllXIBLupP2U4ruu6jl6c2m9M9ezLWH28RV+K
F5GUgntYVNoO3GsMls/bpqb5+rXhslqpIe5b6kBYPu93DNe/LdxLbLg/DVw80rqiScNR
z/yA==
X-Gm-Message-State: AOAM531EXy2JnaqjyQG5sP/Due3s64UAZHSsIW3WzaygUAh44lvBFA/k
asXjroDfhcS4vIcqknDAGdpu3qtoJBaVZD2LlYBLhIzk
X-Google-Smtp-Source: ABdhPJyJGB08oRifkaKyrUkCuAxdS0dx2NToBqbhzWt74qHZlnotrZdlcgBFmu0mcpOLThRTlJ2cG0hCYwixjEVo//0=
X-Received: by 2002:a05:6000:507:b0:20a:a549:d3ab with SMTP id
a7-20020a056000050700b0020aa549d3abmr8595408wrf.243.1650752505264; Sat, 23
Apr 2022 15:21:45 -0700 (PDT)
In-Reply-To: <YmR5KHiegnFT3Zt+@cskk.homeip.net>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <CAPTjJmo7oNKuuiQPa_t3X9Sws3veDuX4660We5ZKFPeajC9+Ng@mail.gmail.com>
X-Mailman-Original-References: <CAPTjJmrJacamKq1V5T8FECkm4jURdYQgj0VsC+JK5Db0NoFaww@mail.gmail.com>
<YmR5KHiegnFT3Zt+@cskk.homeip.net>
 by: Chris Angelico - Sat, 23 Apr 2022 22:21 UTC

On Sun, 24 Apr 2022 at 08:18, Cameron Simpson <cs@cskk.id.au> wrote:
>
> On 24Apr2022 07:15, Chris Angelico <rosuav@gmail.com> wrote:
> >On Sun, 24 Apr 2022 at 07:13, Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
> >> Emh, why chunks? My function simply reads byte per byte and compares
> >> it to b"\n". When it find it, it stops and do a readline():
> [...]
> >> This is only for one line and in utf8, but it can be generalised.
>
> For some encodings that generalisation might be hard. But mostly, yes.
>
> >Ah. Well, then, THAT is why it's inefficient: you're seeking back one
> >single byte at a time, then reading forwards. That is NOT going to
> >play nicely with file systems or buffers.
>
> An approach I think you both may have missed: mmap the file and use
> mmap.rfind(b'\n') to locate line delimiters.
> https://docs.python.org/3/library/mmap.html#mmap.mmap.rfind

Yeah, I made a vague allusion to use of mmap, but didn't elaborate
because I actually have zero idea of how efficient this would be.
Would it be functionally equivalent to the chunking, but with the
chunk size defined by the system as whatever's most optimal? It would
need to be tested.

I've never used mmap for this kind of job, so it's not something I'm
comfortable predicting the performance of.

ChrisA

1
server_pubkey.txt

rocksolid light 0.9.7
clearnet tor