Message-ID:

There are bugs and then there are bugs. And then there are bugs. -- Karl Lehenbauer

devel / comp.lang.python / Re: Question(s)

Re: Question(s)

<mailman.48.1698198084.3828.python-list@python.org>

https://www.novabbs.com/devel/article-flat.php?id=24192&group=comp.lang.python#24192

copy link Newsgroups: comp.lang.python

Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: ros...@gmail.com (Chris Angelico)
Newsgroups: comp.lang.python
Subject: Re: Question(s)
Date: Wed, 25 Oct 2023 12:41:10 +1100
Lines: 38
Message-ID: <mailman.48.1698198084.3828.python-list@python.org>
References: <CAPpdf58WXyJs+Rd_GnbS-6dW8bPNt5DsojnVN5_ep+ebNx36Pw@mail.gmail.com>
<58b56dbe-646c-4a94-8102-ac2cf6efe233@tompassin.net>
<CAPpdf5_WVe07YVm-dZVV1mpZSWB6v4Z0itRsfYxYVdt8bWt0Ng@mail.gmail.com>
<062901da06e1$46a074e0$d3e15ea0$@gmail.com>
<CAPTjJmqhnQcK0Jr_foTXdHQNYz_-ROp13S6Uz0nP04Muk4DQnw@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Trace: news.uni-berlin.de 6l4bQRKcneLbDFvn1mnwCQ1sMtnAd2hpHcUy9RnW4POA==
Cancel-Lock: sha1:Q9M4OwX3plS+7Xa0S1eh2kgd45s= sha256:0TgiOZHcwVwLpMsgATlqU/rwQjPRURs5G5dj5+QwJYY=
Return-Path: <rosuav@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=gmail.com header.i=@gmail.com header.b=SWDM9jPt;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.038
X-Spam-Evidence: '*H*': 0.92; '*S*': 0.00; 'absolute': 0.05;
'chances': 0.05; 'random': 0.05; '2023': 0.07; 'codes': 0.07;
'cpu': 0.07; 'blocking': 0.09; 'byte': 0.09; 'intel': 0.09;
'rarely': 0.09; 'log': 0.12; 'memory': 0.15; 'avi': 0.16; 'bits':
0.16; 'chrisa': 0.16; 'correction': 0.16; 'easily.': 0.16;
'efficiently': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris
angelico': 0.16; 'gross': 0.16; 'happening.': 0.16; 'naive': 0.16;
'places,': 0.16; 'redundant': 0.16; 'strategy,': 0.16; 'wrote:':
0.16; 'probably': 0.17; 'subject:Question': 0.19; 'to:addr:python-
list': 0.20; 'code': 0.23; 'bit': 0.27; 'example,': 0.28; 'wrong':
0.28; 'error': 0.29; 'code,': 0.31; 'program': 0.31; 'assume':
0.32; 'python-list': 0.32; 'message-id:@mail.gmail.com': 0.32;
'but': 0.32; 'there': 0.33; 'able': 0.34; 'mean': 0.34; 'header
:In-Reply-To:1': 0.34; 'received:google.com': 0.34; '"if': 0.35;
'runs': 0.35; 'from:addr:gmail.com': 0.35; 'fix': 0.36; 'really':
0.37; 'using': 0.37; 'hard': 0.37; 'read': 0.38; 'two': 0.39;
'copies': 0.39; 'wed,': 0.39; 'something': 0.40; 'in,': 0.60;
'url-ip:216/8': 0.60; 'likely': 0.61; 'url-ip:142.250.179/24':
0.61; 'data,': 0.63; 'url-ip:142.250.179.174/32': 0.63; 'great':
0.63; 'everything': 0.63; "you'd": 0.64; 'subject:(': 0.64;
'url:watch': 0.64; 'your': 0.64; 'years': 0.65; 'rate': 0.67;
'back': 0.67; 'maximum': 0.67; 'order': 0.69; 'cost': 0.69;
'times': 0.69; 'ignore': 0.71; 'note:': 0.71; 'causing': 0.75;
'five': 0.75; 'detection': 0.76; 'guarantee': 0.76; 'see,': 0.76;
'url:v': 0.78; 'happens': 0.84; 'elsewhere.': 0.84; 'implies':
0.84; 'received:mail-lf1-x133.google.com': 0.84; 'say,': 0.84;
'cheap': 0.91; 'storage': 0.95
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20230601; t=1698198082; x=1698802882; darn=python.org;
h=content-transfer-encoding:to:subject:message-id:date:from
:in-reply-to:references:mime-version:from:to:cc:subject:date
:message-id:reply-to;
bh=7W3zkJV41UUD3uF8oOAw19zL9BeOA3YBgCFWpvZiDfk=;
b=SWDM9jPthdH90FL0QemyDYWVnoFbcJKTznA0G7zPw/IbUEKV/i2dF47W6XYq8A7MSs
IJ9kxCFG5FpjCQD044tslr0mLPvqIFPhxqnlaukfKyI7FcmC3WQdB4H3KTsbc8Dp4HHZ
OMawctgVJp6j2OxNY07IQOeQdWgKQHMO92/3UY5ojuBFn+CyL056CalthIkEeDFFOpfG
qEh3O4glfnIFm08MK1pFO/mlJeWvFTihRJD8FK2U8aX0u2oqd80HMidDf7xqoneRtNGf
tChpINaBMrrVlM838sLrBe/R+8DYpJPyaJm57Skv8BPf8hChjRzL9NlcEN8d25mnXylp
a6eg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20230601; t=1698198082; x=1698802882;
h=content-transfer-encoding:to:subject:message-id:date:from
:in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
:subject:date:message-id:reply-to;
bh=7W3zkJV41UUD3uF8oOAw19zL9BeOA3YBgCFWpvZiDfk=;
b=d5nEN+I0kJyvXIk0mNGjipJdbAjqJ4XnAZyAIp1w6pPMYWVWAdQEykhKrkArASrXQO
ENi3BUAX35Ct5EWQVqd74AsHClJnEV+kX/yKjol9bS1Sp1C63iFOV9rVh4QIaMNw/nwM
an27dmfIP7hnKAHPR7osRRrnfkMg5adjbJKckK8OPHSEyIx0+/AvCBXC3s1BqDCh56Fv
BfB7Q4/6toO41kQI8grifh/83uWP8wbWCTlTlhDvqoAre2TwWX05NDU+t0xHsQobnDBB
zjykHsCJGsFPxyMjhO0GD0PrLc5a4aUjIk54rthmeoAR+N909+yLbEtb4WkaHNwiNe3Z
OPPg==
X-Gm-Message-State: AOJu0YwKxbbAqizhvf47UFHvY2ZdEraom5tmSanLMSYPHBX4VyLVNLPL
T3uVlVbcVpF4M9u0tMPsltX6uvPCQtG2jbzp3Q0FVCQvqGA=
X-Google-Smtp-Source: AGHT+IHlsnR8TbT7Fwb8q19rHmmAQxTSj+ygbWYMDp6Nou5asXEdWFLYqqmaZ29HE9cMgL4MvRTPW5NTsWitOnacHQk=
X-Received: by 2002:a05:6512:2815:b0:508:126a:751e with SMTP id
cf21-20020a056512281500b00508126a751emr1467621lfb.36.1698198082177; Tue, 24
Oct 2023 18:41:22 -0700 (PDT)
In-Reply-To: <062901da06e1$46a074e0$d3e15ea0$@gmail.com>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <CAPTjJmqhnQcK0Jr_foTXdHQNYz_-ROp13S6Uz0nP04Muk4DQnw@mail.gmail.com>
X-Mailman-Original-References: <CAPpdf58WXyJs+Rd_GnbS-6dW8bPNt5DsojnVN5_ep+ebNx36Pw@mail.gmail.com>
<58b56dbe-646c-4a94-8102-ac2cf6efe233@tompassin.net>
<CAPpdf5_WVe07YVm-dZVV1mpZSWB6v4Z0itRsfYxYVdt8bWt0Ng@mail.gmail.com>
<062901da06e1$46a074e0$d3e15ea0$@gmail.com>

by: Chris Angelico - Wed, 25 Oct 2023 01:41 UTC

On Wed, 25 Oct 2023 at 12:20, AVI GROSS via Python-list
<python-list@python.org> wrote:
> Consider an example of bit rot. I mean what if your CPU or hard disk has a location where you can write a byte and read it back multiple times and sometimes get the wrong result. To be really cautions, you might need your software to write something in multiple locations and when it reads it back in, check all of them and if most agree, ignore the one or two that don't while blocking that memory area off and moving your data elsewhere. Or consider a memory leak that happens rarely but if a program runs for years or decades, may end up causing an unanticipated error.
>

True, but there are FAR more efficient ways to do error correction :)
Hamming codes give you single-bit correction and two-bit detection at
a cost of log N bits, which is incredibly cheap - even if you were to
go for a granularity of 64 bytes (one cache line in a modern Intel
CPU), you would need just 11 bits of Hamming code for every 512 bits
of data and you can guarantee to fix any single-bit error in any cache
line. The "if most agree, ignore the one or two that don't" design
implies that you're writing to an absolute minimum of three places,
and in order to be able to ignore two that disagree, you'd probably
need five copies of everything - that is to say, to store 512 bits of
data, you would need 2560 bits of storage. But with a Hamming code,
you need just 523 bits to store 512 reliably.

Here's a great run-down on how efficiently this can be done, and how
easily. https://www.youtube.com/watch?v=X8jsijhllIA

Side note: If we assume that random bit flips occur at a rate of one
every X storage bits, having redundant copies of data will increase
the chances of a failure happening. For example, using a naive and
horrendously wasteful "store 256 copies of everything" strategy, you
would be 256 times more likely to have a random bitflip, which is
insane :) You would also be able to guarantee detection of up to 128
random bitflips. But as you can see, this puts a maximum on your
storage ratio.

ChrisA