I am not sure what to make of this, but as a software guy, I expect the
hardware to always do the right thing or tell me it didn't.

https://www.theregister.com/2021/06/04/google_chip_flaws/

If you don't like The Register, the article contains a link to the PDF
of the ACM paper.

So, is the phenomenon real? (I assume Google is seeing what the are
seeing). Are they right that it will get worse as the technology
improves. As I said above, I don't like software solutions to this
problem, but what is the right hardware solution? Core duplication and
checking? Back to internal parity checks? Other solutions?

--
- Stephen Fuld
(e-mail address disguised to prevent spam)

On Wednesday, August 11, 2021 at 1:01:23 PM UTC-5, Stephen Fuld wrote:
> I am not sure what to make of this, but as a software guy, I expect the
> hardware to always do the right thing or tell me it didn't.
>
> https://www.theregister.com/2021/06/04/google_chip_flaws/
>
> If you don't like The Register, the article contains a link to the PDF
> of the ACM paper.
>
> So, is the phenomenon real? (I assume Google is seeing what the are
> seeing). Are they right that it will get worse as the technology
> improves. As I said above, I don't like software solutions to this
> problem, but what is the right hardware solution? Core duplication and
> checking? Back to internal parity checks? Other solutions?
<
Back in the 68020 days (say around 30 MHz) we found chips that would run
at 33ns (30 MHz) forever* but if pushed to 32ns fail through various speed
paths in the chip. These speed paths were different on different chips based
on which transistors were faster (than average) and which were slower (than
average). Moto, back then, would sell '020s in bins which were 2ns slower
than the chips would run on the testers.
>
My guess is that this phenomenon is real, and that if they declocked the
chip from 20× to 19× the vast majority of them would quit creating these
kinds of failures.
<
In a physical sense, each transistor is an experiment, and at todays
lithography, one side of the transistor may see 137 ions implanted,
while the other side (7nm away) sees 142 ions implanted. So the
transistor operates differently than theory, and sooner or later one
of the transistors in one of the millions of speed paths on a die is
slower than when the path was tested on the tester. There are all
sorts of aging phenomenon also going on.
<
In addition to the above, the transistors are now in the range where one
can count the number of atoms across the gate on 2 hands (soon 1 hand)!
At these kind of scales, quantum effects start to appear which make the
operation of the transistor more difficult to describe, model, and design
around. Instead of a MOSFET having a smooth V-I curve based on gate
voltage, the curve has wiggles in it based not only on what voltage is
being applied, but upon the change in voltage over the last picosecond.
<
At modern speeds (5GHz) even wires are displaying certain effects like
"skin effects" on voltage edges faster than 6ps and Ampere's effects on
edge speeds faster than 4ps. These effects add to the problems spoken
of above.
>
>
> --
> - Stephen Fuld
> (e-mail address disguised to prevent spam)

On Wednesday, August 11, 2021 at 2:57:20 PM UTC-5, MitchAlsup wrote:
> On Wednesday, August 11, 2021 at 1:01:23 PM UTC-5, Stephen Fuld wrote:
> > I am not sure what to make of this, but as a software guy, I expect the
> > hardware to always do the right thing or tell me it didn't.
> >
> > https://www.theregister.com/2021/06/04/google_chip_flaws/
> >
> > If you don't like The Register, the article contains a link to the PDF
> > of the ACM paper.
> >
> > So, is the phenomenon real? (I assume Google is seeing what the are
> > seeing). Are they right that it will get worse as the technology
> > improves. As I said above, I don't like software solutions to this
> > problem, but what is the right hardware solution? Core duplication and
> > checking? Back to internal parity checks? Other solutions?
> <
> Back in the 68020 days (say around 30 MHz) we found chips that would run
> at 33ns (30 MHz) forever* but if pushed to 32ns fail through various speed
> paths in the chip. These speed paths were different on different chips based
> on which transistors were faster (than average) and which were slower (than
> average). Moto, back then, would sell '020s in bins which were 2ns slower
> than the chips would run on the testers.
> >
> My guess is that this phenomenon is real, and that if they declocked the
> chip from 20× to 19× the vast majority of them would quit creating these
> kinds of failures.
> <
> In a physical sense, each transistor is an experiment, and at todays
> lithography, one side of the transistor may see 137 ions implanted,
> while the other side (7nm away) sees 142 ions implanted. So the
> transistor operates differently than theory, and sooner or later one
> of the transistors in one of the millions of speed paths on a die is
> slower than when the path was tested on the tester. There are all
> sorts of aging phenomenon also going on.
> <
> In addition to the above, the transistors are now in the range where one
> can count the number of atoms across the gate on 2 hands (soon 1 hand)!
> At these kind of scales, quantum effects start to appear which make the
> operation of the transistor more difficult to describe, model, and design
> around. Instead of a MOSFET having a smooth V-I curve based on gate
> voltage, the curve has wiggles in it based not only on what voltage is
> being applied, but upon the change in voltage over the last picosecond.
> <
> At modern speeds (5GHz) even wires are displaying certain effects like
> "skin effects" on voltage edges faster than 6ps and Ampere's effects on
> edge speeds faster than 4ps. These effects add to the problems spoken
> of above.
<
After reading the PDF and thinking about it for an hour::
<
This reads a lot like SIMD considered bad. For example, they change a
library (probably from straight x86 to one using the vector instructions
:: SSE[k]) and all of a sudden, various corruptions start happening.
<
They describe a CPU like a set of common registers surrounded by an
amalgam of accelerators--these accelerators are not the stuff x86
had originally.
<
It sounds a bit like the Pentium FDIV bug and a bit like -- "if it hurts
QUIT doing it."
> >
> >
> > --
> > - Stephen Fuld
> > (e-mail address disguised to prevent spam)

> My guess is that this phenomenon is real, and that if they declocked the
> chip from 20× to 19× the vast majority of them would quit creating these
> kinds of failures.

I guess there's also the difficulty to account for all the possible
cases during testing, due to the complexity of the cores, multiplied by
the possible operating circumstances (temperature, voltage, frequency,
multiple frequency domains, ...). Also I'm not sure how voltage and
frequency transitions are implemented but I'd be surprised if they don't
make all of that yet a bit more interesting since they are applied quite
often nowadays so they need to be reasonably fast.

All in all, the reliability we experience is nothing more than astounding.

Stefan

On Wednesday, August 11, 2021 at 3:49:49 PM UTC-5, Stefan Monnier wrote:
> > My guess is that this phenomenon is real, and that if they declocked the
> > chip from 20× to 19× the vast majority of them would quit creating these
> > kinds of failures.
<
> I guess there's also the difficulty to account for all the possible
> cases during testing, due to the complexity of the cores, multiplied by
> the possible operating circumstances (temperature, voltage, frequency,
> multiple frequency domains, ...).
<
Back in the ½µ days, with asynchronous interfaces, it could take 1T test
vectors to properly test a part. This corresponds to 10% of the chips service
life !! Not to mention the $5,000 per hour of tester "value"...........
<
> Also I'm not sure how voltage and
> frequency transitions are implemented but I'd be surprised if they don't
> make all of that yet a bit more interesting since they are applied quite
> often nowadays so they need to be reasonably fast.
<
You just multiplied the problem space by about 1000×
>
> All in all, the reliability we experience is nothing more than astounding..
<
If those data centers did not have 100,000 of cores, they might never notice
the problem, either.
>
>
> Stefan

MitchAlsup wrote:
> On Wednesday, August 11, 2021 at 2:57:20 PM UTC-5, MitchAlsup wrote:
>> On Wednesday, August 11, 2021 at 1:01:23 PM UTC-5, Stephen Fuld wrote:
>>> I am not sure what to make of this, but as a software guy, I expect the
>>> hardware to always do the right thing or tell me it didn't.
>>>
>>> https://www.theregister.com/2021/06/04/google_chip_flaws/
>>>
>>> If you don't like The Register, the article contains a link to the PDF
>>> of the ACM paper.
>>>
>>> So, is the phenomenon real? (I assume Google is seeing what the are
>>> seeing). Are they right that it will get worse as the technology
>>> improves. As I said above, I don't like software solutions to this
>>> problem, but what is the right hardware solution? Core duplication and
>>> checking? Back to internal parity checks? Other solutions?
>> <
> <
> After reading the PDF and thinking about it for an hour::
> <
> This reads a lot like SIMD considered bad. For example, they change a
> library (probably from straight x86 to one using the vector instructions
> :: SSE[k]) and all of a sudden, various corruptions start happening.
> <
> They describe a CPU like a set of common registers surrounded by an
> amalgam of accelerators--these accelerators are not the stuff x86
> had originally.
> <
> It sounds a bit like the Pentium FDIV bug and a bit like -- "if it hurts
> QUIT doing it."

I read it as a plea for more automatic internal self checks
so they can detect the intermittent problems when they occur.
Equivalent to memory ECC but for logic.

If they can't determine which cpu's are hurting and when,
they can't quit doing it.

On Wednesday, August 11, 2021 at 5:35:02 PM UTC-5, EricP wrote:
> MitchAlsup wrote:
> > On Wednesday, August 11, 2021 at 2:57:20 PM UTC-5, MitchAlsup wrote:
> >> On Wednesday, August 11, 2021 at 1:01:23 PM UTC-5, Stephen Fuld wrote:
> >>> I am not sure what to make of this, but as a software guy, I expect the
> >>> hardware to always do the right thing or tell me it didn't.
> >>>
> >>> https://www.theregister.com/2021/06/04/google_chip_flaws/
> >>>
> >>> If you don't like The Register, the article contains a link to the PDF
> >>> of the ACM paper.
> >>>
> >>> So, is the phenomenon real? (I assume Google is seeing what the are
> >>> seeing). Are they right that it will get worse as the technology
> >>> improves. As I said above, I don't like software solutions to this
> >>> problem, but what is the right hardware solution? Core duplication and
> >>> checking? Back to internal parity checks? Other solutions?
> >> <
> > <
> > After reading the PDF and thinking about it for an hour::
> > <
> > This reads a lot like SIMD considered bad. For example, they change a
> > library (probably from straight x86 to one using the vector instructions
> > :: SSE[k]) and all of a sudden, various corruptions start happening.
> > <
> > They describe a CPU like a set of common registers surrounded by an
> > amalgam of accelerators--these accelerators are not the stuff x86
> > had originally.
> > <
> > It sounds a bit like the Pentium FDIV bug and a bit like -- "if it hurts
> > QUIT doing it."
<
> I read it as a plea for more automatic internal self checks
> so they can detect the intermittent problems when they occur.
> Equivalent to memory ECC but for logic.
<
Expensive. Consider, for the moment, how much logic would it take to
verify a multiplier gave you the correct bit patterns ?
<
Store {registers, caches, SRAM, DRAM} are all easily protected using various
kinds of error detecting and correcting codes. Buses and wires that simply
move bits around can utilize the same.
<
Verifying arithmetic is hard, verifying control units is even harder. I once
worked on the design of a machine which could deliver all of the correct
bit patterns to all the correct locations, taking the correct number of
cycles, and having done all this in the WRONG sequence order !!
>
> If they can't determine which cpu's are hurting and when,
> they can't quit doing it.
<
They can take all of the suspected units and turn the clock multiplier down
by 1 or 2 units of multiplication. That is what I meant by quit doing that.

On 8/11/2021 4:10 PM, MitchAlsup wrote:
> On Wednesday, August 11, 2021 at 5:35:02 PM UTC-5, EricP wrote:
>> MitchAlsup wrote:
>>> On Wednesday, August 11, 2021 at 2:57:20 PM UTC-5, MitchAlsup wrote:
>>>> On Wednesday, August 11, 2021 at 1:01:23 PM UTC-5, Stephen Fuld wrote:
>>>>> I am not sure what to make of this, but as a software guy, I expect the
>>>>> hardware to always do the right thing or tell me it didn't.
>>>>>
>>>>> https://www.theregister.com/2021/06/04/google_chip_flaws/
>>>>>
>>>>> If you don't like The Register, the article contains a link to the PDF
>>>>> of the ACM paper.
>>>>>
>>>>> So, is the phenomenon real? (I assume Google is seeing what the are
>>>>> seeing). Are they right that it will get worse as the technology
>>>>> improves. As I said above, I don't like software solutions to this
>>>>> problem, but what is the right hardware solution? Core duplication and
>>>>> checking? Back to internal parity checks? Other solutions?
>>>> <
>>> <
>>> After reading the PDF and thinking about it for an hour::
>>> <
>>> This reads a lot like SIMD considered bad. For example, they change a
>>> library (probably from straight x86 to one using the vector instructions
>>> :: SSE[k]) and all of a sudden, various corruptions start happening.
>>> <
>>> They describe a CPU like a set of common registers surrounded by an
>>> amalgam of accelerators--these accelerators are not the stuff x86
>>> had originally.
>>> <
>>> It sounds a bit like the Pentium FDIV bug and a bit like -- "if it hurts
>>> QUIT doing it."
> <
>> I read it as a plea for more automatic internal self checks
>> so they can detect the intermittent problems when they occur.
>> Equivalent to memory ECC but for logic.
> <
> Expensive. Consider, for the moment, how much logic would it take to
> verify a multiplier gave you the correct bit patterns ?
> <
> Store {registers, caches, SRAM, DRAM} are all easily protected using various
> kinds of error detecting and correcting codes. Buses and wires that simply
> move bits around can utilize the same.
> <
> Verifying arithmetic is hard, verifying control units is even harder. I once
> worked on the design of a machine which could deliver all of the correct
> bit patterns to all the correct locations, taking the correct number of
> cycles, and having done all this in the WRONG sequence order !!
>>
>> If they can't determine which cpu's are hurting and when,
>> they can't quit doing it.
> <
> They can take all of the suspected units and turn the clock multiplier down
> by 1 or 2 units of multiplication. That is what I meant by quit doing that.
>

At a certain point voting redundancy is cheaper.

On Wednesday, August 11, 2021 at 7:35:32 PM UTC-5, Ivan Godard wrote:
> On 8/11/2021 4:10 PM, MitchAlsup wrote:
> > On Wednesday, August 11, 2021 at 5:35:02 PM UTC-5, EricP wrote:
> >> MitchAlsup wrote:
> >>> On Wednesday, August 11, 2021 at 2:57:20 PM UTC-5, MitchAlsup wrote:
> >>>> On Wednesday, August 11, 2021 at 1:01:23 PM UTC-5, Stephen Fuld wrote:
> >>>>> I am not sure what to make of this, but as a software guy, I expect the
> >>>>> hardware to always do the right thing or tell me it didn't.
> >>>>>
> >>>>> https://www.theregister.com/2021/06/04/google_chip_flaws/
> >>>>>
> >>>>> If you don't like The Register, the article contains a link to the PDF
> >>>>> of the ACM paper.
> >>>>>
> >>>>> So, is the phenomenon real? (I assume Google is seeing what the are
> >>>>> seeing). Are they right that it will get worse as the technology
> >>>>> improves. As I said above, I don't like software solutions to this
> >>>>> problem, but what is the right hardware solution? Core duplication and
> >>>>> checking? Back to internal parity checks? Other solutions?
> >>>> <
> >>> <
> >>> After reading the PDF and thinking about it for an hour::
> >>> <
> >>> This reads a lot like SIMD considered bad. For example, they change a
> >>> library (probably from straight x86 to one using the vector instructions
> >>> :: SSE[k]) and all of a sudden, various corruptions start happening.
> >>> <
> >>> They describe a CPU like a set of common registers surrounded by an
> >>> amalgam of accelerators--these accelerators are not the stuff x86
> >>> had originally.
> >>> <
> >>> It sounds a bit like the Pentium FDIV bug and a bit like -- "if it hurts
> >>> QUIT doing it."
> > <
> >> I read it as a plea for more automatic internal self checks
> >> so they can detect the intermittent problems when they occur.
> >> Equivalent to memory ECC but for logic.
> > <
> > Expensive. Consider, for the moment, how much logic would it take to
> > verify a multiplier gave you the correct bit patterns ?
> > <
> > Store {registers, caches, SRAM, DRAM} are all easily protected using various
> > kinds of error detecting and correcting codes. Buses and wires that simply
> > move bits around can utilize the same.
> > <
> > Verifying arithmetic is hard, verifying control units is even harder. I once
> > worked on the design of a machine which could deliver all of the correct
> > bit patterns to all the correct locations, taking the correct number of
> > cycles, and having done all this in the WRONG sequence order !!
> >>
> >> If they can't determine which cpu's are hurting and when,
> >> they can't quit doing it.
> > <
> > They can take all of the suspected units and turn the clock multiplier down
> > by 1 or 2 units of multiplication. That is what I meant by quit doing that.
> >
> At a certain point voting redundancy is cheaper.
<
DIV 3 is cheaper than -5%

On Wednesday, August 11, 2021 at 12:01:23 PM UTC-6, Stephen Fuld wrote:

> So, is the phenomenon real?

Well, since chips today are being made on ever-finer process nodes, it
makes sense that in addition to yields being affected - which, of course,
the foundries work on until chips can be economically produced - the
chance of other, subtler problems that are harder to detect also
increases.

IBM designed their mainframes with a lot of internal arithmetic
checking to prevent part failures from leading to erroneous results,
as these would be unacceptable to commercial data processing
customers. So techniques are known for dealing with this - the
kind that IBM used, and more exotic techniques used in spacecraft
like triple redundancy.

When I saw the title "mercurial cores", though, right away it made
me think of "Jellicle songs for Jellicle cats"!

John Savard

Stephen Fuld <sfuld@alumni.cmu.edu.invalid> writes:
>I am not sure what to make of this, but as a software guy, I expect the
>hardware to always do the right thing or tell me it didn't.
>
>https://www.theregister.com/2021/06/04/google_chip_flaws/
>
>If you don't like The Register, the article contains a link to the PDF
>of the ACM paper.

Is it in any way forbidden to post that link here?

https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s01-hochschild.pdf

There, I did it.

>So, is the phenomenon real? (I assume Google is seeing what the are
>seeing).

I have had two personal machines (IIRC in 1993 and 2003) that
corrupted data rarely enough that the machines did not crash. On the
job a few years ago a Northwood-based machine started acting up after
~15 years in service, but did not crash outright, so our technician at
first did not suspect a hardware failure; I did, because we had not
changed anything in the software, and eventually the technician agreed
with me.

>Are they right that it will get worse as the technology
>improves.

What kind of improvement would that be?

>As I said above, I don't like software solutions to this
>problem, but what is the right hardware solution? Core duplication and
>checking?

We already have core duplication, doing checking of the stuff that,
say, goes outside the per-core state (i.e., outside L2 in current
CPUs) should be achievable (although probably not easy, or CPU
manufacturers would put it in their server CPUs already).

>Back to internal parity checks?

I guess that current Intel and AMD CPUs have a lot of that already.
Overclockers report that overclocking these CPUs with ordinary (not
LN2) cooling and within their usual power limits produces hardly any
improvement, so the CPU manufacturers manage to run these CPUs pretty
close to their limits. I doubt that they ran these CPUs for hours and
hours on the expensive testers to determine which core works at which
clock at a given frequency. Instead, I guess that the CPUs do much of
the testing themselves, and repeat a part of it on startup or maybe
now and then in SMM to account for degradation. And maybe they also
have something like parity checks during production work.

Maybe what Google is seeing are some cases where the startup testing
is not good enough. In that case, the problem may be solvable with
additional test hardware and test vectors.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>

MitchAlsup <MitchAlsup@aol.com> writes:
>Expensive. Consider, for the moment, how much logic would it take to
>verify a multiplier gave you the correct bit patterns ?

Compute the hexadecimal digital root (repeated digital sum) dr of the
multipliers a and b, and the result p. Now,

dr(p)=dr(dr(a)*dr(b))

Should be pretty easy to check, and you can use other bases than hex
to vary cost and precision. You can also use the digital root for
checking addition and subtraction.

>Verifying arithmetic is hard,

For data in GPRs, if computation is wrong now and then, it will affect
address computation after a while, and that will show up as
exceptions. For data in FPRs and SIMD registers, errors tend to show
up only as wrong results (witness how long it took to notice the
Pentium FDIV bug). So for that we certainly want extra checking.

> verifying control units is even harder.

But then, control units have an effect through the functional units
they control. So, again, when it's a functional unit that deals with
addresses and control flow, errors tend to result in exceptions. When
it's a functional unit that deals with FP or SIMD data, errors tend to
result in silent wrong results; and when the wrong control input also
goes to the checking circuit, even that does not help; so the checking
circuit should have its own control signal circuitry as far as
possible.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>

On 8/12/2021 8:31 AM, Anton Ertl wrote:
> Stephen Fuld <sfuld@alumni.cmu.edu.invalid> writes:
>> I am not sure what to make of this, but as a software guy, I expect the
>> hardware to always do the right thing or tell me it didn't.
>>
>> https://www.theregister.com/2021/06/04/google_chip_flaws/
>>
>> If you don't like The Register, the article contains a link to the PDF
>> of the ACM paper.
>
> Is it in any way forbidden to post that link here?
>
> https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s01-hochschild.pdf
>
> There, I did it.
>
>> So, is the phenomenon real? (I assume Google is seeing what the are
>> seeing).
>
> I have had two personal machines (IIRC in 1993 and 2003) that
> corrupted data rarely enough that the machines did not crash. On the
> job a few years ago a Northwood-based machine started acting up after
> ~15 years in service, but did not crash outright, so our technician at
> first did not suspect a hardware failure; I did, because we had not
> changed anything in the software, and eventually the technician agreed
> with me.
>
>> Are they right that it will get worse as the technology
>> improves.
>
> What kind of improvement would that be?
>
>> As I said above, I don't like software solutions to this
>> problem, but what is the right hardware solution? Core duplication and
>> checking?
>
> We already have core duplication, doing checking of the stuff that,
> say, goes outside the per-core state (i.e., outside L2 in current
> CPUs) should be achievable (although probably not easy, or CPU
> manufacturers would put it in their server CPUs already).
>
>> Back to internal parity checks?
>
> I guess that current Intel and AMD CPUs have a lot of that already.
> Overclockers report that overclocking these CPUs with ordinary (not
> LN2) cooling and within their usual power limits produces hardly any
> improvement, so the CPU manufacturers manage to run these CPUs pretty
> close to their limits. I doubt that they ran these CPUs for hours and
> hours on the expensive testers to determine which core works at which
> clock at a given frequency. Instead, I guess that the CPUs do much of
> the testing themselves, and repeat a part of it on startup or maybe
> now and then in SMM to account for degradation. And maybe they also
> have something like parity checks during production work.
>
> Maybe what Google is seeing are some cases where the startup testing
> is not good enough. In that case, the problem may be solvable with
> additional test hardware and test vectors.
>

I had a Phenom II which worked fine for years at the stock speed (though
did run a little hot), but at one point started having problems so I
ended up needing to underclock it for stability.

Then, later, I ran an AMD FX for which I also ended up needing to
disable the turbo feature and also underclock it slightly (running it at
3.8 rather than 4.2). Partly this was because in my tests, the turbo
on-average actually made performance worse under load, and it still ran
pretty hot.

At least for the Ryzen I have, I am able to run it at stock speeds...

Though it does still have some bizarre scheduling behavior that I have
not been able to figure out the cause:
Namely, by default, any given process seems only able to use 2 "logical
processors" on a single core; though Cinebench somehow sidesteps this
and uses all the cores.

Though, I guess it does sort of work out OK as I am frequently running 4
or so Verilog simulations at the same time in the background.

....

BGB wrote:
> On 8/12/2021 8:31 AM, Anton Ertl wrote:
>> Stephen Fuld <sfuld@alumni.cmu.edu.invalid> writes:
>>> I am not sure what to make of this, but as a software guy, I expect the
>>> hardware to always do the right thing or tell me it didn't.
>>>
>>> https://www.theregister.com/2021/06/04/google_chip_flaws/
>>>
>>> If you don't like The Register, the article contains a link to the PDF
>>> of the ACM paper.
>>
>> Is it in any way forbidden to post that link here?
>>
>> https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s01-hochschild.pdf
>>
>>
>> There, I did it.
>>
>>> So, is the phenomenon real?  (I assume Google is seeing what the are
>>> seeing).
>>
>> I have had two personal machines (IIRC in 1993 and 2003) that
>> corrupted data rarely enough that the machines did not crash.  On the
>> job a few years ago a Northwood-based machine started acting up after
>> ~15 years in service, but did not crash outright, so our technician at
>> first did not suspect a hardware failure; I did, because we had not
>> changed anything in the software, and eventually the technician agreed
>> with me.
>>
>>> Are they right that it will get worse as the technology
>>> improves.
>>
>> What kind of improvement would that be?
>>
>>> As I said above, I don't like software solutions to this
>>> problem, but what is the right hardware solution?  Core duplication and
>>> checking?
>>
>> We already have core duplication, doing checking of the stuff that,
>> say, goes outside the per-core state (i.e., outside L2 in current
>> CPUs) should be achievable (although probably not easy, or CPU
>> manufacturers would put it in their server CPUs already).
>>
>>> Back to internal parity checks?
>>
>> I guess that current Intel and AMD CPUs have a lot of that already.
>> Overclockers report that overclocking these CPUs with ordinary (not
>> LN2) cooling and within their usual power limits produces hardly any
>> improvement, so the CPU manufacturers manage to run these CPUs pretty
>> close to their limits.  I doubt that they ran these CPUs for hours and
>> hours on the expensive testers to determine which core works at which
>> clock at a given frequency.  Instead, I guess that the CPUs do much of
>> the testing themselves, and repeat a part of it on startup or maybe
>> now and then in SMM to account for degradation.  And maybe they also
>> have something like parity checks during production work.
>>
>> Maybe what Google is seeing are some cases where the startup testing
>> is not good enough.  In that case, the problem may be solvable with
>> additional test hardware and test vectors.
>>
>
> I had a Phenom II which worked fine for years at the stock speed (though
> did run a little hot), but at one point started having problems so I
> ended up needing to underclock it for stability.
>
>
> Then, later, I ran an AMD FX for which I also ended up needing to
> disable the turbo feature and also underclock it slightly (running it at
> 3.8 rather than 4.2). Partly this was because in my tests, the turbo
> on-average actually made performance worse under load, and it still ran
> pretty hot.
>
>
> At least for the Ryzen I have, I am able to run it at stock speeds...
>
> Though it does still have some bizarre scheduling behavior that I have
> not been able to figure out the cause:
> Namely, by default, any given process seems only able to use 2 "logical
> processors" on a single core; though Cinebench somehow sidesteps this
> and uses all the cores.
>
> Though, I guess it does sort of work out OK as I am frequently running 4
> or so Verilog simulations at the same time in the background.

I have seen the same with at least two laptops which have been used for
batch processing of ~25 TB of lidar data: I had to underclock them to
avoid crashes or sudden shutdowns (probably caused by vector ops on all
cores that increased the core temperature too quickly?) with no warnings
or crash dumps.

No problems after that 20% reduction in max speed, and only a much
smaller throughput degradation.

Terje

--
- <Terje.Mathisen at tmsw.no>
"almost all programming can be viewed as an exercise in caching"

On 8/12/2021 1:44 PM, Terje Mathisen wrote:
> BGB wrote:
>> On 8/12/2021 8:31 AM, Anton Ertl wrote:
>>> Stephen Fuld <sfuld@alumni.cmu.edu.invalid> writes:
>>>> I am not sure what to make of this, but as a software guy, I expect the
>>>> hardware to always do the right thing or tell me it didn't.
>>>>
>>>> https://www.theregister.com/2021/06/04/google_chip_flaws/
>>>>
>>>> If you don't like The Register, the article contains a link to the PDF
>>>> of the ACM paper.
>>>
>>> Is it in any way forbidden to post that link here?
>>>
>>> https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s01-hochschild.pdf
>>>
>>>
>>> There, I did it.
>>>
>>>> So, is the phenomenon real?  (I assume Google is seeing what the are
>>>> seeing).
>>>
>>> I have had two personal machines (IIRC in 1993 and 2003) that
>>> corrupted data rarely enough that the machines did not crash.  On the
>>> job a few years ago a Northwood-based machine started acting up after
>>> ~15 years in service, but did not crash outright, so our technician at
>>> first did not suspect a hardware failure; I did, because we had not
>>> changed anything in the software, and eventually the technician agreed
>>> with me.
>>>
>>>> Are they right that it will get worse as the technology
>>>> improves.
>>>
>>> What kind of improvement would that be?
>>>
>>>> As I said above, I don't like software solutions to this
>>>> problem, but what is the right hardware solution?  Core duplication and
>>>> checking?
>>>
>>> We already have core duplication, doing checking of the stuff that,
>>> say, goes outside the per-core state (i.e., outside L2 in current
>>> CPUs) should be achievable (although probably not easy, or CPU
>>> manufacturers would put it in their server CPUs already).
>>>
>>>> Back to internal parity checks?
>>>
>>> I guess that current Intel and AMD CPUs have a lot of that already.
>>> Overclockers report that overclocking these CPUs with ordinary (not
>>> LN2) cooling and within their usual power limits produces hardly any
>>> improvement, so the CPU manufacturers manage to run these CPUs pretty
>>> close to their limits.  I doubt that they ran these CPUs for hours and
>>> hours on the expensive testers to determine which core works at which
>>> clock at a given frequency.  Instead, I guess that the CPUs do much of
>>> the testing themselves, and repeat a part of it on startup or maybe
>>> now and then in SMM to account for degradation.  And maybe they also
>>> have something like parity checks during production work.
>>>
>>> Maybe what Google is seeing are some cases where the startup testing
>>> is not good enough.  In that case, the problem may be solvable with
>>> additional test hardware and test vectors.
>>>
>>
>> I had a Phenom II which worked fine for years at the stock speed
>> (though did run a little hot), but at one point started having
>> problems so I ended up needing to underclock it for stability.
>>
>>
>> Then, later, I ran an AMD FX for which I also ended up needing to
>> disable the turbo feature and also underclock it slightly (running it
>> at 3.8 rather than 4.2). Partly this was because in my tests, the
>> turbo on-average actually made performance worse under load, and it
>> still ran pretty hot.
>>
>>
>> At least for the Ryzen I have, I am able to run it at stock speeds...
>>
>> Though it does still have some bizarre scheduling behavior that I have
>> not been able to figure out the cause:
>> Namely, by default, any given process seems only able to use 2
>> "logical processors" on a single core; though Cinebench somehow
>> sidesteps this and uses all the cores.
>>
>> Though, I guess it does sort of work out OK as I am frequently running
>> 4 or so Verilog simulations at the same time in the background.
>
> I have seen the same with at least two laptops which have been used for
> batch processing of ~25 TB of lidar data: I had to underclock them to
> avoid crashes or sudden shutdowns (probably caused by vector ops on all
> cores that increased the core temperature too quickly?) with no warnings
> or crash dumps.
>
> No problems after that 20% reduction in max speed, and only a much
> smaller throughput degradation.
>

Yeah, pretty much.

The Phenom II became unstable at the base clock and needed to be
underclocked to be moderately stable.

Mostly it was thermal issues with the AMD FX (Piledriver). In its stock
settings, it would only run for short bursts at the turbo frequency, and
then throttle down really bad.

Say, it jumped 4.3 GHz briefly, overheat, drop to 1.3GHz, stay there for
a while, jump back to 4.3 GHz briefly, ...

Disabling turbo reduced this issue, but it still ran pretty hot and was
prone to throttling or crashing under load.

Dropping it down to 3.8 (with turbo disabled) caused a notable reduction
in temperature with minimal impact on performance (and I could run it at
close to full load without it going into thermal throttle or crashing).

Granted, I was using an air-cooled setup. Maybe people had better
results with water cooling, but alas.

When I later got a Ryzen, it has a lower Base and Turbo frequency than
the FX had, but still gives better performance, and runs a lot cooler
under load.

Not sure how much was due to efficiency, lower clock speeds, or that the
Ryzen came with a comparably larger stock heatsink (~ 5x5x4 inches).

But, yeah, possibly it is a somewhat different workload between a "Leet
Gamer" and "someone who leaves their PC at 40-70% CPU load for extended
periods while running Verilog simulations and genetic-algorithms, and
similar...".

On Thursday, August 12, 2021 at 1:44:46 PM UTC-5, Terje Mathisen wrote:
> BGB wrote:

> I have seen the same with at least two laptops which have been used for
> batch processing of ~25 TB of lidar data: I had to underclock them to
> avoid crashes or sudden shutdowns (probably caused by vector ops on all
> cores that increased the core temperature too quickly?) with no warnings
> or crash dumps.
>
> No problems after that 20% reduction in max speed, and only a much
> smaller throughput degradation.
<
Wouldn't it be nice if more applications were like games ? In most games
a poor calculation only results in a single pixel have the wrong shade of
{RGB} on the screen and vanishes the next frame--the phenomenon is
called "shimmer"
<
>
> Terje
>
> --
> - <Terje.Mathisen at tmsw.no>
> "almost all programming can be viewed as an exercise in caching"

On Thursday, August 12, 2021 at 3:02:02 PM UTC-6, MitchAlsup wrote:

> Wouldn't it be nice if more applications were like games ? In most games
> a poor calculation only results in a single pixel have the wrong shade of
> {RGB} on the screen and vanishes the next frame--the phenomenon is
> called "shimmer"

Well, given the increasing importance of AI based on matrices of low-precision
floats, you may get your wish.

But a lot of applications are not anything like games, and demand perfect
accuracy from trillions of calculations. As the technology to provide this is
known, the problem is the intensity of the competition to provide the highest
possible performance at the lowest possible cost...

which leads manufacturers to claim to provide more performance than is
possible at a lower price than is possible, at least with the constraint of
error-free computation.

Since many people only really drive the computational capabilities of their
computers heavily when playing games... perhaps there's room here for
some sort of dual-mode operation.

My initial reaction to that sentence, though, was of the order of "dream
on". Unless _all_ applications are "like games" in the sense you outline,
a computer has to be capable of error-free operation. I don't see that there's
really an escape from that.

John Savard

On Thursday, August 12, 2021 at 6:39:42 PM UTC-5, Quadibloc wrote:
> On Thursday, August 12, 2021 at 3:02:02 PM UTC-6, MitchAlsup wrote:
>
> > Wouldn't it be nice if more applications were like games ? In most games
> > a poor calculation only results in a single pixel have the wrong shade of
> > {RGB} on the screen and vanishes the next frame--the phenomenon is
> > called "shimmer"
> Well, given the increasing importance of AI based on matrices of low-precision
> floats, you may get your wish.
>
> But a lot of applications are not anything like games, and demand perfect
> accuracy from trillions of calculations. As the technology to provide this is
> known, the problem is the intensity of the competition to provide the highest
> possible performance at the lowest possible cost...
<
There is the old adage:: Good, fast, cheap -- choose any 2.
<
So, for the most part they trow out good in order to get fast and cheap
>
> which leads manufacturers to claim to provide more performance than is
> possible at a lower price than is possible, at least with the constraint of
> error-free computation.
<
Reminds me of 1950s automobiles and the public resistance to seat belts.
>
> Since many people only really drive the computational capabilities of their
> computers heavily when playing games... perhaps there's room here for
> some sort of dual-mode operation.
>
> My initial reaction to that sentence, though, was of the order of "dream
> on". Unless _all_ applications are "like games" in the sense you outline,
> a computer has to be capable of error-free operation. I don't see that there's
> really an escape from that.
>
> John Savard

On 8/12/2021 6:31 AM, Anton Ertl wrote:
> Stephen Fuld <sfuld@alumni.cmu.edu.invalid> writes:
>> I am not sure what to make of this, but as a software guy, I expect the
>> hardware to always do the right thing or tell me it didn't.
>>
>> https://www.theregister.com/2021/06/04/google_chip_flaws/
>>
>> If you don't like The Register, the article contains a link to the PDF
>> of the ACM paper.
>
> Is it in any way forbidden to post that link here?
>
> https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s01-hochschild.pdf
>
> There, I did it.

No reason not to post the link. I first came upon the story in The
Register through my Apple news feed, so that is the link I used. On
behalf of those who wanted the full story right away, thanks for posting
the link.

>
>> So, is the phenomenon real? (I assume Google is seeing what the are
>> seeing).
>
> I have had two personal machines (IIRC in 1993 and 2003) that
> corrupted data rarely enough that the machines did not crash. On the
> job a few years ago a Northwood-based machine started acting up after
> ~15 years in service, but did not crash outright, so our technician at
> first did not suspect a hardware failure; I did, because we had not
> changed anything in the software, and eventually the technician agreed
> with me.
>
>> Are they right that it will get worse as the technology
>> improves.
>
> What kind of improvement would that be?

The indicate things like smaller lithography.

>
>> As I said above, I don't like software solutions to this
>> problem, but what is the right hardware solution? Core duplication and
>> checking?
>
> We already have core duplication, doing checking of the stuff that,
> say, goes outside the per-core state (i.e., outside L2 in current
> CPUs) should be achievable (although probably not easy, or CPU
> manufacturers would put it in their server CPUs already).

Presumably it also increased cost, as it halves the number of cores per
chip that can do useful work.

>> Back to internal parity checks?
>
> I guess that current Intel and AMD CPUs have a lot of that already.

Certainly someone like Mitch would be better able to answer that, but my
impression was that they didn't.

--
- Stephen Fuld
(e-mail address disguised to prevent spam)

According to Stephen Fuld <sfuld@alumni.cmu.edu.invalid>:
>So, is the phenomenon real? (I assume Google is seeing what they are seeing).

Back in the 1950s and 1950s computers were, by our standards extremely unreliable.

In 1948 the IBM SSEC did everything twice in parallel and compared.
Programmers had to hang around while their programs ran to recover
manually when the compares failed, or other things failed, and get it
restarted.

The IAS machine in Princeton sort of started working in 1951 but it
required huge effort and another year (and as it turned out huge
cooling) to get it to work for more than a few seconds at a time.

The IBM 704 was more reliable than its predecessors, partly because they had learned
more about making reliable vacuum tubes, and mostly because it used core memory rather
than flaky Williams tubes or Selectrons. Nonetheless, I hear that the maximum practical
size for a Fortran program was limited not by the size of memory but by how long the
compiler could run before the hardware flaked and it crashed.

The details of the mercurial failures aren't the same as the failures of vacuum tubes,
of course, but they share the fundamental problem of trying to build digital systems
out of fundamentally analog components.

--
Regards,
John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

On 8/12/2021 8:00 PM, John Levine wrote:
> According to Stephen Fuld <sfuld@alumni.cmu.edu.invalid>:
>> So, is the phenomenon real? (I assume Google is seeing what they are seeing).
>
> Back in the 1950s and 1950s computers were, by our standards extremely unreliable.
>
> In 1948 the IBM SSEC did everything twice in parallel and compared.
> Programmers had to hang around while their programs ran to recover
> manually when the compares failed, or other things failed, and get it
> restarted.
>
> The IAS machine in Princeton sort of started working in 1951 but it
> required huge effort and another year (and as it turned out huge
> cooling) to get it to work for more than a few seconds at a time.
>
> The IBM 704 was more reliable than its predecessors, partly because they had learned
> more about making reliable vacuum tubes, and mostly because it used core memory rather
> than flaky Williams tubes or Selectrons. Nonetheless, I hear that the maximum practical
> size for a Fortran program was limited not by the size of memory but by how long the
> compiler could run before the hardware flaked and it crashed.
>
> The details of the mercurial failures aren't the same as the failures of vacuum tubes,
> of course, but they share the fundamental problem of trying to build digital systems
> out of fundamentally analog components.
>

When writing some of the system software for the B6500, using the System
101 hardware prototype, the others in the software team had a MTTF of
~five minutes. For me, however, 101 would stay up for ~half an hour. I
am convinced my success came because, when it finished a compile, I
would thank it and pat the console.

Stephen Fuld <sfuld@alumni.cmu.edu.invalid> writes:
>On 8/12/2021 6:31 AM, Anton Ertl wrote:
>> We already have core duplication, doing checking of the stuff that,
>> say, goes outside the per-core state (i.e., outside L2 in current
>> CPUs) should be achievable (although probably not easy, or CPU
>> manufacturers would put it in their server CPUs already).
>
>Presumably it also increased cost, as it halves the number of cores per
>chip that can do useful work.

Only if you run the cores in the checking mode. For the manufacturers
that would be just an optional feature that they offer to their
customers. You might consider that the customers may not want to
double the number of cores they buy/rent in order to get this
checking, but I think that there are customers that would use that
option. After all, Tandem was a successful business for a while, and
I think their later fate was not due to the customer base vanishing.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>

MitchAlsup wrote:
> On Thursday, August 12, 2021 at 1:44:46 PM UTC-5, Terje Mathisen wrote:
>> BGB wrote:
>
>> I have seen the same with at least two laptops which have been used for
>> batch processing of ~25 TB of lidar data: I had to underclock them to
>> avoid crashes or sudden shutdowns (probably caused by vector ops on all
>> cores that increased the core temperature too quickly?) with no warnings
>> or crash dumps.
>>
>> No problems after that 20% reduction in max speed, and only a much
>> smaller throughput degradation.
> <
> Wouldn't it be nice if more applications were like games ? In most games
> a poor calculation only results in a single pixel have the wrong shade of
> {RGB} on the screen and vanishes the next frame--the phenomenon is
> called "shimmer"

I've told the story about how iD Software (Mike Abrash/John Carmack/etc)
spent a couple of weeks chasing exactly such a shimmer where Mike's
daughter with fresh young eyes saw the glitch. They finally found out
that their white box PC vendor had uplocked a 90 MHz cpu to 100 MHz
("because testing has shown it to be good").

Terje

--
- <Terje.Mathisen at tmsw.no>
"almost all programming can be viewed as an exercise in caching"

John Levine wrote:
> According to Stephen Fuld <sfuld@alumni.cmu.edu.invalid>:
>> So, is the phenomenon real? (I assume Google is seeing what they are seeing).
>
> Back in the 1950s and 1950s computers were, by our standards extremely unreliable.

Even in the seventies mainframes competed on how fast they could
snapshot, crash, reboot the os, reload the snapshot and recover in order
to handle faults.
:-)

I think they got that process down to a few seconds?

Terje

--
- <Terje.Mathisen at tmsw.no>
"almost all programming can be viewed as an exercise in caching"

On Friday, August 13, 2021 at 12:00:30 PM UTC+3, Terje Mathisen wrote:
> John Levine wrote:
> > According to Stephen Fuld <sf...@alumni.cmu.edu.invalid>:
> >> So, is the phenomenon real? (I assume Google is seeing what they are seeing).
> >
> > Back in the 1950s and 1950s computers were, by our standards extremely unreliable.
> Even in the seventies mainframes competed on how fast they could
> snapshot, crash, reboot the os, reload the snapshot and recover in order
> to handle faults.
> :-)
>
> I think they got that process down to a few seconds?
> Terje
>
> --
> - <Terje.Mathisen at tmsw.no>
> "almost all programming can be viewed as an exercise in caching"

My impression was that in the second half of the 70s IBM started to take reliability
seriously and very quickly achieved huge progress on this front.
The same happened to [mainframe] security few years later.
But I was not around back then.