Anyone else use bug reporting frequency as a gross indicator
of system stability?

On Mon, 4 Sep 2023 06:30:44 -0700, Don Y <blockedofcourse@foo.invalid>
wrote:

>Anyone else use bug reporting frequency as a gross indicator
>of system stability?

One bug is an engineering failure and should be fixed immediately.

On Monday, September 4, 2023 at 11:30:55 PM UTC+10, Don Y wrote:
>
> Anyone else use bug reporting frequency as a gross indicator
> of system stability?

IBM claimed to be doing that some thirty years ago. They also claimed to have used their debugging system on Bill Gates MS/DOS software and made it much more reliable.

It must have been truly appalling when Bill Gates originally offered it to them for the IBM PC.

--
Bill Sloman, Sydney

On 04/09/2023 14:30, Don Y wrote:
> Anyone else use bug reporting frequency as a gross indicator
> of system stability?

Just about everyone who runs a beta test program.
MTBF is another metric that can be used for something that is intended
to run 24/7 and recover gracefully from anything that may happen to it.

It is inevitable that a new release will have some bugs and minor
differences from its predecessor that real life users will find PDQ.

The trick is to gain enough information from each in service failure to
identify and fix the root cause bug in a single iteration and without
breaking something else. Modern optimisers make that more difficult now
than it used to be back when I was involved in commercial development.

--
Martin Brown

On Tue, 5 Sep 2023 13:13:51 +0100, Martin Brown
<'''newspam'''@nonad.co.uk> wrote:

>On 04/09/2023 14:30, Don Y wrote:
>> Anyone else use bug reporting frequency as a gross indicator
>> of system stability?
>
>Just about everyone who runs a beta test program.
>MTBF is another metric that can be used for something that is intended
>to run 24/7 and recover gracefully from anything that may happen to it.
>
>It is inevitable that a new release will have some bugs and minor
>differences from its predecessor that real life users will find PDQ.

That's the story of software: bugs are inevitable, so why bother to be
careful coding or testing? You can always wait for bug reports from
users and post regular fixes of the worst ones.

>
>The trick is to gain enough information from each in service failure to
>identify and fix the root cause bug in a single iteration and without
>breaking something else. Modern optimisers make that more difficult now
>than it used to be back when I was involved in commercial development.

There have been various drives to write reliable code, but none were
popular. Quite the contrary, the software world loves abstraction and
ever new, bizarre languages... namely playing games instead of coding
boring, reliable applications in some klunky, reliable language.

Electronic design, and FPGA coding, are intended to be bug-free first
pass and often are, when done right.

FPGAs are halfway software, so the coders tend to be less careful than
hardware designers. FPGA bug fixes are easy, so why bother to read
your own code?

That's ironic, when you think about it. The hardest bits, the physical
electronics, has the least bugs.

On Tue, 05 Sep 2023 08:57:22 -0700, John Larkin
<jlarkin@highlandSNIPMEtechnology.com> wrote:

>On Tue, 5 Sep 2023 13:13:51 +0100, Martin Brown
><'''newspam'''@nonad.co.uk> wrote:
>
>>On 04/09/2023 14:30, Don Y wrote:
>>> Anyone else use bug reporting frequency as a gross indicator
>>> of system stability?
>>
>>Just about everyone who runs a beta test program.
>>MTBF is another metric that can be used for something that is intended
>>to run 24/7 and recover gracefully from anything that may happen to it.
>>
>>It is inevitable that a new release will have some bugs and minor
>>differences from its predecessor that real life users will find PDQ.
>
>That's the story of software: bugs are inevitable, so why bother to be
>careful coding or testing? You can always wait for bug reports from
>users and post regular fixes of the worst ones.
>
>>
>>The trick is to gain enough information from each in service failure to
>>identify and fix the root cause bug in a single iteration and without
>>breaking something else. Modern optimisers make that more difficult now
>>than it used to be back when I was involved in commercial development.
>
>There have been various drives to write reliable code, but none were
>popular. Quite the contrary, the software world loves abstraction and
>ever new, bizarre languages... namely playing games instead of coding
>boring, reliable applications in some klunky, reliable language.
>
>Electronic design, and FPGA coding, are intended to be bug-free first
>pass and often are, when done right.
>
>FPGAs are halfway software, so the coders tend to be less careful than
>hardware designers. FPGA bug fixes are easy, so why bother to read
>your own code?
>
>That's ironic, when you think about it. The hardest bits, the physical
>electronics, has the least bugs.

There is a complication. Modern software is tens of millions of lines
of code, far exceeding the inspection capabilities of humans. Hardware
is far simpler in terms of lines of FPGA code. But it's creeping up.

On a project some decades ago, the customer wanted us to verify every
path through the code, which was about 100,000 lines (large at the
time) of C or assembler (don't recall, doesn't actually matter).

In round numbers, one in five lines of code is an IF statement, so in
100,000 lines of code there will be 20,000 IF statements. So, there
are up to 2^20000 unique paths through the code. Which chokes my HP
calculator, so we must resort to logarithms, yielding 10^6021, which
is a *very* large number. The age of the Universe is only 14 billion
years, call it 10^10 years, so one would never be able to test even a
tiny fraction of the possible paths.

The customer withdrew the requirement.

Joe Gwinn

On 05/09/2023 16:57, John Larkin wrote:
> On Tue, 5 Sep 2023 13:13:51 +0100, Martin Brown
> <'''newspam'''@nonad.co.uk> wrote:
>
>> On 04/09/2023 14:30, Don Y wrote:
>>> Anyone else use bug reporting frequency as a gross indicator
>>> of system stability?
>>
>> Just about everyone who runs a beta test program.
>> MTBF is another metric that can be used for something that is intended
>> to run 24/7 and recover gracefully from anything that may happen to it.
>>
>> It is inevitable that a new release will have some bugs and minor
>> differences from its predecessor that real life users will find PDQ.
>
> That's the story of software: bugs are inevitable, so why bother to be
> careful coding or testing? You can always wait for bug reports from
> users and post regular fixes of the worst ones.

Don't blame the engineers for that - it is the ship it and be damned
senior management that is responsible for most buggy code being shipped.
Even more so now that 1+GB upgrades are essentially free. :(

First to market is worth enough that people live with buggy code. The
worst major release I can recall in a very long time was MS Excel 2007
(although bugs in Vista took a lot more flack - rather unfairly IMHO).

(which reminds me it is a MS patch Tuesday today)

>> The trick is to gain enough information from each in service failure to
>> identify and fix the root cause bug in a single iteration and without
>> breaking something else. Modern optimisers make that more difficult now
>> than it used to be back when I was involved in commercial development.
>
> There have been various drives to write reliable code, but none were
> popular. Quite the contrary, the software world loves abstraction and
> ever new, bizarre languages... namely playing games instead of coding
> boring, reliable applications in some klunky, reliable language.

The only ones which actually could be truly relied upon used formal
mathematical proof techniques to ensure reliability. Very few
practitioners are able to do it properly and it is pretty much reserved
for ultra high reliability safety and mission critical code.

It could be all be done to that standard iff commercial developers and
their customers were prepared to pay for it. However, they want it now
and they keep changing their minds about what it is they actually want
so the goalposts are forever shifting around. That sort of functionality
creep is much less common in hardware.

UK's NATS system is supposedly 6 sigma coding but its misbehaviour on
Bank Holiday Monday peak travel time was somewhat disastrous. It seems
someone managed to input the halt and catch fire instruction and the
buffers ran out before they were able to fix it. There will be a
technical report out in due course - my guess is that they have reduced
overheads and no longer have some of the key people who understand its
internals. Malformed flight plan data should not have been able to kill
it stone dead - but apparently that is exactly what happened!

https://www.ft.com/content/9fe22207-5867-4c4f-972b-620cdab10790
(might be paywalled)

If so Google "UK air traffic control outage caused by unusual flight
plan data"

> Electronic design, and FPGA coding, are intended to be bug-free first
> pass and often are, when done right.

But using design and simulation *software* that you fail to acknowledge
is actually pretty good. If you had to do it with pencil and paper your
would be there forever.

> FPGAs are halfway software, so the coders tend to be less careful than
> hardware designers. FPGA bug fixes are easy, so why bother to read
> your own code?
>
> That's ironic, when you think about it. The hardest bits, the physical
> electronics, has the least bugs.

So do physical mechanical interlocks. I don't trust software or even
electronic interlocks to protect me compared to a damn great beam stop
and a padlock on it with the key in my pocket.

--
Martin Brown

On 05/09/2023 17:45, Joe Gwinn wrote:
> On Tue, 05 Sep 2023 08:57:22 -0700, John Larkin
> <jlarkin@highlandSNIPMEtechnology.com> wrote:
>
>> On Tue, 5 Sep 2023 13:13:51 +0100, Martin Brown
>> <'''newspam'''@nonad.co.uk> wrote:
>>
>>> On 04/09/2023 14:30, Don Y wrote:
>>>> Anyone else use bug reporting frequency as a gross indicator
>>>> of system stability?
>>>
>>> Just about everyone who runs a beta test program.
>>> MTBF is another metric that can be used for something that is intended
>>> to run 24/7 and recover gracefully from anything that may happen to it.
>>>
>>> It is inevitable that a new release will have some bugs and minor
>>> differences from its predecessor that real life users will find PDQ.
>>
>> That's the story of software: bugs are inevitable, so why bother to be
>> careful coding or testing? You can always wait for bug reports from
>> users and post regular fixes of the worst ones.
>>
>>>
>>> The trick is to gain enough information from each in service failure to
>>> identify and fix the root cause bug in a single iteration and without
>>> breaking something else. Modern optimisers make that more difficult now
>>> than it used to be back when I was involved in commercial development.
>>
>> There have been various drives to write reliable code, but none were
>> popular. Quite the contrary, the software world loves abstraction and
>> ever new, bizarre languages... namely playing games instead of coding
>> boring, reliable applications in some klunky, reliable language.
>>
>> Electronic design, and FPGA coding, are intended to be bug-free first
>> pass and often are, when done right.
>>
>> FPGAs are halfway software, so the coders tend to be less careful than
>> hardware designers. FPGA bug fixes are easy, so why bother to read
>> your own code?
>>
>> That's ironic, when you think about it. The hardest bits, the physical
>> electronics, has the least bugs.
>
> There is a complication. Modern software is tens of millions of lines
> of code, far exceeding the inspection capabilities of humans. Hardware
> is far simpler in terms of lines of FPGA code. But it's creeping up.
>
> On a project some decades ago, the customer wanted us to verify every
> path through the code, which was about 100,000 lines (large at the
> time) of C or assembler (don't recall, doesn't actually matter).
>
> In round numbers, one in five lines of code is an IF statement, so in
> 100,000 lines of code there will be 20,000 IF statements. So, there
> are up to 2^20000 unique paths through the code. Which chokes my HP

Although that is true it is also true that a small number of cunningly
constructed test datasets can explore a very high proportion of the most
frequently traversed paths in any given codebase. One snag is that
testing is invariably cut short by management when development overruns.

The bits that fail to get explored tend to be weird error recovery
routines. I recall one latent on the VAX for ages which was that when it
ran out of IO handles (because someone was opening them inside a loop)
the first thing the recovery routine tried to do was open an IO channel!

> calculator, so we must resort to logarithms, yielding 10^6021, which
> is a *very* large number. The age of the Universe is only 14 billion
> years, call it 10^10 years, so one would never be able to test even a
> tiny fraction of the possible paths.

McCabe's complexity metric provides a way to test paths in components
and subsystems reasonably thoroughly and catch most of the common
programmer errors. Static dataflow analysis is also a lot better now
than in the past.

Then you only need at most 40000 test vectors to take each branch of
every binary if statement (60000 if it is Fortran with 3 way branches
all used). That is a rather more tractable number (although still large).

Any routine with too high a CCI count is practically certain to contain
latent bugs - which makes it worth looking at more carefully.

--
Martin Brown

On 9/5/2023 5:13 AM, Martin Brown wrote:
> On 04/09/2023 14:30, Don Y wrote:
>> Anyone else use bug reporting frequency as a gross indicator
>> of system stability?
>
> Just about everyone who runs a beta test program. > MTBF is another metric that can be used for something that is intended to run
> 24/7 and recover gracefully from anything that may happen to it.

I'm looking at the pre-release period (you wouldn't want to release
something that wasn't "stable").

I commit often (dozens of times a day) so I can have a record of
each problem encountered and, thereafter, how it was "fixed".
As the number of messages related to fixups decreases, confidence
in the codebase rises.

> It is inevitable that a new release will have some bugs and minor differences
> from its predecessor that real life users will find PDQ.

The "bugs" that tend to show up after release are specification
shortcomings. E.g., I had a case where a guy wired a motor
incorrectly and the software just kept driving it further and further
from it's desired setpoint -- until it smashed into the "wrong"
limit switches (which, of course, weren't examined because it
wasn't SUPPOSED to be traveling in that direction).

When you've got 7-figures at stake, you can't resort to blaming
the "electrician" for the failure ("Why didn't the software
sense that it was running the wrong way?" Um, why didn't it sense
that the electrician's wife had been ragging on him before he
came to work and left him in a distracted mood??)

Bugs (as in "coding errors") should never leave the lab.

> The trick is to gain enough information from each in service failure to
> identify and fix the root cause bug in a single iteration and without breaking
> something else. Modern  optimisers make that more difficult now than it used to
> be back when I was involved in commercial development.

Good problem decomposition goes a long way towards that goal.
If you try to do "too much" you quickly overwhelm the developer's
ability to manage complexity (7 items in STM?). And, as you can't
*see* the entire implementation, there's nothing to REMIND you
of some salient issue that might impact your local efforts.

[Hence the value of eschewing globals and the languages that
tolerate/encourage them! This dramatically cuts down the
number of ways X can influence Y.]

On Tue, 05 Sep 2023 12:45:01 -0400, Joe Gwinn <joegwinn@comcast.net>
wrote:

>On Tue, 05 Sep 2023 08:57:22 -0700, John Larkin
><jlarkin@highlandSNIPMEtechnology.com> wrote:
>
>>On Tue, 5 Sep 2023 13:13:51 +0100, Martin Brown
>><'''newspam'''@nonad.co.uk> wrote:
>>
>>>On 04/09/2023 14:30, Don Y wrote:
>>>> Anyone else use bug reporting frequency as a gross indicator
>>>> of system stability?
>>>
>>>Just about everyone who runs a beta test program.
>>>MTBF is another metric that can be used for something that is intended
>>>to run 24/7 and recover gracefully from anything that may happen to it.
>>>
>>>It is inevitable that a new release will have some bugs and minor
>>>differences from its predecessor that real life users will find PDQ.
>>
>>That's the story of software: bugs are inevitable, so why bother to be
>>careful coding or testing? You can always wait for bug reports from
>>users and post regular fixes of the worst ones.
>>
>>>
>>>The trick is to gain enough information from each in service failure to
>>>identify and fix the root cause bug in a single iteration and without
>>>breaking something else. Modern optimisers make that more difficult now
>>>than it used to be back when I was involved in commercial development.
>>
>>There have been various drives to write reliable code, but none were
>>popular. Quite the contrary, the software world loves abstraction and
>>ever new, bizarre languages... namely playing games instead of coding
>>boring, reliable applications in some klunky, reliable language.
>>
>>Electronic design, and FPGA coding, are intended to be bug-free first
>>pass and often are, when done right.
>>
>>FPGAs are halfway software, so the coders tend to be less careful than
>>hardware designers. FPGA bug fixes are easy, so why bother to read
>>your own code?
>>
>>That's ironic, when you think about it. The hardest bits, the physical
>>electronics, has the least bugs.
>
>There is a complication. Modern software is tens of millions of lines
>of code, far exceeding the inspection capabilities of humans.

After you type a line of code, read it. When we did that, entire
applications often worked first try.

Hardware
>is far simpler in terms of lines of FPGA code. But it's creeping up.

FPGAs are at least (usually) organized state machines. Mistakes are
typically hard failures, not low-rate bugs discovered in the field.
Avoiding race and metastability hazards is common practise.

>
>On a project some decades ago, the customer wanted us to verify every
>path through the code, which was about 100,000 lines (large at the
>time) of C or assembler (don't recall, doesn't actually matter).

Software provability was a brief fad once. It wasn't popular or, as
code is now done, possible.

>
>In round numbers, one in five lines of code is an IF statement, so in
>100,000 lines of code there will be 20,000 IF statements. So, there
>are up to 2^20000 unique paths through the code. Which chokes my HP
>calculator, so we must resort to logarithms, yielding 10^6021, which
>is a *very* large number. The age of the Universe is only 14 billion
>years, call it 10^10 years, so one would never be able to test even a
>tiny fraction of the possible paths.

An FPGA is usually coded as a state machine, where the designer
understands that the machine has a finite number of states and handles
every one. A computer program has an impossibly large number of
states, unknown and certainly not managed. Code is like hairball async
logic design.

>
>The customer withdrew the requirement.

It was naiive of him to want correct code.

>
>Joe Gwinn

On Tue, 5 Sep 2023 17:47:41 +0100, Martin Brown
<'''newspam'''@nonad.co.uk> wrote:

>On 05/09/2023 16:57, John Larkin wrote:
>> On Tue, 5 Sep 2023 13:13:51 +0100, Martin Brown
>> <'''newspam'''@nonad.co.uk> wrote:
>>
>>> On 04/09/2023 14:30, Don Y wrote:
>>>> Anyone else use bug reporting frequency as a gross indicator
>>>> of system stability?
>>>
>>> Just about everyone who runs a beta test program.
>>> MTBF is another metric that can be used for something that is intended
>>> to run 24/7 and recover gracefully from anything that may happen to it.
>>>
>>> It is inevitable that a new release will have some bugs and minor
>>> differences from its predecessor that real life users will find PDQ.
>>
>> That's the story of software: bugs are inevitable, so why bother to be
>> careful coding or testing? You can always wait for bug reports from
>> users and post regular fixes of the worst ones.
>
>Don't blame the engineers for that - it is the ship it and be damned
>senior management that is responsible for most buggy code being shipped.
>Even more so now that 1+GB upgrades are essentially free. :(
>
>First to market is worth enough that people live with buggy code. The
>worst major release I can recall in a very long time was MS Excel 2007
>(although bugs in Vista took a lot more flack - rather unfairly IMHO).
>
>(which reminds me it is a MS patch Tuesday today)
>
>>> The trick is to gain enough information from each in service failure to
>>> identify and fix the root cause bug in a single iteration and without
>>> breaking something else. Modern optimisers make that more difficult now
>>> than it used to be back when I was involved in commercial development.
>>
>> There have been various drives to write reliable code, but none were
>> popular. Quite the contrary, the software world loves abstraction and
>> ever new, bizarre languages... namely playing games instead of coding
>> boring, reliable applications in some klunky, reliable language.
>
>The only ones which actually could be truly relied upon used formal
>mathematical proof techniques to ensure reliability. Very few
>practitioners are able to do it properly and it is pretty much reserved
>for ultra high reliability safety and mission critical code.
>
>It could be all be done to that standard iff commercial developers and
>their customers were prepared to pay for it. However, they want it now
>and they keep changing their minds about what it is they actually want
>so the goalposts are forever shifting around. That sort of functionality
>creep is much less common in hardware.
>
>UK's NATS system is supposedly 6 sigma coding but its misbehaviour on
>Bank Holiday Monday peak travel time was somewhat disastrous. It seems
>someone managed to input the halt and catch fire instruction and the
>buffers ran out before they were able to fix it. There will be a
>technical report out in due course - my guess is that they have reduced
>overheads and no longer have some of the key people who understand its
>internals. Malformed flight plan data should not have been able to kill
>it stone dead - but apparently that is exactly what happened!
>
>https://www.ft.com/content/9fe22207-5867-4c4f-972b-620cdab10790
>(might be paywalled)
>
>If so Google "UK air traffic control outage caused by unusual flight
>plan data"
>
>> Electronic design, and FPGA coding, are intended to be bug-free first
>> pass and often are, when done right.
>
>But using design and simulation *software* that you fail to acknowledge
>is actually pretty good. If you had to do it with pencil and paper your
>would be there forever.

We did serious electronic design without simulation, and most of it
worked first time, or had dumb mistake hard failures that were easily
hacked. It didn't take forever. If one didn't understand some part or
circuit, it could be breadboarded and tested.

>
>> FPGAs are halfway software, so the coders tend to be less careful than
>> hardware designers. FPGA bug fixes are easy, so why bother to read
>> your own code?
>>
>> That's ironic, when you think about it. The hardest bits, the physical
>> electronics, has the least bugs.
>
>So do physical mechanical interlocks. I don't trust software or even
>electronic interlocks to protect me compared to a damn great beam stop
>and a padlock on it with the key in my pocket.

On 9/5/2023 9:47 AM, Martin Brown wrote:
> Don't blame the engineers for that - it is the ship it and be damned senior
> management that is responsible for most buggy code being shipped. Even more so
> now that 1+GB upgrades are essentially free. :(

Note how the latest coding styles inherently acknowledge that.
Agile? How-to-write-code-without-knowing-what-it-has-to-do?

> First to market is worth enough that people live with buggy code. The worst

Of course! Anyone think their Windows/Linux box is bug-free?
USENET client? Browser? yet, somehow, they all seem to provide
real value to their users!

> major release I can recall in a very long time was MS Excel 2007 (although bugs
> in Vista took a lot more flack - rather unfairly IMHO).

Of course. Folks run Linux with 20M+ LoC? So, a ballpark estimate
of 20K+ *bugs* in the RELEASED product??

<https://en.wikipedia.org/wiki/Linux_kernel#/media/File:Linux_kernel_map.png>

The era of monolithic kernels is over. Unless folks keep wanting
to DONATE their time to maintaining them.

<https://en.wikipedia.org/wiki/Linux_kernel#/media/File:Redevelopment_costs_of_Linux_kernel.png>

Amusing that it's pursuing a 50 year old dream... (let's get together
an effort to recreate the Wright flyer so we can all take 100 yard flights!)

> (which reminds me it is a MS patch Tuesday today)

Surrender your internet connection, for the day...

> The only ones which actually could be truly relied upon used formal
> mathematical proof techniques to ensure reliability. Very few practitioners are
> able to do it properly and it is pretty much reserved for ultra high
> reliability safety and mission critical code.

And only applies to the smallest parts of the codebase. The "engineering"
comes in figuring out how to live with systems that aren't verifiable.
(you can't ensure hardware WILL work as advertised unless you have tested
every component that you put into the fabrication -- ah, but you can blame
someone else for YOUR system's failure)

> It could be all be done to that standard iff commercial developers and their
> customers were prepared to pay for it. However, they want it now and they keep
> changing their minds about what it is they actually want so the goalposts are
> forever shifting around. That sort of functionality creep is much less common
> in hardware.

Exactly. And, software often is told to COMPENSATE for hardware shortcomings.

One of the sound systems used in early video games used a CVSD as an ARB.
But, the idiot who designed the hardware was 200% clueless about how the
software would use the hardware. So, the (dedicated!) processor had
to sit in a tight loop SHIFTING bits into the CVSD. Of course, each
path through the loop had to be balanced in terms of execution time
lest you get a beat component (as every 8th bit requires a new byte
to be fetched -- which takes a different amount of time than shifting
the current byte by one bit).

Hardware designers are typically clueless as to how their decisions
impact the software. And, as the company may have invested a "couple
of kilobucks" on a design and layout, Manglement's shortsightedness
fails to realize the tens of kilobucks that their penny-pinching
will cost!

[I once had a spectacular FAIL in a bit of hardware that I designed.
It was a custom CPU ("chip"). The guy writing the code (and the
tools to write it!) assumed addresses were byte-oriented. But,
the processor was truly a 16b machine and all of the addresses
were for 16b objects. So, all of the addresses generated by his tools
were exactly twice what they should have been ("Didn't you notice
how the LSb was ALWAYS '0'?") Simple fix but embarassing as we each
relied on assumptions that seemed natural to us where the wiser
approach would have made that statement explicit]

> UK's NATS system is supposedly 6 sigma coding but its misbehaviour on Bank
> Holiday Monday peak travel time was somewhat disastrous. It seems someone
> managed to input the halt and catch fire instruction and the buffers ran out
> before they were able to fix it. There will be a technical report out in due
> course - my guess is that they have reduced overheads and no longer have some
> of the key people who understand its internals. Malformed flight plan data
> should not have been able to kill it stone dead - but apparently that is
> exactly what happened!

Lunar landers, etc. Software is complex. Hardware is a walk in the
park. For anything but a trivial piece of code, you can't see all of the
interconnects/interdependencies.

> https://www.ft.com/content/9fe22207-5867-4c4f-972b-620cdab10790
> (might be paywalled)
>
> If so Google "UK air traffic control outage caused by unusual flight plan data"
>
>> Electronic design, and FPGA coding, are intended to be bug-free first
>> pass and often are, when done right.
>
> But using design and simulation *software* that you fail to acknowledge is
> actually pretty good. If you had to do it with pencil and paper your would be
> there forever.

When was the last time your calculator PROGRAM produced a verifiable error?
And, desktop software is considerably less complex than software used
in products where interactions arising from temporal differences can
prove unpredictable.

We bought a new stove/oven some time ago. Specify which oven, heat source,
setpoint temperature and time. START.

Ah, but if you want to change the time remaining (because you peeked
at the item and realize it could use another few minutes) AND the
timer expires WHILE YOU ARE TRYING TO CHANGE IT, the user interface
locks up (!). Your recourse is to shut off the oven (abort the
process) and then restart it using the settings you just CANCELED.

It's easy to see how this can evade testing -- if the test engineer
didn't have a good understanding of how the code worked so he
could challenge it with specially crafted test cases.

When drafting system specifications, I (try to) imagine every
situation that can come up and describe how each should be handled.
So, the test scaffolding and actual tests can be designed to verify
that behavior in the resulting product.

[How do you test for the case where the user tries to change the
remaining time AS the timer is expiring? How do you test for the
case where the process on the remote host crashes AFTER it has
received a request for service but before it has acknolwedged
that? Or, BEFORE it receives it? Or, WHILE acknowledging it?]

Hardware is easy to test: set voltage/current/freq/etc. and
observe result.

[We purchased a glass titty many years ago. At one point, we turned
it on, then off, then on again -- in relatively short order. I
guess the guy who designed the power supply hadn't considered this
possibility as the magic smoke rushed out of it! How hard can it
be to design a power supply???]

>> FPGAs are halfway software, so the coders tend to be less careful than
>> hardware designers. FPGA bug fixes are easy, so why bother to read
>> your own code?
>>
>> That's ironic, when you think about it. The hardest bits, the physical
>> electronics, has the least bugs.

No, the physical electronics are the EASIEST bits. If designing
hardware was so difficult, then the solution to the software
"problem" would be to just have all the hardware designers switch
over to designing software! Problem solved INSTANTLY!

In practice, the problem would be worsened by a few orders of
magnitude as they suddenly found themselves living in an opaque world.

> So do physical mechanical interlocks. I don't trust software or even electronic
> interlocks to protect me compared to a damn great beam stop and a padlock on it
> with the key in my pocket.

Note the miswired motor example, above. If the limit switches had
been hardwired, the problem still would have been present as the
problem was in the hardware -- the wiring of the motor.

On 9/5/2023 9:45 AM, Joe Gwinn wrote:
> There is a complication. Modern software is tens of millions of lines
> of code, far exceeding the inspection capabilities of humans. Hardware
> is far simpler in terms of lines of FPGA code. But it's creeping up.

Even small projects defy hardware implementations.

BUILD a speech synthesizer, entirely out of hardware.
Make sure there is a way the user can adjust the voice
their individual liking. (*you*, not your TEAM, have
3 months to produce a working prototype).

Or, something that recognizes faces, voices, etc.
Or, something that knows which plants should be watered,
today (if any), and how much water to dispense.
Or, something that examines the text in a document
and flags grammatical and spelling errors.
Or...

> On a project some decades ago, the customer wanted us to verify every
> path through the code, which was about 100,000 lines (large at the
> time) of C or assembler (don't recall, doesn't actually matter).
>
> In round numbers, one in five lines of code is an IF statement, so in
> 100,000 lines of code there will be 20,000 IF statements. So, there
> are up to 2^20000 unique paths through the code. Which chokes my HP
> calculator, so we must resort to logarithms, yielding 10^6021, which
> is a *very* large number. The age of the Universe is only 14 billion
> years, call it 10^10 years, so one would never be able to test even a
> tiny fraction of the possible paths.

The *first* problem is codifying how the code should behave in
*each* of those test cases.

> The customer withdrew the requirement.

"Verify your sqrt() function produces correct answers over the
range of inputs"

On 9/5/2023 10:02 AM, Martin Brown wrote:
>> In round numbers, one in five lines of code is an IF statement, so in
>> 100,000 lines of code there will be 20,000 IF statements.  So, there
>> are up to 2^20000 unique paths through the code.  Which chokes my HP
>
> Although that is true it is also true that a small number of cunningly
> constructed test datasets can explore a very high proportion of the most
> frequently traversed paths in any given codebase. One snag is that testing is
> invariably cut short by management when development overruns.

"We'll fix it in version 2"

I always found this an amusing delusion.

If the product is successful, there will be lots of people clamoring
for fixes so you won't have any manpower to devote to designing
version 2 (but your competitors will see the appeal your product
has and will start designing THEIR replacement for it!)

If the product is a dud (possibly because of these problems),
there won't be a need for a version 2.

> The bits that fail to get explored tend to be weird error recovery routines. I

Because, by design, they are seldom encountered.
So, don't benefit from being exercised in the normal
course of operation.

> recall one latent on the VAX for ages which was that when it ran out of IO
> handles (because someone was opening them inside a loop) the first thing the
> recovery routine tried to do was open an IO channel!
>
>> calculator, so we must resort to logarithms, yielding 10^6021, which
>> is a *very* large number.  The age of the Universe is only 14 billion
>> years, call it 10^10 years, so one would never be able to test even a
>> tiny fraction of the possible paths.
>
> McCabe's complexity metric provides a way to test paths in components and
> subsystems reasonably thoroughly and catch most of the common programmer
> errors. Static dataflow analysis is also a lot better now than in the past.

But some test cases can mask other paths through the code.
There is no guarantee that a given piece of code *can* be
thoroughly tested -- especially if you take into account the
fact that the underlying hardware isn't infallible;
"if (x % )" can yield one result, now, and a different
result, 5 lines later -- even though x hasn't been
altered (but the hardware farted).

So:

if (x % 2) {
do this;
do that;
do another_thing;
} else {
do that;
}

can execute differently than:

if (x % 2) {
do this;
}

do that;

if (x % 2) {
do another_thing;
}

Years ago, this possibility wasn't ever considered.

[Yes, optimizers can twiddle this but the point remains]

And, that doesn't begin to address hostile actors in a
system!

> Then you only need at most 40000 test vectors to take each branch of every
> binary if statement (60000 if it is Fortran with 3 way branches all used). That
> is a rather more tractable number (although still large).
>
> Any routine with too high a CCI count is practically certain to contain latent
> bugs - which makes it worth looking at more carefully.

"A 'program' should fit on a single piece of paper"

On Tue, 5 Sep 2023 10:44:08 -0700, Don Y <blockedofcourse@foo.invalid>
wrote:

>On 9/5/2023 9:47 AM, Martin Brown wrote:
>> Don't blame the engineers for that - it is the ship it and be damned senior
>> management that is responsible for most buggy code being shipped. Even more so
>> now that 1+GB upgrades are essentially free. :(
>
>Note how the latest coding styles inherently acknowledge that.
>Agile? How-to-write-code-without-knowing-what-it-has-to-do?
>
>> First to market is worth enough that people live with buggy code. The worst
>
>Of course! Anyone think their Windows/Linux box is bug-free?
>USENET client? Browser? yet, somehow, they all seem to provide
>real value to their users!
>
>> major release I can recall in a very long time was MS Excel 2007 (although bugs
>> in Vista took a lot more flack - rather unfairly IMHO).
>
>Of course. Folks run Linux with 20M+ LoC? So, a ballpark estimate
>of 20K+ *bugs* in the RELEASED product??
>
><https://en.wikipedia.org/wiki/Linux_kernel#/media/File:Linux_kernel_map.png>
>
>The era of monolithic kernels is over. Unless folks keep wanting
>to DONATE their time to maintaining them.
>
><https://en.wikipedia.org/wiki/Linux_kernel#/media/File:Redevelopment_costs_of_Linux_kernel.png>
>
>Amusing that it's pursuing a 50 year old dream... (let's get together
>an effort to recreate the Wright flyer so we can all take 100 yard flights!)
>
>> (which reminds me it is a MS patch Tuesday today)
>
>Surrender your internet connection, for the day...
>
>> The only ones which actually could be truly relied upon used formal
>> mathematical proof techniques to ensure reliability. Very few practitioners are
>> able to do it properly and it is pretty much reserved for ultra high
>> reliability safety and mission critical code.
>
>And only applies to the smallest parts of the codebase. The "engineering"
>comes in figuring out how to live with systems that aren't verifiable.
>(you can't ensure hardware WILL work as advertised unless you have tested
>every component that you put into the fabrication -- ah, but you can blame
>someone else for YOUR system's failure)
>
>> It could be all be done to that standard iff commercial developers and their
>> customers were prepared to pay for it. However, they want it now and they keep
>> changing their minds about what it is they actually want so the goalposts are
>> forever shifting around. That sort of functionality creep is much less common
>> in hardware.
>
>Exactly. And, software often is told to COMPENSATE for hardware shortcomings.
>
>One of the sound systems used in early video games used a CVSD as an ARB.
>But, the idiot who designed the hardware was 200% clueless about how the
>software would use the hardware. So, the (dedicated!) processor had
>to sit in a tight loop SHIFTING bits into the CVSD. Of course, each
>path through the loop had to be balanced in terms of execution time
>lest you get a beat component (as every 8th bit requires a new byte
>to be fetched -- which takes a different amount of time than shifting
>the current byte by one bit).
>
>Hardware designers are typically clueless as to how their decisions
>impact the software. And, as the company may have invested a "couple
>of kilobucks" on a design and layout, Manglement's shortsightedness
>fails to realize the tens of kilobucks that their penny-pinching
>will cost!
>
>[I once had a spectacular FAIL in a bit of hardware that I designed.
>It was a custom CPU ("chip"). The guy writing the code (and the
>tools to write it!) assumed addresses were byte-oriented. But,
>the processor was truly a 16b machine and all of the addresses
>were for 16b objects. So, all of the addresses generated by his tools
>were exactly twice what they should have been ("Didn't you notice
>how the LSb was ALWAYS '0'?") Simple fix but embarassing as we each
>relied on assumptions that seemed natural to us where the wiser
>approach would have made that statement explicit]
>
>> UK's NATS system is supposedly 6 sigma coding but its misbehaviour on Bank
>> Holiday Monday peak travel time was somewhat disastrous. It seems someone
>> managed to input the halt and catch fire instruction and the buffers ran out
>> before they were able to fix it. There will be a technical report out in due
>> course - my guess is that they have reduced overheads and no longer have some
>> of the key people who understand its internals. Malformed flight plan data
>> should not have been able to kill it stone dead - but apparently that is
>> exactly what happened!
>
>Lunar landers, etc. Software is complex. Hardware is a walk in the
>park. For anything but a trivial piece of code, you can't see all of the
>interconnects/interdependencies.
>
>> https://www.ft.com/content/9fe22207-5867-4c4f-972b-620cdab10790
>> (might be paywalled)
>>
>> If so Google "UK air traffic control outage caused by unusual flight plan data"
>>
>>> Electronic design, and FPGA coding, are intended to be bug-free first
>>> pass and often are, when done right.
>>
>> But using design and simulation *software* that you fail to acknowledge is
>> actually pretty good. If you had to do it with pencil and paper your would be
>> there forever.
>
>When was the last time your calculator PROGRAM produced a verifiable error?
>And, desktop software is considerably less complex than software used
>in products where interactions arising from temporal differences can
>prove unpredictable.
>
>We bought a new stove/oven some time ago. Specify which oven, heat source,
>setpoint temperature and time. START.
>
>Ah, but if you want to change the time remaining (because you peeked
>at the item and realize it could use another few minutes) AND the
>timer expires WHILE YOU ARE TRYING TO CHANGE IT, the user interface
>locks up (!). Your recourse is to shut off the oven (abort the
>process) and then restart it using the settings you just CANCELED.
>
>It's easy to see how this can evade testing -- if the test engineer
>didn't have a good understanding of how the code worked so he
>could challenge it with specially crafted test cases.
>
>When drafting system specifications, I (try to) imagine every
>situation that can come up and describe how each should be handled.
>So, the test scaffolding and actual tests can be designed to verify
>that behavior in the resulting product.
>
>[How do you test for the case where the user tries to change the
>remaining time AS the timer is expiring? How do you test for the
>case where the process on the remote host crashes AFTER it has
>received a request for service but before it has acknolwedged
>that? Or, BEFORE it receives it? Or, WHILE acknowledging it?]
>
>Hardware is easy to test: set voltage/current/freq/etc. and
>observe result.
>
>[We purchased a glass titty many years ago. At one point, we turned
>it on, then off, then on again -- in relatively short order. I
>guess the guy who designed the power supply hadn't considered this
>possibility as the magic smoke rushed out of it! How hard can it
>be to design a power supply???]
>
>>> FPGAs are halfway software, so the coders tend to be less careful than
>>> hardware designers. FPGA bug fixes are easy, so why bother to read
>>> your own code?
>>>
>>> That's ironic, when you think about it. The hardest bits, the physical
>>> electronics, has the least bugs.
>
>No, the physical electronics are the EASIEST bits. If designing
>hardware was so difficult, then the solution to the software
>"problem" would be to just have all the hardware designers switch
>over to designing software! Problem solved INSTANTLY!

The state of software development is a disgrace. We are plagued with
absurd user interfaces, hidden states, and massive numbers of bugs.

There is no science, math, or discipline to programming. What famous
person said that "anybody can learn to code"? One study fould that
English majors, on average, were better programmers than CE or CS
majors.

>
>In practice, the problem would be worsened by a few orders of
>magnitude as they suddenly found themselves living in an opaque world.
>
>> So do physical mechanical interlocks. I don't trust software or even electronic
>> interlocks to protect me compared to a damn great beam stop and a padlock on it
>> with the key in my pocket.
>
>Note the miswired motor example, above. If the limit switches had
>been hardwired, the problem still would have been present as the
>problem was in the hardware -- the wiring of the motor.

On 9/5/2023 10:03 AM, Don Y wrote:
> Good problem decomposition goes a long way towards that goal.
> If you try to do "too much" you quickly overwhelm the developer's
> ability to manage complexity (7 items in STM?).  And, as you can't
> *see* the entire implementation, there's nothing to REMIND you
> of some salient issue that might impact your local efforts.
>
> [Hence the value of eschewing globals and the languages that
> tolerate/encourage them!  This dramatically cuts down the
> number of ways X can influence Y.]

Of course, if you've never had any formal training ("you're
just a coder"), then you don't even realize these hazards exist!
You just pick at your code until it SEEMS to work and then
walk away.

Hence the need for the "managed environments" and
languages du jour that try to compensate for the
lack of formal training in schools and businesses.

[I worked with a Fortune *100* company on a 30 man project
where The Boss assigned the software for the product to a
*technician* whose sole qualification was that he
had a CoCo at home! Really? You're putting your good
name in the hands of a tinkerer??]

Sadly, most businessmen don't understand software or the
process and, rather than admit their ignorance, blunder
onward wondering (later) why everything turns to shite.
Anyone whose had to explain why a "little change" in
the product specification requires a major change to
the schedule understands the "ignorance at the top".

[I had a manager who wrote BASIC programs to keep track of the
DOG SHOWS that he'd entered (what is that? just a bunch
of PRINT statements??) and considered himself qualified to
make decisions regarding the software in the products for
which he was responsible. *Anyone* can write code.]

And, engineers turned managers tend to be the worst as they
THINK they understand the current state of the art (because
they used to practice it) without realizing that it's a moving
target and if you're using last year's technology, you are 2 or
3 (!) years out of date!

Would you promote a *technician* to run an electronics DESIGN
department and expect him to be current wrt the latest
generation of components, design and manufacturing practices?
If he *thought* he was, how quickly would you disabuse him of
that belief?

On Tue, 5 Sep 2023 18:02:05 +0100, Martin Brown
<'''newspam'''@nonad.co.uk> wrote:

>On 05/09/2023 17:45, Joe Gwinn wrote:
>> On Tue, 05 Sep 2023 08:57:22 -0700, John Larkin
>> <jlarkin@highlandSNIPMEtechnology.com> wrote:
>>
>>> On Tue, 5 Sep 2023 13:13:51 +0100, Martin Brown
>>> <'''newspam'''@nonad.co.uk> wrote:
>>>
>>>> On 04/09/2023 14:30, Don Y wrote:
>>>>> Anyone else use bug reporting frequency as a gross indicator
>>>>> of system stability?
>>>>
>>>> Just about everyone who runs a beta test program.
>>>> MTBF is another metric that can be used for something that is intended
>>>> to run 24/7 and recover gracefully from anything that may happen to it.
>>>>
>>>> It is inevitable that a new release will have some bugs and minor
>>>> differences from its predecessor that real life users will find PDQ.
>>>
>>> That's the story of software: bugs are inevitable, so why bother to be
>>> careful coding or testing? You can always wait for bug reports from
>>> users and post regular fixes of the worst ones.
>>>
>>>>
>>>> The trick is to gain enough information from each in service failure to
>>>> identify and fix the root cause bug in a single iteration and without
>>>> breaking something else. Modern optimisers make that more difficult now
>>>> than it used to be back when I was involved in commercial development.
>>>
>>> There have been various drives to write reliable code, but none were
>>> popular. Quite the contrary, the software world loves abstraction and
>>> ever new, bizarre languages... namely playing games instead of coding
>>> boring, reliable applications in some klunky, reliable language.
>>>
>>> Electronic design, and FPGA coding, are intended to be bug-free first
>>> pass and often are, when done right.
>>>
>>> FPGAs are halfway software, so the coders tend to be less careful than
>>> hardware designers. FPGA bug fixes are easy, so why bother to read
>>> your own code?
>>>
>>> That's ironic, when you think about it. The hardest bits, the physical
>>> electronics, has the least bugs.
>>
>> There is a complication. Modern software is tens of millions of lines
>> of code, far exceeding the inspection capabilities of humans. Hardware
>> is far simpler in terms of lines of FPGA code. But it's creeping up.
>>
>> On a project some decades ago, the customer wanted us to verify every
>> path through the code, which was about 100,000 lines (large at the
>> time) of C or assembler (don't recall, doesn't actually matter).
>>
>> In round numbers, one in five lines of code is an IF statement, so in
>> 100,000 lines of code there will be 20,000 IF statements. So, there
>> are up to 2^20000 unique paths through the code. Which chokes my HP
>
>Although that is true it is also true that a small number of cunningly
>constructed test datasets can explore a very high proportion of the most
>frequently traversed paths in any given codebase. One snag is that
>testing is invariably cut short by management when development overruns.
>
>The bits that fail to get explored tend to be weird error recovery
>routines. I recall one latent on the VAX for ages which was that when it
>ran out of IO handles (because someone was opening them inside a loop)
>the first thing the recovery routine tried to do was open an IO channel!
>
>> calculator, so we must resort to logarithms, yielding 10^6021, which
>> is a *very* large number. The age of the Universe is only 14 billion
>> years, call it 10^10 years, so one would never be able to test even a
>> tiny fraction of the possible paths.
>
>McCabe's complexity metric provides a way to test paths in components
>and subsystems reasonably thoroughly and catch most of the common
>programmer errors. Static dataflow analysis is also a lot better now
>than in the past.
>
>Then you only need at most 40000 test vectors to take each branch of
>every binary if statement (60000 if it is Fortran with 3 way branches
>all used). That is a rather more tractable number (although still large).
>
>Any routine with too high a CCI count is practically certain to contain
>latent bugs - which makes it worth looking at more carefully.

I must say that I fail to see how this can overcome 10^6021 paths,
even if it is wondrously effective, reducing the space to be tested by
a trillion to one (10^-12) - only 10^6009 paths to explore.

Joe Gwinn

On Tue, 05 Sep 2023 10:19:15 -0700, John Larkin
<jlarkin@highlandSNIPMEtechnology.com> wrote:

>On Tue, 05 Sep 2023 12:45:01 -0400, Joe Gwinn <joegwinn@comcast.net>
>wrote:
>
>>On Tue, 05 Sep 2023 08:57:22 -0700, John Larkin
>><jlarkin@highlandSNIPMEtechnology.com> wrote:
>>
>>>On Tue, 5 Sep 2023 13:13:51 +0100, Martin Brown
>>><'''newspam'''@nonad.co.uk> wrote:
>>>
>>>>On 04/09/2023 14:30, Don Y wrote:
>>>>> Anyone else use bug reporting frequency as a gross indicator
>>>>> of system stability?
>>>>
>>>>Just about everyone who runs a beta test program.
>>>>MTBF is another metric that can be used for something that is intended
>>>>to run 24/7 and recover gracefully from anything that may happen to it.
>>>>
>>>>It is inevitable that a new release will have some bugs and minor
>>>>differences from its predecessor that real life users will find PDQ.
>>>
>>>That's the story of software: bugs are inevitable, so why bother to be
>>>careful coding or testing? You can always wait for bug reports from
>>>users and post regular fixes of the worst ones.
>>>
>>>>
>>>>The trick is to gain enough information from each in service failure to
>>>>identify and fix the root cause bug in a single iteration and without
>>>>breaking something else. Modern optimisers make that more difficult now
>>>>than it used to be back when I was involved in commercial development.
>>>
>>>There have been various drives to write reliable code, but none were
>>>popular. Quite the contrary, the software world loves abstraction and
>>>ever new, bizarre languages... namely playing games instead of coding
>>>boring, reliable applications in some klunky, reliable language.
>>>
>>>Electronic design, and FPGA coding, are intended to be bug-free first
>>>pass and often are, when done right.
>>>
>>>FPGAs are halfway software, so the coders tend to be less careful than
>>>hardware designers. FPGA bug fixes are easy, so why bother to read
>>>your own code?
>>>
>>>That's ironic, when you think about it. The hardest bits, the physical
>>>electronics, has the least bugs.
>>
>>There is a complication. Modern software is tens of millions of lines
>>of code, far exceeding the inspection capabilities of humans.
>
>After you type a line of code, read it. When we did that, entire
>applications often worked first try.
>
>Hardware
>>is far simpler in terms of lines of FPGA code. But it's creeping up.
>
>FPGAs are at least (usually) organized state machines. Mistakes are
>typically hard failures, not low-rate bugs discovered in the field.
>Avoiding race and metastability hazards is common practise.
>
>>
>>On a project some decades ago, the customer wanted us to verify every
>>path through the code, which was about 100,000 lines (large at the
>>time) of C or assembler (don't recall, doesn't actually matter).
>
>Software provability was a brief fad once. It wasn't popular or, as
>code is now done, possible.
>
>
>>
>>In round numbers, one in five lines of code is an IF statement, so in
>>100,000 lines of code there will be 20,000 IF statements. So, there
>>are up to 2^20000 unique paths through the code. Which chokes my HP
>>calculator, so we must resort to logarithms, yielding 10^6021, which
>>is a *very* large number. The age of the Universe is only 14 billion
>>years, call it 10^10 years, so one would never be able to test even a
>>tiny fraction of the possible paths.
>
>An FPGA is usually coded as a state machine, where the designer
>understands that the machine has a finite number of states and handles
>every one. A computer program has an impossibly large number of
>states, unknown and certainly not managed. Code is like hairball async
>logic design.

In recent FPGAs you have done, how many states and events (their
Cartesian product being the entire state table) are there?

By the way, back in the day when I was specifying state machines
(often for implementation in software), I had a rule that all cells
would have an entry, even the combinations of state and event that
"couldn't happen". This was essential for achieving robustness in
practice.

>>The customer withdrew the requirement.
>
>It was naive of him to want correct code.

No, only a bit unrealistic.

But it was naive of him to think that total correctness can be tested
into anything.

The state of the art in verifying safety-critical code (as in for
safety of flight) is DO-178, which is an immensely heavy process. The
original objective was a probability of error not exceeding 10^-6,
this has been tightened to 10^-7 or 10^-8 because of the "headline
risk".

..<https://en.wikipedia.org/wiki/DO-178C>

Correctness can be mathematically proven only for extremely simple
mechanisms, using a sharply restricted set of allowed operations. See
The Halting Problem.

..<https://en.wikipedia.org/wiki/Halting_problem#:~:text=The%20halting%20problem%20is%20undecidable,usually%20via%20a%20Turing%20machine.>

Joe Gwinn

On Tue, 05 Sep 2023 18:33:47 -0400, Joe Gwinn <joegwinn@comcast.net>
wrote:

>On Tue, 05 Sep 2023 10:19:15 -0700, John Larkin
><jlarkin@highlandSNIPMEtechnology.com> wrote:
>
>>On Tue, 05 Sep 2023 12:45:01 -0400, Joe Gwinn <joegwinn@comcast.net>
>>wrote:
>>
>>>On Tue, 05 Sep 2023 08:57:22 -0700, John Larkin
>>><jlarkin@highlandSNIPMEtechnology.com> wrote:
>>>
>>>>On Tue, 5 Sep 2023 13:13:51 +0100, Martin Brown
>>>><'''newspam'''@nonad.co.uk> wrote:
>>>>
>>>>>On 04/09/2023 14:30, Don Y wrote:
>>>>>> Anyone else use bug reporting frequency as a gross indicator
>>>>>> of system stability?
>>>>>
>>>>>Just about everyone who runs a beta test program.
>>>>>MTBF is another metric that can be used for something that is intended
>>>>>to run 24/7 and recover gracefully from anything that may happen to it.
>>>>>
>>>>>It is inevitable that a new release will have some bugs and minor
>>>>>differences from its predecessor that real life users will find PDQ.
>>>>
>>>>That's the story of software: bugs are inevitable, so why bother to be
>>>>careful coding or testing? You can always wait for bug reports from
>>>>users and post regular fixes of the worst ones.
>>>>
>>>>>
>>>>>The trick is to gain enough information from each in service failure to
>>>>>identify and fix the root cause bug in a single iteration and without
>>>>>breaking something else. Modern optimisers make that more difficult now
>>>>>than it used to be back when I was involved in commercial development.
>>>>
>>>>There have been various drives to write reliable code, but none were
>>>>popular. Quite the contrary, the software world loves abstraction and
>>>>ever new, bizarre languages... namely playing games instead of coding
>>>>boring, reliable applications in some klunky, reliable language.
>>>>
>>>>Electronic design, and FPGA coding, are intended to be bug-free first
>>>>pass and often are, when done right.
>>>>
>>>>FPGAs are halfway software, so the coders tend to be less careful than
>>>>hardware designers. FPGA bug fixes are easy, so why bother to read
>>>>your own code?
>>>>
>>>>That's ironic, when you think about it. The hardest bits, the physical
>>>>electronics, has the least bugs.
>>>
>>>There is a complication. Modern software is tens of millions of lines
>>>of code, far exceeding the inspection capabilities of humans.
>>
>>After you type a line of code, read it. When we did that, entire
>>applications often worked first try.
>>
>>Hardware
>>>is far simpler in terms of lines of FPGA code. But it's creeping up.
>>
>>FPGAs are at least (usually) organized state machines. Mistakes are
>>typically hard failures, not low-rate bugs discovered in the field.
>>Avoiding race and metastability hazards is common practise.
>>
>>>
>>>On a project some decades ago, the customer wanted us to verify every
>>>path through the code, which was about 100,000 lines (large at the
>>>time) of C or assembler (don't recall, doesn't actually matter).
>>
>>Software provability was a brief fad once. It wasn't popular or, as
>>code is now done, possible.
>>
>>
>>>
>>>In round numbers, one in five lines of code is an IF statement, so in
>>>100,000 lines of code there will be 20,000 IF statements. So, there
>>>are up to 2^20000 unique paths through the code. Which chokes my HP
>>>calculator, so we must resort to logarithms, yielding 10^6021, which
>>>is a *very* large number. The age of the Universe is only 14 billion
>>>years, call it 10^10 years, so one would never be able to test even a
>>>tiny fraction of the possible paths.
>>
>>An FPGA is usually coded as a state machine, where the designer
>>understands that the machine has a finite number of states and handles
>>every one. A computer program has an impossibly large number of
>>states, unknown and certainly not managed. Code is like hairball async
>>logic design.
>
>In recent FPGAs you have done, how many states and events (their
>Cartesian product being the entire state table) are there?

A useful state machine might have 4 or maybe 16 states. I'm not sure
what you mean by 'events'. Sometimes we have a state word and a
counter, which technically gives us more states but it's convnient to
think of them separately. As in "repeat state 4 until the counter hits
zero."

A state machine can have many more inputs and outputs than it has
states. It is critical that no inputs can be changing when the clock
ticks.

On Tue, 05 Sep 2023 17:00:13 -0700, John Larkin
<jlarkin@highlandSNIPMEtechnology.com> wrote:

>On Tue, 05 Sep 2023 18:33:47 -0400, Joe Gwinn <joegwinn@comcast.net>
>wrote:
>
>>On Tue, 05 Sep 2023 10:19:15 -0700, John Larkin
>><jlarkin@highlandSNIPMEtechnology.com> wrote:
>>
>>>On Tue, 05 Sep 2023 12:45:01 -0400, Joe Gwinn <joegwinn@comcast.net>
>>>wrote:
>>>
>>>>On Tue, 05 Sep 2023 08:57:22 -0700, John Larkin
>>>><jlarkin@highlandSNIPMEtechnology.com> wrote:
>>>>
>>>>>On Tue, 5 Sep 2023 13:13:51 +0100, Martin Brown
>>>>><'''newspam'''@nonad.co.uk> wrote:
>>>>>
>>>>>>On 04/09/2023 14:30, Don Y wrote:
>>>>>>> Anyone else use bug reporting frequency as a gross indicator
>>>>>>> of system stability?
>>>>>>
>>>>>>Just about everyone who runs a beta test program.
>>>>>>MTBF is another metric that can be used for something that is intended
>>>>>>to run 24/7 and recover gracefully from anything that may happen to it.
>>>>>>
>>>>>>It is inevitable that a new release will have some bugs and minor
>>>>>>differences from its predecessor that real life users will find PDQ.
>>>>>
>>>>>That's the story of software: bugs are inevitable, so why bother to be
>>>>>careful coding or testing? You can always wait for bug reports from
>>>>>users and post regular fixes of the worst ones.
>>>>>
>>>>>>
>>>>>>The trick is to gain enough information from each in service failure to
>>>>>>identify and fix the root cause bug in a single iteration and without
>>>>>>breaking something else. Modern optimisers make that more difficult now
>>>>>>than it used to be back when I was involved in commercial development.
>>>>>
>>>>>There have been various drives to write reliable code, but none were
>>>>>popular. Quite the contrary, the software world loves abstraction and
>>>>>ever new, bizarre languages... namely playing games instead of coding
>>>>>boring, reliable applications in some klunky, reliable language.
>>>>>
>>>>>Electronic design, and FPGA coding, are intended to be bug-free first
>>>>>pass and often are, when done right.
>>>>>
>>>>>FPGAs are halfway software, so the coders tend to be less careful than
>>>>>hardware designers. FPGA bug fixes are easy, so why bother to read
>>>>>your own code?
>>>>>
>>>>>That's ironic, when you think about it. The hardest bits, the physical
>>>>>electronics, has the least bugs.
>>>>
>>>>There is a complication. Modern software is tens of millions of lines
>>>>of code, far exceeding the inspection capabilities of humans.
>>>
>>>After you type a line of code, read it. When we did that, entire
>>>applications often worked first try.
>>>
>>>Hardware
>>>>is far simpler in terms of lines of FPGA code. But it's creeping up.
>>>
>>>FPGAs are at least (usually) organized state machines. Mistakes are
>>>typically hard failures, not low-rate bugs discovered in the field.
>>>Avoiding race and metastability hazards is common practise.
>>>
>>>>
>>>>On a project some decades ago, the customer wanted us to verify every
>>>>path through the code, which was about 100,000 lines (large at the
>>>>time) of C or assembler (don't recall, doesn't actually matter).
>>>
>>>Software provability was a brief fad once. It wasn't popular or, as
>>>code is now done, possible.
>>>
>>>
>>>>
>>>>In round numbers, one in five lines of code is an IF statement, so in
>>>>100,000 lines of code there will be 20,000 IF statements. So, there
>>>>are up to 2^20000 unique paths through the code. Which chokes my HP
>>>>calculator, so we must resort to logarithms, yielding 10^6021, which
>>>>is a *very* large number. The age of the Universe is only 14 billion
>>>>years, call it 10^10 years, so one would never be able to test even a
>>>>tiny fraction of the possible paths.
>>>
>>>An FPGA is usually coded as a state machine, where the designer
>>>understands that the machine has a finite number of states and handles
>>>every one. A computer program has an impossibly large number of
>>>states, unknown and certainly not managed. Code is like hairball async
>>>logic design.
>>
>>In recent FPGAs you have done, how many states and events (their
>>Cartesian product being the entire state table) are there?
>
>
>A useful state machine might have 4 or maybe 16 states. I'm not sure
>what you mean by 'events'. Sometimes we have a state word and a
>counter, which technically gives us more states but it's convnient to
>think of them separately. As in "repeat state 4 until the counter hits
>zero."

We'll call it 16 states for the present purposes.

An event is anything that can cause the state to change, including
expiration of a timer. This is basically a design choice.

>A state machine can have many more inputs and outputs than it has
>states.

Yes, that's typical.

> It is critical that no inputs can be changing when the clock
>ticks.

That's also essential in hardware state machines.

In software state machines, events are most often the arrival of
messages, and the mechanism that provides these messages ensures that
they are presented in serial order (even if the underlying hardware
does not ensure ordering).

Joe Gwinn

On 9/5/2023 3:14 PM, Joe Gwinn wrote:
>> Then you only need at most 40000 test vectors to take each branch of
>> every binary if statement (60000 if it is Fortran with 3 way branches
>> all used). That is a rather more tractable number (although still large).
>>
>> Any routine with too high a CCI count is practically certain to contain
>> latent bugs - which makes it worth looking at more carefully.
>
> I must say that I fail to see how this can overcome 10^6021 paths,
> even if it is wondrously effective, reducing the space to be tested by
> a trillion to one (10^-12) - only 10^6009 paths to explore.

You don't have to exercise every path with a unique set of stimuli.
The number of N-way branches only sets an upper limit on the
number of paths through a piece of code. The actual number of
paths can be much less:

if (x == 1)
doXisone;

if (x == 2)
doXistwo;

if (x == 3)
doXisthree;

has three 2-way branches but only three distinct paths
through the code (instead of 2^3 = 8)

The point of complexity metrics is to alert you that maybe you
have factored the problem poorly.

Or, failed to recognize some underlying relationship(s)
that could simplify the process.

What's more amusing is how few orgainzations USE any of these
measures. And, how poorly they institute design controls on
software, in general!

[I had a colleague ask me to help one of his clients
because their codebase "suddenly" stopped working.
It was "suggested" that an employee who had been "made
redundant" may have sabotaged the codebase.

"Simple: retrieve the snapshot that would have been current
on the day before he was aware that he would be terminated.
Then, step forward until the day he was actually *gone*
and look for changes..."

But, no controls on who had access to the repository (nor
guarantees that the identity of the ACTOR was accurately stored
with each ACTION. The employee had gone back and cooked
the contents of the repository so a more detailed forensic
examination was required; you couldn't just checkout a
particular release and be assured it represented the
state of the codebase on the assumed day!

Can YOUR developers dick with the repository in unexpected ways?]

On 9/5/2023 3:33 PM, Joe Gwinn wrote:
> In recent FPGAs you have done, how many states and events (their
> Cartesian product being the entire state table) are there?

There is undoubtedly far more state *in* the CPU (neglecting
the application!) than in most FPGA designs!

> By the way, back in the day when I was specifying state machines
> (often for implementation in software), I had a rule that all cells
> would have an entry, even the combinations of state and event that
> "couldn't happen". This was essential for achieving robustness in
> practice.

It also helps if the machine can power up into a "random"
state (no "reset" to the state variables)... just let it run
for a few clocks until it finds its way to a known state.

> The state of the art in verifying safety-critical code (as in for
> safety of flight) is DO-178, which is an immensely heavy process. The
> original objective was a probability of error not exceeding 10^-6,
> this has been tightened to 10^-7 or 10^-8 because of the "headline
> risk".
>
> .<https://en.wikipedia.org/wiki/DO-178C>

But, this is still just a game of chance. And, with the number of
external things that can potentially impact the operation of the
state machine, even a provably correct implementation can fail
because the hardware isn't "ideal".

> Correctness can be mathematically proven only for extremely simple
> mechanisms, using a sharply restricted set of allowed operations. See

Exactly. Just like provably correct programs are extremely
simple.

And, if you try to *build* a (more complex) system atop that
proven (sub)system, you discover you're right back where you
started. (Else, if knowing the underlying system was
provably correct, you would rationalize that ANY program
is correct because the opcodes on which it relies are correctly
implemented!)

> The Halting Problem.
>
> .<https://en.wikipedia.org/wiki/Halting_problem#:~:text=The%20halting%20problem%20is%20undecidable,usually%20via%20a%20Turing%20machine.>

On 9/5/2023 5:11 PM, Joe Gwinn wrote:
> In software state machines, events are most often the arrival of
> messages, and the mechanism that provides these messages ensures that
> they are presented in serial order (even if the underlying hardware
> does not ensure ordering).

But software state machines can get EASILY creative in how they
process events. E.g., "go back from whence you came" (which isn't
present in most hardware machines)

A typical user interface can have hundreds of states, not counting
the information/controls they are "accumulating".

On 9/5/2023 6:04 PM, Don Y wrote:
> On 9/5/2023 3:14 PM, Joe Gwinn wrote:
>>> Then you only need at most 40000 test vectors to take each branch of
>>> every binary if statement (60000 if it is Fortran with 3 way branches
>>> all used). That is a rather more tractable number (although still large).
>>>
>>> Any routine with too high a CCI count is practically certain to contain
>>> latent bugs - which makes it worth looking at more carefully.
>>
>> I must say that I fail to see how this can overcome 10^6021 paths,
>> even if it is wondrously effective, reducing the space to be tested by
>> a trillion to one (10^-12) - only 10^6009 paths to explore.
>
> You don't have to exercise every path with a unique set of stimuli.
> The number of N-way branches only sets an upper limit on the
> number of paths through a piece of code.  The actual number of
> paths can be much less:
>
> if (x == 1)
>    doXisone;
>
> if (x == 2)
>    doXistwo;
>
> if (x == 3)
>    doXisthree;

actually, four

On 05/09/2023 23:14, Joe Gwinn wrote:
> On Tue, 5 Sep 2023 18:02:05 +0100, Martin Brown
> <'''newspam'''@nonad.co.uk> wrote:
>
>> On 05/09/2023 17:45, Joe Gwinn wrote:

>>> calculator, so we must resort to logarithms, yielding 10^6021, which
>>> is a *very* large number. The age of the Universe is only 14 billion
>>> years, call it 10^10 years, so one would never be able to test even a
>>> tiny fraction of the possible paths.
>>
>> McCabe's complexity metric provides a way to test paths in components
>> and subsystems reasonably thoroughly and catch most of the common
>> programmer errors. Static dataflow analysis is also a lot better now
>> than in the past.
>>
>> Then you only need at most 40000 test vectors to take each branch of
>> every binary if statement (60000 if it is Fortran with 3 way branches
>> all used). That is a rather more tractable number (although still large).
>>
>> Any routine with too high a CCI count is practically certain to contain
>> latent bugs - which makes it worth looking at more carefully.
>
> I must say that I fail to see how this can overcome 10^6021 paths,
> even if it is wondrously effective, reducing the space to be tested by
> a trillion to one (10^-12) - only 10^6009 paths to explore.

It is a divide and conquer strategy. It guarantees to execute every path
at least once and common core paths many times with different data. The
number of test vectors required scales as log2 of the number of paths.

That is an enormous reduction in the state space and makes coverage
testing possible. There are even some automatic tools that can help
create the test vectors now. Not a recommendation but just to make you
aware of some of the options for this sort of thing.

www.accelq.com/blog/test-coverage-techniques/

Here is a simple example for N = 5 (you wouldn't code it this way)

if (i<16)
{ if (i<8)
{
if (i<4)
{
if (i<2)
{
if (i<1) //zero
else // one
}
else
{
if (i<3) // two
else // three
}
else
{
}
}
else
{
}
}
else
{
}
} else
{ }

So that at each level i=0 ..N in the trivial example there are 2^N if
statements to select between numbers in the range 0 to 31 inclusive.
There is a deliberate error left in.

Counted your way there are 2^(N+1)-1 if statements and so 2^(2^(N+1))
distinct paths through the software (plus a few more with invalid data).

However integers -1 through 2^N will be sufficient to explore every path
at least once and test for high/low fence post errors.

Concrete example of N = 5
total if statements 63
naive paths through code 2^63 ~ 10^19
CCI test vectors to test every path 34

The example is topologically equivalent to real* code you merely have to
construct input data that will force execution down each binary choice
in turn at every level. Getting the absolute minimum number of test
vectors for full coverage is a much harder problem but a good enough
solution is possible in most practical cases.

--
Martin Brown

*real code can be grubby in reality with different depths of tree.
(I'd hope that few routines go deeper than 5 nested if statements)