On Friday, July 23, 2021 at 1:42:56 PM UTC-5, Anton Ertl wrote:
> Ivan Godard <iv...@millcomputing.com> writes:
> >Back when I was active on the IEEE committee, I once asked Kahan
> >whether, if quad (128-bit) were as fast as double, would he still have
> >denorms. He answered an unequivocal "No!".
> That is hard to believe, given that a major argument for denormal
> numbers is that
>
> a-b=0
>
> should give the same result as
>
> a=b
<
a==b
>
> - anton
> --
> 'Anton should try for "industrial quality" comp.arch responses.'
> Mitch Alsup, <c17fcd89-f024-40e7...@googlegroups.com>

On 7/23/2021 7:15 AM, Quadibloc wrote:
> On Friday, July 23, 2021 at 4:46:33 AM UTC-6, Marcus wrote:
>
>> I agree. I tried approaching the IEEE 754 working group [1] with a
>> suggestion to standardize a leaner subset of the current standard that
>> would better acknowledge the current reality (i.e. that many floating-
>> point implementations lack some of the mandatory IEEE 754 features),
>> and to analyze the effects of such implementations w.r.t. the numerical
>> and operational guarantees that the standard aims to provide. At least
>> to preempt an explosion of fragmented, non-conforming implementations.
>
>> To no avail...
>
> I can see why there would be issues.
>
> The characteristics of the "leaner subset" would depend *strongly* on
> the level of the implementation.
>
> Thus, for the kind of floating-point arithmetic unit I might prefer, the
> leaner subset would have these characteristics:
>
> - Denormals would work. What would not guarantee is that they
> would *fail* when they were supposed to.
>
> - Accurate rounding would be guaranteed for addition, subtraction,
> and multiplication, but *not* division, which would instead be accurate
> to something like 0.6 or 0.51 units in the last place.
>
> That's because I would want to implement division using one of the
> fast algorithms which require significant additional overhead to get
> accurate rounding...
>
> and I intend to support denormals with no loss in speed by converting
> floats to a format with no hidden bit inside registers, so it would be
> extra work to reproduce the exact numerical range and precision of
> the IEEE 754 floating-point format. Instead, the denormals - and some
> numeric range beyond them - would still have full precision, and only
> get rounded down when it came time to store values in memory.
>
> People implementing floating-point in a *different* manner would have
> entirely different choices for what parts of the IEEE 754 standard they
> would want omitted from a 'leaner subset'. An implementation with
> significantly less hardware would omit the entire denormal range, but
> support exact rounding for division, for example.
>
> So there's a 'lightweight' leaner subset, and there's a 'high-performance'
> leaner subset, at the very least. Which are disjoint rather than one being
> a subset of the other.
>

As mentioned various times, for example, my subset looks like:
No denormals, they are treated as zeroes and flushed to zero (1);
Hardware only does FADD/FSUB/FMUL, and optionally FMAC;
Rounding is only "probably" correct (2);
FDIV and FSQRT are done in software (cost reasons, mostly, 3);
No FP status flags or similar;
Rounding behavior is hard-wired
(typically either round-to-nearest or truncate);
...

I have basically tried to achieve an FPU that is "acceptably cheap" for
my uses.

1: Besides just their impact on FADD/FMUL, they have a fairly obvious
impact on the relative cost of format-conversion operators.

The dedicated FP conversion ops perform rounding at least, though the
converters used in SIMD ops may truncate instead (in which case, the
logic is mostly bit-repacking with a few special cases to deal with
out-of-range exponents, which causes the results to become either Inf or
Zero).

Denormals mostly make sense for getting more accuracy out of smaller
formats, but then again, if a person is using a smaller format, then
typically precision isn't that important in the first place.

It would almost make more sense IMO to standardize of denormal-as-zero
semantics.

2: There are only a small number of bits below the ULP in my case, and
the final rounding step has "limited carry propagation".

Note that the FADD/FSUB unit does implement twos complement operations
internally such that the limited carry propagation does not normally
effect the visible result of operations (unlike what would happen with
ones' complement).

Internally, the FADD/FSUB unit is using a mantissa big enough to manage
conversion to/from 64 bit integers, and which generally gives ~ 10 bits
below the ULP.

So, probably, FADD/FSUB falls within a 0.51 ULP window.

FMUL shaves it a little closer, but still falls within "most of the
time" territory.

If the program depends on bit-exact results, well, one is probably still
going to have problems. Hardly anything depends on this though.

In general, the implementation is correct enough that doing integer math
via the FPU works as expected (and there is some software that depends
on the assumption that doing integer math via FPU ops works correctly).

....

3: My initial attempts at hardware FDIV and FSQRT were "not free" and
were actually slower than what I could be managed in software. This is
due mostly to software being able to use a "more proper" Newton-Raphson,
which converges more quickly. This operation is still essentially serial
whether or not it was done in hardware or software, essentially limited
by the internal latency of the FADD and FMUL units.

The "starting guess" for these operations is easy enough to generate
using basic integer arithmetic.

I had partly experimented with a 96-bit format (binary128 with the low
32-bits ignored/set-to-zero), but this caused the FPU to be too
expensive to afford in a dual-core configuration with the FPGA I am using.

This FPU would also have given slightly higher precision (probability of
a correctly-rounded result) with double operations.

( Decided to leave out a big thing related to how the 96-bit FPU could
potentially also be used for 64-bit integer multiply and divide, then
going into more about integer operations and large-integer formats in
general. )

Though, FWIW, even without a large-format FPU, one can still implement
float128 semi-acceptably with partial hardware-support for 128-bit
integer operations (which in-turn are also semi-useful in implementing
operations on larger integer types, ...).

Note that even with hardware support for the 96-bit format, the idea was
that "__float128" would still use runtime calls for the full 128-bit
format, whereas "long double" would use the same representation but may
use the hardware-supported operations for a little extra speed.

If I can afford the bigger FPU with SMT (with the FPU as probably a
shared resource between the two threads), it might be worthwhile, but I
have my own concerns for the viability of SMT (some of the code I had
written for a 12R6W register file is, concerning...).

I have a few possible trick that could reduce SMT cost some, at the cost
of adding extra cycles (using interlocks pulling the results from a
snapshot of the pipeline outputs from the previous cycle, rather than
using forwarding of the current results, along most of the "lower
priority" paths). Where in this case, the main register-file module also
partly assumes control over the management of pipeline interlocks.

A more traditional VLIW might have avoided this issue by not using
register forwarding, which combined with no interlocks, would mean not
being able to use ALU results or similar until 2 cycles later (or 4
cycles if one needs to wait for writeback).

A few other parts of the core would likely require a fair bit of
restructuring (still deciding what I would do with "ExUnit", ...).

....

MitchAlsup <MitchAlsup@aol.com> schrieb:

> IEEE 754 was put together by people who really understood FP arithmetic
> but HW not so much.

Who was on that committee?

I have read that Intel and Motorola were there, but what about
Cray, CDC or IBM? They should have had some understanding of the
difficulties involved (but then again at least Cray was famous
for not caring about such niceties, and IBM was still caught in
their horrible radix-16 system at that time and probably could
not even dream that they would one day implement another floating
point format).

On Friday, July 23, 2021 at 2:22:13 PM UTC-5, Thomas Koenig wrote:
> MitchAlsup <Mitch...@aol.com> schrieb:
> > IEEE 754 was put together by people who really understood FP arithmetic
> > but HW not so much.
<
> Who was on that committee?
<
Ivan will be here shortly to list the perpetrators.
>
> I have read that Intel and Motorola were there, but what about
> Cray, CDC or IBM? They should have had some understanding of the
> difficulties involved (but then again at least Cray was famous
> for not caring about such niceties, and IBM was still caught in
> their horrible radix-16 system at that time and probably could
> not even dream that they would one day implement another floating
> point format).
<
IBM's was worse the simply radix 16, it was also a truncation system
with a guard digit (½ a byte) in calculations. To make maters worse, its
competitors were 36-bit, 48-bit, and 60-bit (single precision).

On 7/23/2021 12:59 PM, MitchAlsup wrote:
> On Friday, July 23, 2021 at 12:43:25 PM UTC-5, Quadibloc wrote:
>> On Friday, July 23, 2021 at 9:30:20 AM UTC-6, MitchAlsup wrote:
>>
>>> IEEE 754 was put together by people who really understood FP arithmetic
>>> but HW not so much.
> <
>> My own very limited understanding of hardware was put to use
>> in my reply to Marcus' comment about the need for a "leaner
>> subset" of IEEE 754; I noted that if one were designing a large
>> system aiming at the fastest possible speed, one would want
>> to leave _certain_ features of IEEE 754 out, whereas if one were
>> designing a really tiny system, one would want to leave *different*
>> features out... which was probably too much to ask, thus
>> skewering any hopes of adding such a subset to the standard.
>>
>> Since IEEE 754, in its original form, simply ratified what Intel
>> had planned to include in its forthcoming (but then secret)
>> 8087, though, a knowledge of hardware certainly was...
>> applied... to its contents, even if the committee that signed
>> off on it did not include hardware experts.
> <
> IEEE 754 had to compromise between x87 and 881 the intel
> and Moto chips both nearing tapeout, one maybe as far along
> as having seen silicon. This is the part about when rounding versus
> normalization transpires.
>>
>> And what I see, then, as the issue would be...
>>
>> The fact that IEEE 754 derived from Intel's 8087 work meant that
>> the standard was capable of being implemented in hardware once.
> <
> Remember that an FADD was 84-240 cycles..........but I digress......

Bleh...

Granted, I don't expect doing it in software on an 8086 would have been
exactly all that much faster.

I guess a baseline requirement for "semi-fast" software floating point
is having hardware-supported integer operations which are comparable in
size to the floating-point type in question (if not greater).

Also having enough register space that the values can be held in
registers, which the 8086 didn't exactly have going for it either.

....

Though, kind of funny that Binary16 was a recent development; presumably
it would have been fairly useful in that era.

>>
>> Of course, that meant that it was implementable many times in
>> other systems *of comparable size*.
>>
>> But denormals were a pain in very small systems... and the
>> requirement for perfect rounding was a pain in larger systems
>> that aimed at the highest possible speed using either Goldschmidt
>> or Newton-Raphson for division.
> <
> Having watched this from inside:
> a) HW designers know a lot more about this today than in 1980
> b) even systems that started out as IEEE-format gradually went
> closer and closer to full IEEE-compliant (GPUs) until there is no
> useful difference in the quality of the arithmetic.
> c) once 754-2009 came out the overhead to do denorms went to
> zero, and there is no reason to avoid full speed denorms in practice.
> (BGB's small FPGA prototyping environment aside.)
> d) HW designers have learned how to perform all of the rounding
> modes at no overhead compared to RNE.

Probably.

I suspect the boards and FPGAs I am using are fairly common in the
hobbyist space.

But, yeah, I had started working on a dedicated FMAC unit at one point,
but partly shelved this effort for now, as my initial "viability tests"
didn't look promising (it would have been fairly expensive), and the
design would have had a higher latency for FADD and FMUL than my current
FPU.

>>
>> So that's where I see that having HW knowleldge on the IEEE
>> 754 committee would have helped. With your much greater
>> knowledge of hardware, you may see other things which are
>> more important than what I thought of.
>>
>> John Savard

In article <fc5a33d0-7c17-4855-8ab3-162884bd6b7bn@googlegroups.com>,
jsavard@ecn.ab.ca (Quadibloc) wrote:

> Since IEEE 754, in its original form, simply ratified what Intel
> had planned to include in its forthcoming (but then secret)
> 8087, though, a knowledge of hardware certainly was...
> applied... to its contents, even if the committee that signed
> off on it did not include hardware experts.

At that stage, Intel do not seem to have understood very well how to make
/fast/ hardware. Remember the iAPX 432?

The obvious design goal of the 8087 was very compact code, leading to the
floating-point register stack, and its bad effects on IPC. Its arithmetic
works fine, but it seems to have been intended for assembly language
programming, rather than compiled languages.

John

On 7/23/2021 12:45 PM, Quadibloc wrote:
> On Friday, July 23, 2021 at 9:30:20 AM UTC-6, MitchAlsup wrote:
>
>> No, doubling the widths just hides the problem and makes the actual
>> problem harder to find. You can take the point of view that that is
>> enough, but it is not in reality.
>
> It's enough to get one Moon lander on the surface. But solving the
> real problem is still better - but it's nice if there's a way to meet
> deadlines and the like.
>

I would expect for a physical system, the inherent "noisiness" of
physical reality would matter a lot more for the outcome a lot more than
any bit-exact arithmetic would help make the landing.

Some expertly crafted and computed flight plan could be thrown off by
much larger factors like which direction the wind was blowing during
liftoff, solar wind or flars blowing the spacecraft slightly off course,
collisions with space dust, ...

Presumably these missions have some sort of closed-loop control, such as
course-correction, ability to detect the distance from the target, ...

Granted, this is assuming it is using sufficient numerical precision.
Trying to do long range navigation using binary32 or similar is probably
just asking for it to crash into the surface (or miss the moon entirely,
say because jitter in the math miscalculated the location of the moon by
like 150km or something, ...).

On Friday, July 23, 2021 at 4:38:02 PM UTC-5, BGB wrote:
> On 7/23/2021 12:45 PM, Quadibloc wrote:
> > On Friday, July 23, 2021 at 9:30:20 AM UTC-6, MitchAlsup wrote:
> >
> >> No, doubling the widths just hides the problem and makes the actual
> >> problem harder to find. You can take the point of view that that is
> >> enough, but it is not in reality.
> >
> > It's enough to get one Moon lander on the surface. But solving the
> > real problem is still better - but it's nice if there's a way to meet
> > deadlines and the like.
> >
> I would expect for a physical system, the inherent "noisiness" of
> physical reality would matter a lot more for the outcome a lot more than
> any bit-exact arithmetic would help make the landing.
>
> Some expertly crafted and computed flight plan could be thrown off by
> much larger factors like which direction the wind was blowing during
> liftoff, solar wind or flars blowing the spacecraft slightly off course,
> collisions with space dust, ...
>
> Presumably these missions have some sort of closed-loop control, such as
> course-correction, ability to detect the distance from the target, ...
<
Only if you intend "closed loop" to incorporate ground based radars and
using a on spacecraft sextant to measure some star "angles", and having
ground based computers figure out the adjustments and radio them up
to the spacecraft, punching them into the computer and causing a correction
burn; as "closed loop".
<
But this is an important point:: many feed back systems have enough
time between corrections that the accuracy of the FP numbers did not
have to be "all that great".
>
>
> Granted, this is assuming it is using sufficient numerical precision.
> Trying to do long range navigation using binary32 or similar is probably
> just asking for it to crash into the surface (or miss the moon entirely,
> say because jitter in the math miscalculated the location of the moon by
> like 150km or something, ...).
<
Remember we are still flying fighter jets with computer controls where the
jet will invert itself when crossing the equator--all because it too too many
instructions to get either SIN() or COS() right.

On Friday, July 23, 2021 at 1:13:48 PM UTC-6, MitchAlsup wrote:
> On Friday, July 23, 2021 at 1:42:56 PM UTC-5, Anton Ertl wrote:

> > should give the same result as
> >
> > a=b
> <
> a==b

That's only a correction if you assume all pseudocode must be in C.

On my web page, I have defined a higher-level language where a
multiple assignment statement doesn't look like

a,b,c=5

or

a=b=c=5

but instead

. a/b/c=5

and there is a reason for that. Or maybe more than one reason:

1) The "." is a short form of the keyword "LET". By beginning all statements
with a keyword, the need for reserved words in the language can be completely
avoided, without complicated gyrations as done in FORTRAN.

2) A READ statement may look like this:

READ [5,10,END=99,ERR=999] X,Y,Z

therefore the syntax of the language must allow assignments, not just expressions,
to be passed as arguments to subroutines.

3) The following statement assigns _T (true) to L1 and L2 if I and J are equal,
and _F to L1 and L2 otherwise.

. L1/L2=I=J

Only the first equals sign in an assignment statement indicates an assignment.
Within the expression following that equals sign, any other equals signs are
equality operators.
Therefore, the equals sign can't be used as a separator for multiple assignments.
Neither can the comma, because then you're breaking the assignment into two
pieces; a READ statement may _also_ look like this:

READ [5,10,END/ERR=99] X,Y,Z

Essentially, the language is intended to look a lot like FORTRAN, but it
also borrows from AWK the idea of having a comma at the end of the text
on a line indicate a continuation (no semicolons after each statement).
Parentheses are used within expressions to control the order of evaluation,
for function argument lists, _and_ array subscripts; it's in the case of the
device, format clause in I/O statements that square brackets were needed
to avoid an ambiguity.

John Savard

On 7/23/2021 5:23 PM, MitchAlsup wrote:
> On Friday, July 23, 2021 at 4:38:02 PM UTC-5, BGB wrote:
>> On 7/23/2021 12:45 PM, Quadibloc wrote:
>>> On Friday, July 23, 2021 at 9:30:20 AM UTC-6, MitchAlsup wrote:
>>>
>>>> No, doubling the widths just hides the problem and makes the actual
>>>> problem harder to find. You can take the point of view that that is
>>>> enough, but it is not in reality.
>>>
>>> It's enough to get one Moon lander on the surface. But solving the
>>> real problem is still better - but it's nice if there's a way to meet
>>> deadlines and the like.
>>>
>> I would expect for a physical system, the inherent "noisiness" of
>> physical reality would matter a lot more for the outcome a lot more than
>> any bit-exact arithmetic would help make the landing.
>>
>> Some expertly crafted and computed flight plan could be thrown off by
>> much larger factors like which direction the wind was blowing during
>> liftoff, solar wind or flars blowing the spacecraft slightly off course,
>> collisions with space dust, ...
>>
>> Presumably these missions have some sort of closed-loop control, such as
>> course-correction, ability to detect the distance from the target, ...
> <
> Only if you intend "closed loop" to incorporate ground based radars and
> using a on spacecraft sextant to measure some star "angles", and having
> ground based computers figure out the adjustments and radio them up
> to the spacecraft, punching them into the computer and causing a correction
> burn; as "closed loop".
> <
> But this is an important point:: many feed back systems have enough
> time between corrections that the accuracy of the FP numbers did not
> have to be "all that great".

More or less, they would presumably have some way to determine if they
are on-course to the target, and some hydrazine rockets or similar to
allow for fine adjustment.

But, yeah, I meant as opposed to doing the space launch and then trying
to fly an entirely precomputed path and then assume that the spacecraft
gets to its destination if one did all the math correctly (and then have
it "all go south" because the wind changed direction on launch day
relative to what was calculated in the simulations; or because the
spacecraft was getting pushed by solar winds, ...).

>>
>>
>> Granted, this is assuming it is using sufficient numerical precision.
>> Trying to do long range navigation using binary32 or similar is probably
>> just asking for it to crash into the surface (or miss the moon entirely,
>> say because jitter in the math miscalculated the location of the moon by
>> like 150km or something, ...).
> <
> Remember we are still flying fighter jets with computer controls where the
> jet will invert itself when crossing the equator--all because it too too many
> instructions to get either SIN() or COS() right.
>

Hmm...

I think I had also remembered something like fighter jet computers
crashing when flying over parts of Jordan or similar, because they were
not meant to deal with the altitude going negative...

On 7/23/2021 5:50 PM, BGB wrote:
> On 7/23/2021 5:23 PM, MitchAlsup wrote:
>> On Friday, July 23, 2021 at 4:38:02 PM UTC-5, BGB wrote:
>>> On 7/23/2021 12:45 PM, Quadibloc wrote:
>>>> On Friday, July 23, 2021 at 9:30:20 AM UTC-6, MitchAlsup wrote:
>>>>
>>>>> No, doubling the widths just hides the problem and makes the actual
>>>>> problem harder to find. You can take the point of view that that is
>>>>> enough, but it is not in reality.
>>>>
>>>> It's enough to get one Moon lander on the surface. But solving the
>>>> real problem is still better - but it's nice if there's a way to meet
>>>> deadlines and the like.
>>>>
>>> I would expect for a physical system, the inherent "noisiness" of
>>> physical reality would matter a lot more for the outcome a lot more than
>>> any bit-exact arithmetic would help make the landing.
>>>
>>> Some expertly crafted and computed flight plan could be thrown off by
>>> much larger factors like which direction the wind was blowing during
>>> liftoff, solar wind or flars blowing the spacecraft slightly off course,
>>> collisions with space dust, ...
>>>
>>> Presumably these missions have some sort of closed-loop control, such as
>>> course-correction, ability to detect the distance from the target, ...
>> <
>> Only if you intend "closed loop" to incorporate ground based radars and
>> using a on spacecraft sextant to measure some star "angles", and having
>> ground based computers figure out the adjustments and radio them up
>> to the spacecraft, punching them into the computer and causing a
>> correction
>> burn; as "closed loop".
>> <
>> But this is an important point:: many feed back systems have enough
>> time between corrections that the accuracy of the FP numbers did not
>> have to be "all that great".
>
> More or less, they would presumably have some way to determine if they
> are on-course to the target, and some hydrazine rockets or similar to
> allow for fine adjustment.
>
>
> But, yeah, I meant as opposed to doing the space launch and then trying
> to fly an entirely precomputed path and then assume that the spacecraft
> gets to its destination if one did all the math correctly (and then have
> it "all go south" because the wind changed direction on launch day
> relative to what was calculated in the simulations; or because the
> spacecraft was getting pushed by solar winds, ...).
>
>
>>>
>>>
>>> Granted, this is assuming it is using sufficient numerical precision.
>>> Trying to do long range navigation using binary32 or similar is probably
>>> just asking for it to crash into the surface (or miss the moon entirely,
>>> say because jitter in the math miscalculated the location of the moon by
>>> like 150km or something, ...).
>> <
>> Remember we are still flying fighter jets with computer controls where
>> the
>> jet will invert itself when crossing the equator--all because it too
>> too many
>> instructions to get either SIN() or COS() right.
>>
>
> Hmm...
>
> I think I had also remembered something like fighter jet computers
> crashing when flying over parts of Jordan or similar, because they were
> not meant to deal with the altitude going negative...
>

Skinwalker ranch made a radar altimeter made my Garmin report around 40
feet to ground when they were several thousand of feet off the ground.
5000 feet would be 10000 feet from sea level.

On Friday, July 23, 2021 at 4:23:59 PM UTC-6, MitchAlsup wrote:

> But this is an important point:: many feed back systems have enough
> time between corrections that the accuracy of the FP numbers did not
> have to be "all that great".

I remember specifically that one particular failed Ariane launch was
held up as what bad floating-point can cause... so, of course, while there
are cases where feedback makes things less of an issue, there are other
cases where numerical accuracy is critical.

John Savard

On Saturday, July 24, 2021 at 12:01:35 AM UTC-6, Quadibloc wrote:

> I remember specifically that one particular failed Ariane launch was
> held up as what bad floating-point can cause...

Ah. A Google search led me to what really happened.

The maiden flight of Ariane 5 failed because conversion from
floating-point to 16-bit integer caused an exception, as the
floating-point valule was not within the range of such integers.

No doubt bad software design, but not really necessarily
about the kind of numerical analysis concern that IEEE 754
addresses.

John Savard

On 7/24/2021 1:05 AM, Quadibloc wrote:
> On Saturday, July 24, 2021 at 12:01:35 AM UTC-6, Quadibloc wrote:
>
>> I remember specifically that one particular failed Ariane launch was
>> held up as what bad floating-point can cause...
>
> Ah. A Google search led me to what really happened.
>
> The maiden flight of Ariane 5 failed because conversion from
> floating-point to 16-bit integer caused an exception, as the
> floating-point valule was not within the range of such integers.
>
> No doubt bad software design, but not really necessarily
> about the kind of numerical analysis concern that IEEE 754
> addresses.
>

IME:
Inexact rounding is hardly ever the source of problems (excluding
integer-exact math in codecs, but this sort of stuff is typically
fixed-point).

However, more serious bugs, like unexpected overflow, saturation, or
traps, these sorts of things can ruin ones' day.

Likewise for algorithms whose domain is "all real numbers except zero",
which just so happens to fail catastrophically and nuke the program if
the input just so happens to be zero.

Insufficient precision can also be a source of problems.

Issues with the use of single precision coordinates in games trying to
deal with large worlds is a fairly common example (typically a lot of
workarounds are needed to make all the math work for worlds much larger
than a few km or so).

> John Savard
>

BGB <cr88192@gmail.com> schrieb:
> On 7/23/2021 12:45 PM, Quadibloc wrote:
>> On Friday, July 23, 2021 at 9:30:20 AM UTC-6, MitchAlsup wrote:
>>
>>> No, doubling the widths just hides the problem and makes the actual
>>> problem harder to find. You can take the point of view that that is
>>> enough, but it is not in reality.
>>
>> It's enough to get one Moon lander on the surface. But solving the
>> real problem is still better - but it's nice if there's a way to meet
>> deadlines and the like.
>>
>
> I would expect for a physical system, the inherent "noisiness" of
> physical reality would matter a lot more for the outcome a lot more than
> any bit-exact arithmetic would help make the landing.
>
> Some expertly crafted and computed flight plan could be thrown off by
> much larger factors like which direction the wind was blowing during
> liftoff, solar wind or flars blowing the spacecraft slightly off course,
> collisions with space dust, ...

One major issue was gravitational anomalies of the Moon, which are
big enough to have a significant impact on the orbit of spacecraft.

> Presumably these missions have some sort of closed-loop control, such as
> course-correction, ability to detect the distance from the target, ...

Having a spacecraft travel around in space is an engieering problem.
You always have error bars on every measurement and assumption -
mass of your spacecraft, duration and efficiency of a burn, actual
mass expended on a burn, orientation, ...

Any of the above uncertainties is _much_ higher than the floating
point precision of a 32-bit real, let alone of a 64-bit real.

Of course, if you subtract two numbers of almost equal magnigude,
that could be much different...

This is why space missions have course corrections somehwere in
the middle of their trajectory. After sufficient time has elapsed
to gather data on the velocity and position of the spacecraft, but
not yet enough that a course correction would be too expensive,
you alter the spacecraft's velocity by a relatively small amount
of delta v.

> Granted, this is assuming it is using sufficient numerical precision.
> Trying to do long range navigation using binary32 or similar is probably
> just asking for it to crash into the surface (or miss the moon entirely,
> say because jitter in the math miscalculated the location of the moon by
> like 150km or something, ...).

A few years ago (pre-Corona) I visited the Space Center in Houston.
A tour guide for the Saturn-V on exhibition there told the group
that he had worked on the courses for Apollo, and that they had
actually mostly used analytical methods. I was a bit surprised
at that, I would have thought numerical solution of ODEs would
have been employed more.

MitchAlsup <MitchAlsup@aol.com> schrieb:

> Remember we are still flying fighter jets with computer controls where the
> jet will invert itself when crossing the equator--all because it too too many
> instructions to get either SIN() or COS() right.

From what I read, that bug was found and fixed in 1986.

MitchAlsup <MitchAlsup@aol.com> schrieb:
> On Friday, July 23, 2021 at 2:22:13 PM UTC-5, Thomas Koenig wrote:
>> MitchAlsup <Mitch...@aol.com> schrieb:
>> > IEEE 754 was put together by people who really understood FP arithmetic
>> > but HW not so much.
><
>> Who was on that committee?
><
> Ivan will be here shortly to list the perpetrators.
>>
>> I have read that Intel and Motorola were there, but what about
>> Cray, CDC or IBM? They should have had some understanding of the
>> difficulties involved (but then again at least Cray was famous
>> for not caring about such niceties, and IBM was still caught in
>> their horrible radix-16 system at that time and probably could
>> not even dream that they would one day implement another floating
>> point format).
><
> IBM's was worse the simply radix 16, it was also a truncation system
> with a guard digit (½ a byte) in calculations. To make maters worse, its
> competitors were 36-bit, 48-bit, and 60-bit (single precision).

At the time it was designed, yes. Even Henry S. Warren, who worked
for IBM, has a scathing page dedicated to that particular decision.

At the time IEEE was formulated, the 32-bit systems had pretty
much taken over, I think (of course Cray used 64 bit for single
precision, they had no 32-bit format).

Thomas Koenig <tkoenig@netcologne.de> writes:
>MitchAlsup <MitchAlsup@aol.com> schrieb:
>
>> IEEE 754 was put together by people who really understood FP arithmetic
>> but HW not so much.
>
>Who was on that committee?

Obviously enough hardware manufacturers that IEEE 754 was very quickly
implemented in most new hardware (often with microcode or software
assist for denormal numbers).

>I have read that Intel and Motorola were there, but what about
>Cray, CDC or IBM?

IBM certainly adopted IEEE 754. CDC was on the way out by that time.

Concerning Cray: the followons (Cray Y-MP) to the old Cray designs
were of course compatible with the old designs, but Cray (the company)
also did the T3D and T3E, which used Alpha CPUs and thus IEEE 754.
But I think the question was more what Crays customers wanted. They
had earlier bought machines that put speed above correctness (IIRC for
division), so Cray had a reason to believe that their customers would
also choose speed over IEEE 754. However, Cray suffered difficult
times in the 1980s, so maybe there were too few customers who agreed
with that choice.

Anyway, given that we have fast IEEE 754 hardware, the IEEE 754 people
either understood hardware enough or were lucky (I think it was the
former). Admittedly the dynamic rounding mode stuff is slow or
expensive to implement on OoO CPUs, and to a lesser degree on
pipelined CPUs, and who knows how well the people on the IEEE 754
committee understood that (pipelining was not widely used at the time,
and OoO even less), but I think that those who want dynamic rounding
mode changes prefer slowness to not having it at all, and those who
claim it is unnecessary't should not care if it is slow; and the
hardware cost of slow dynamic rounding modes is small.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>

jgd@cix.co.uk (John Dallman) writes:
>At that stage, Intel do not seem to have understood very well how to make
>/fast/ hardware. Remember the iAPX 432?

The iAPX 432 seems to be the overambitious project that growing
companies tend to engage in, many of which fail. IBM Stretch did not
achieve its performance goals, either, and was also considered a
failure.

By contrast, the 8087 seems to be very successful, and its
architecture even more so, surviving until this day (although with
competition from SSE2 since its introduction in 2000 and especially
since AMD64 in 2003, where SSE2 became the standard FP instruction set).

>The obvious design goal of the 8087 was very compact code, leading to the
>floating-point register stack, and its bad effects on IPC.

What bad effects on IPC do you mean? Earlier this year
<2021Jan8.194939@mips.complang.tuwien.ac.at>
<2021Jan9.075657@mips.complang.tuwien.ac.at> I measured daxpy
implemented in C:

void daxpy(double ra, double *f_x, double *f_y, long stride, unsigned long ucount)
{ for (; ucount>0; ucount--) {
*f_y += ra * *f_x;
f_x = (double *)(((char *)f_x)+stride);
f_y = (double *)(((char *)f_y)+stride);
}
}

and compiled with

gcc -O -mfpmath=387

and with

gcc -O -mfpmath=sse

The resulting code was:

387 SSE2
start: start:
fld %st(0) movapd %xmm0,%xmm1
fmull (%rdi) mulsd (%rdi),%xmm1
faddl (%rsi) addsd (%rsi),%xmm1
fstpl (%rsi) movsd %xmm1,(%rsi)
add %rdx,%rdi add %rdx,%rdi
add %rdx,%rsi add %rdx,%rsi
sub $0x1,%rcx sub $0x1,%rcx
jne start jne start

Both loops execute at 2 cycles/iteration (IPC=4) on a Skylake. I
don't see a bad effect in IPC from the stack architecture of the 387,
despite the 387 having taken a back seat since AMD64 became the
dominant instruction set on Intel CPUs.

>Its arithmetic
>works fine, but it seems to have been intended for assembly language
>programming, rather than compiled languages.

Compilers can generate code for stack machines just fine, actually it
is very easy to generate code for stack machines. Therefore, the
Burroughs B5500 architecture was designed as a stack architecture to
go with the Burroughs Algol compiler. The transputer (another stack
machine) was designed in tandem with the Occam language and compiler.
Both machines were not intended to be programmed much in assembly
language.

One case where register machines have an advantage is for partial
redundancy elimination (a generalization of common subexpression
elimination and loop-invariant code motion). I am positively
surprised by gcc keeping ra on the stack in the 387 example above.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>

On 2021-07-24, Anton Ertl <anton@mips.complang.tuwien.ac.at> wrote:
> jgd@cix.co.uk (John Dallman) writes:
>>At that stage, Intel do not seem to have understood very well how to make
>>/fast/ hardware. Remember the iAPX 432?
>
> The iAPX 432 seems to be the overambitious project that growing
> companies tend to engage in, many of which fail. IBM Stretch did not
> achieve its performance goals, either, and was also considered a
> failure.
>
> By contrast, the 8087 seems to be very successful, and its
> architecture even more so, surviving until this day (although with
> competition from SSE2 since its introduction in 2000 and especially
> since AMD64 in 2003, where SSE2 became the standard FP instruction set).
>
>>The obvious design goal of the 8087 was very compact code, leading to the
>>floating-point register stack, and its bad effects on IPC.
>
> What bad effects on IPC do you mean? Earlier this year
><2021Jan8.194939@mips.complang.tuwien.ac.at>
><2021Jan9.075657@mips.complang.tuwien.ac.at> I measured daxpy
> implemented in C:
>
> void daxpy(double ra, double *f_x, double *f_y, long stride, unsigned long ucount)
> {
> for (; ucount>0; ucount--) {
> *f_y += ra * *f_x;
> f_x = (double *)(((char *)f_x)+stride);
> f_y = (double *)(((char *)f_y)+stride);
> }
> }
>
> and compiled with
>
> gcc -O -mfpmath=387
>
> and with
>
> gcc -O -mfpmath=sse
>
> The resulting code was:
>
> 387 SSE2
> start: start:
> fld %st(0) movapd %xmm0,%xmm1
> fmull (%rdi) mulsd (%rdi),%xmm1
> faddl (%rsi) addsd (%rsi),%xmm1
> fstpl (%rsi) movsd %xmm1,(%rsi)
> add %rdx,%rdi add %rdx,%rdi
> add %rdx,%rsi add %rdx,%rsi
> sub $0x1,%rcx sub $0x1,%rcx
> jne start jne start
>
> Both loops execute at 2 cycles/iteration (IPC=4) on a Skylake. I
> don't see a bad effect in IPC from the stack architecture of the 387,
> despite the 387 having taken a back seat since AMD64 became the
> dominant instruction set on Intel CPUs.
>
>>Its arithmetic
>>works fine, but it seems to have been intended for assembly language
>>programming, rather than compiled languages.
>
> Compilers can generate code for stack machines just fine, actually it
> is very easy to generate code for stack machines. Therefore, the
> Burroughs B5500 architecture was designed as a stack architecture to
> go with the Burroughs Algol compiler. The transputer (another stack
> machine) was designed in tandem with the Occam language and compiler.
> Both machines were not intended to be programmed much in assembly
> language.
>
> One case where register machines have an advantage is for partial
> redundancy elimination (a generalization of common subexpression
> elimination and loop-invariant code motion). I am positively
> surprised by gcc keeping ra on the stack in the 387 example above.
>
> - anton
Could you provide test program?
I have tested : https://github.com/siposcsaba89/eigen_fast_math_test
on M1
-rw-r--r-- 1 bmaxa staff 9 Jul 24 12:25 .gitignore
-rw-r--r-- 1 bmaxa staff 14120 Jul 24 12:26 CMakeCache.txt
drwxr-xr-x 13 bmaxa staff 416 Jul 24 12:26 CMakeFiles
-rw-r--r-- 1 bmaxa staff 453 Jul 24 12:25 CMakeLists.txt
-rw-r--r-- 1 bmaxa staff 1069 Jul 24 12:25 LICENSE
-rw-r--r-- 1 bmaxa staff 5418 Jul 24 12:26 Makefile
-rw-r--r-- 1 bmaxa staff 56 Jul 24 12:25 README.md
-rw-r--r-- 1 bmaxa staff 1550 Jul 24 12:26 cmake_install.cmake
-rw-r--r-- 1 bmaxa staff 476 Jul 24 12:25 main.cpp
-rwxr-xr-x 1 bmaxa staff 238077 Jul 24 12:26 test_mat_inverse
bmaxa@Branimirs-Air eigen_fast_math_test % ./test_mat_inverse
0.999303 0.0145721 -0.0343709 -0.00753874
-0.0145243 0.999893 0.00164037 0.52384
0.0343912 -0.00114001 0.999408 -5.68314
0 0 0 1
0.999303 -0.0145243 0.0343911 0.210591
0.0145721 0.999893 -0.00114001 -0.530153
-0.034371 0.00164037 0.999408 5.67865
-0 0 -0 1
1 4.93871e-19 1.89553e-18 -1.47451e-17
2.99736e-19 1 -3.26763e-19 -1.11022e-16
-5.65987e-18 1.09284e-19 1 1.77636e-15
0 0 0 1
and
bmaxa@Branimirs-Air eigen_perf_test % ls -la
total 2496
drwxr-xr-x 11 bmaxa staff 352 Jul 24 12:30 .
drwxr-xr-x 55 bmaxa staff 1760 Jul 24 12:29 ..
drwxr-xr-x 12 bmaxa staff 384 Jul 24 12:29 .git
-rw-r--r-- 1 bmaxa staff 12 Jul 24 12:29 .gitignore
-rw-r--r-- 1 bmaxa staff 14045 Jul 24 12:30 CMakeCache.txt
drwxr-xr-x 13 bmaxa staff 416 Jul 24 12:30 CMakeFiles
-rw-r--r-- 1 bmaxa staff 511 Jul 24 12:29 CMakeLists.txt
-rw-r--r-- 1 bmaxa staff 5609 Jul 24 12:30 Makefile
-rw-r--r-- 1 bmaxa staff 1540 Jul 24 12:30 cmake_install.cmake
-rwxr-xr-x 1 bmaxa staff 1235420 Jul 24 12:30 eigen_perf_test
-rw-r--r-- 1 bmaxa staff 1279 Jul 24 12:29 eigen_perf_test.cpp
bmaxa@Branimirs-Air eigen_perf_test % ./eigen_perf_test
Result: -0.000151013 -0.000110068 7.73141e-05 -1.32445e-05 1.02876e-05 -8.45934e-05
compiler is Clang, version 12.0.5
10000 iteration took 0.153339 seconds

Result: -6.36653e-08 -1.11854e-05 -1.19188e-05 -1.15604e-05 9.72996e-07 -1.43802e-05
compiler is Clang, version 12.0.5
100000 iteration took 1.28075 seconds

Result: -1.06923e-07 6.38881e-07 -3.13472e-07 6.69333e-07 -1.99308e-07 -7.20017e-07
compiler is Clang, version 12.0.5
1000000 iteration took 12.8297 seconds
https://github.com/siposcsaba89/eigen_perf_test

--
bmaxa now listens Smile (explicit version) by Lily Allen from Triple J: Hottest 100, Volume 14

On 7/23/2021 1:13 PM, MitchAlsup wrote:
> On Friday, July 23, 2021 at 2:22:13 PM UTC-5, Thomas Koenig wrote:
>> MitchAlsup <Mitch...@aol.com> schrieb:
>>> IEEE 754 was put together by people who really understood FP arithmetic
>>> but HW not so much.
> <
>> Who was on that committee?
> <
> Ivan will be here shortly to list the perpetrators.

Can't say for the original 754 - I wasn't involved until the '00s

>>
>> I have read that Intel and Motorola were there, but what about
>> Cray, CDC or IBM? They should have had some understanding of the
>> difficulties involved (but then again at least Cray was famous
>> for not caring about such niceties, and IBM was still caught in
>> their horrible radix-16 system at that time and probably could
>> not even dream that they would one day implement another floating
>> point format).
> <
> IBM's was worse the simply radix 16, it was also a truncation system
> with a guard digit (½ a byte) in calculations. To make maters worse, its
> competitors were 36-bit, 48-bit, and 60-bit (single precision).
>

Branimir Maksimovic <branimir.maksimovic@gmail.com> writes:
>> Both loops execute at 2 cycles/iteration (IPC=4) on a Skylake. I
....
>Could you provide test program?

http://www.complang.tuwien.ac.at/anton/tmp/axpy.zip

Today I measure 1.8 cycles per iteration (IPC=4.44) from this program
(both compiled for 387 and SSE2) on a Skylake. Strange.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>

On 7/24/2021 3:36 AM, Thomas Koenig wrote:
> BGB <cr88192@gmail.com> schrieb:
>> On 7/23/2021 12:45 PM, Quadibloc wrote:
>>> On Friday, July 23, 2021 at 9:30:20 AM UTC-6, MitchAlsup wrote:
>>>
>>>> No, doubling the widths just hides the problem and makes the actual
>>>> problem harder to find. You can take the point of view that that is
>>>> enough, but it is not in reality.
>>>
>>> It's enough to get one Moon lander on the surface. But solving the
>>> real problem is still better - but it's nice if there's a way to meet
>>> deadlines and the like.
>>>
>>
>> I would expect for a physical system, the inherent "noisiness" of
>> physical reality would matter a lot more for the outcome a lot more than
>> any bit-exact arithmetic would help make the landing.
>>
>> Some expertly crafted and computed flight plan could be thrown off by
>> much larger factors like which direction the wind was blowing during
>> liftoff, solar wind or flars blowing the spacecraft slightly off course,
>> collisions with space dust, ...
>
> One major issue was gravitational anomalies of the Moon, which are
> big enough to have a significant impact on the orbit of spacecraft.
>

Makes sense, but yeah, more factors into the point that unpredictable
physical variables are a lot more likely to factor in to mission success
or failure than whether or not floating-point operations have exact
rounding.

In cases where it matters, it is usually more a case of wanting multiple
implementations to give results which are consistent, rather than
necessarily maximizing accuracy.

One could have a floating point definition which specifies the use of
truncation, and potentially how the results are truncated internally
during operations, rather than one which assumes an infinitely precise
result.

For example, what if we defined double-precision FADD/FSUB relative to
the behavior of a 64-bit twos complement integer mantissa?...

Or, FMUL relative to a triangular multiplier which multiplies two 56-bit
inputs and produces a 56-bit (truncated) output?...

Not necessarily as a "universal standard", but rather as a target which
can be:
Relatively cost-effectively be implemented bit-exact in hardware;
Can be mostly emulated in software mostly using 64-bit integer
operations (*1).

*1: Well, sorta; FMUL would need special treatment to be bit-exact. One
would need to make some extra effort to approximate the behavior of a
truncated multiplier (naively using a 64*64->128 bit widening multiplier
and discarding the low bits would not produce bit-exact results).

Bit-exact would require getting the same results for the bits "hanging
off the bottom", with such a multiplier probably being defined relative
to the behavior of a collection of 16*16->32 sub-multipliers or similar.

There are other options, such as approximating a "smooth bottom"
truncated result, but this is not likely to be cost-effective relative
to the more "jagged" version.

....

Worthwhile is also debatable, as an FPU built to be able to support,
say, an FP96 format, would not produce bit-identical results with one
built to only support Binary64/Double, without special case logic to
artificially truncate or discard bits in the intermediate results, ...

Granted, an "FP standard" whose definitions depend on the size of the
largest output format supported by the FPU in question seems "kinda
useless" on this front.

The "cheaper" alternative is to not make any requirements here, and give
rounding behavior in terms of a probability.

As can be noted though, integer multiply via FP generally survives such
a multiplier because inputs which produce an in-range result will have
zeroes in the low-order parts of the mantissa, and thus all the
sub-multiplies which would have "fallen off the bottom" would have
contained multiplies against zero (and thus not had a visible effect on
the result).

>> Presumably these missions have some sort of closed-loop control, such as
>> course-correction, ability to detect the distance from the target, ...
>
> Having a spacecraft travel around in space is an engieering problem.
> You always have error bars on every measurement and assumption -
> mass of your spacecraft, duration and efficiency of a burn, actual
> mass expended on a burn, orientation, ...
>
> Any of the above uncertainties is _much_ higher than the floating
> point precision of a 32-bit real, let alone of a 64-bit real.
>
> Of course, if you subtract two numbers of almost equal magnigude,
> that could be much different...
>
> This is why space missions have course corrections somehwere in
> the middle of their trajectory. After sufficient time has elapsed
> to gather data on the velocity and position of the spacecraft, but
> not yet enough that a course correction would be too expensive,
> you alter the spacecraft's velocity by a relatively small amount
> of delta v.
>

Makes sense.

>> Granted, this is assuming it is using sufficient numerical precision.
>> Trying to do long range navigation using binary32 or similar is probably
>> just asking for it to crash into the surface (or miss the moon entirely,
>> say because jitter in the math miscalculated the location of the moon by
>> like 150km or something, ...).
>
> A few years ago (pre-Corona) I visited the Space Center in Houston.
> A tour guide for the Saturn-V on exhibition there told the group
> that he had worked on the courses for Apollo, and that they had
> actually mostly used analytical methods. I was a bit surprised
> at that, I would have thought numerical solution of ODEs would
> have been employed more.
>

IME:

Most of what one might want to calculate in practice, can be done using
algebra.

If it can't be done directly, one can subdivide it into smaller
timesteps and do it incrementally. At small enough timesteps, pretty
much everything becomes linear.

Similarly, a lot of stuff one could do with ODE's could instead be done
using B-splines or similar, ...

Also, divide is one of those operations one generally wants to avoid
when possible, not just for speed reasons, but because it is more prone
to adding instability: if you divide by a number which happens to
approach 0, then the numbers involved can get huge. Calculations which
give wonky results, or spit out Inf, NaN, or raises a fault when given
certain inputs, are not desirable.

Usually better when possible to find an alternative which avoids the use
of division, or at least eliminate cases where divide-by-zero exists as
a possibility.

Though, this does seem to be one of those points of division between
doing math on a computer relative to traditional mathematics, which
likes to throw in divide operators all over the place.

On Saturday, July 24, 2021 at 1:03:35 PM UTC-5, BGB wrote:
> On 7/24/2021 3:36 AM, Thomas Koenig wrote:
> > BGB <cr8...@gmail.com> schrieb:
<snip>
>
> Also, divide is one of those operations one generally wants to avoid
> when possible, not just for speed reasons, but because it is more prone
> to adding instability: if you divide by a number which happens to
> approach 0, then the numbers involved can get huge. Calculations which
> give wonky results, or spit out Inf, NaN, or raises a fault when given
> certain inputs, are not desirable.
<
There were several cases in my transcendental studies where one could
write a Newton-Raphson iteration using SQRT(x) or 1/SQRT(x); and in every
case I looked at, the one using 1/SQRT(x) converged faster.
>
> Usually better when possible to find an alternative which avoids the use
> of division, or at least eliminate cases where divide-by-zero exists as
> a possibility.
>
> Though, this does seem to be one of those points of division between
> doing math on a computer relative to traditional mathematics, which
> likes to throw in divide operators all over the place.
<
In traditional math, DIV is "just another operator" with exactly the same
domain and range as any other operator (just like differentiate and
integrate are other operators along with DIV and CURL.) all being of
infinite precision and perfect arithmetic.
<
Computer arithmetics don't have those properties

BGB <cr88192@gmail.com> schrieb:

> IME:
>
> Most of what one might want to calculate in practice, can be done using
> algebra.
>
> If it can't be done directly, one can subdivide it into smaller
> timesteps and do it incrementally. At small enough timesteps, pretty
> much everything becomes linear.

You just described the first-order Euler method of solving ODEs :-)
>
> Similarly, a lot of stuff one could do with ODE's could instead be done
> using B-splines or similar, ...

An ordinary differential equation is a relation between different
quantities that you can solve given starting (and/or boundary)
conditions. There are many methods - Euler, higher-order
Runge-Kutta, Runge-Kutta, hopefully with adaptive stepsize control,
by a predictor-corrector method, Richardson extrapolation, Adams,
expict vs. explicit, ...

Soving an ODE via B-splines could be done via error minimization,
which would lead you towards the field of finite elements.

> Also, divide is one of those operations one generally wants to avoid
> when possible, not just for speed reasons, but because it is more prone
> to adding instability: if you divide by a number which happens to
> approach 0, then the numbers involved can get huge.

You can not reasonably avoid division, nor should you - the problem
is usually not the division itself, but the subtraction beforehand.

However, if you have something like

f(xn) = f(x) + dfdx(x) / (xn-x)

you can play some games of adjusting your stepsize so xn-n
can be represented exactly in floating point number, but
frankly, it usually isn't worth the other.

>Calculations which
> give wonky results, or spit out Inf, NaN, or raises a fault when given
> certain inputs, are not desirable.
>
> Usually better when possible to find an alternative which avoids the use
> of division, or at least eliminate cases where divide-by-zero exists as
> a possibility.

If your stepsize becomes zero, you're in a heap of trouble
already and probably should have errored out long before.

> Though, this does seem to be one of those points of division between
> doing math on a computer relative to traditional mathematics, which
> likes to throw in divide operators all over the place.