For the Thor 2021 version core neural network accelerator instructions
are included. Most of the instructions are for loading values into arrays
like the inputs or weights array, or variables like the bias value. The
neural network performs computations when triggered by software.
The network runs asynchronously and has a status register used to
indicate completion of a computation cycle. Eight neurons all compute
at the same time resulting in a sigmoid output for each neuron.
Thor’s neural network consists of eight neurons computing using 16.16
fixed point arithmetic. Each neuron may have up to 1024 inputs.
Activation values are computed serially with a fixed point multiply and
accumulate operation.
While there are only eight neurons in a single layer, multiple layers may
be built up by reading the output levels and using them as inputs for the
next round of calculations. The input / weights array may be partitioned
so that only part of it is used in any one calculation cycle.
I was wondering if there were any processors supporting neural networks
that I could study?
I had thought the network may be better as a memory mapped I/O device.

On 11/11/2021 10:38 PM, robf...@gmail.com wrote:
> For the Thor 2021 version core neural network accelerator instructions
> are included. Most of the instructions are for loading values into arrays
> like the inputs or weights array, or variables like the bias value. The
> neural network performs computations when triggered by software.
> The network runs asynchronously and has a status register used to
> indicate completion of a computation cycle. Eight neurons all compute
> at the same time resulting in a sigmoid output for each neuron.
> Thor’s neural network consists of eight neurons computing using 16.16
> fixed point arithmetic. Each neuron may have up to 1024 inputs.
> Activation values are computed serially with a fixed point multiply and
> accumulate operation.
> While there are only eight neurons in a single layer, multiple layers may
> be built up by reading the output levels and using them as inputs for the
> next round of calculations. The input / weights array may be partitioned
> so that only part of it is used in any one calculation cycle.
> I was wondering if there were any processors supporting neural networks
> that I could study?

There are some historical chips, and several current, either in use or
under development. I am not sure how much information you can get about
their internal architecture.

You might start with

https://en.wikipedia.org/wiki/AI_accelerator

and check out Google's Tensor Processing Unit, and for a different
approach, IBM's True North.

--
- Stephen Fuld
(e-mail address disguised to prevent spam)

On Thursday, November 11, 2021 at 11:25:42 PM UTC-8, Stephen Fuld wrote:
> On 11/11/2021 10:38 PM, robf...@gmail.com wrote:
> > For the Thor 2021 version core neural network accelerator instructions
> > are included. Most of the instructions are for loading values into arrays
> > like the inputs or weights array, or variables like the bias value. The
> > neural network performs computations when triggered by software.
> > The network runs asynchronously and has a status register used to
> > indicate completion of a computation cycle. Eight neurons all compute
> > at the same time resulting in a sigmoid output for each neuron.
> > Thor’s neural network consists of eight neurons computing using 16.16
> > fixed point arithmetic. Each neuron may have up to 1024 inputs.
> > Activation values are computed serially with a fixed point multiply and
> > accumulate operation.
> > While there are only eight neurons in a single layer, multiple layers may
> > be built up by reading the output levels and using them as inputs for the
> > next round of calculations. The input / weights array may be partitioned
> > so that only part of it is used in any one calculation cycle.
> > I was wondering if there were any processors supporting neural networks
> > that I could study?
> There are some historical chips, and several current, either in use or
> under development. I am not sure how much information you can get about
> their internal architecture.
>
> You might start with
>
> https://en.wikipedia.org/wiki/AI_accelerator
>
> and check out Google's Tensor Processing Unit, and for a different
> approach, IBM's True North.
>
>
>
> --
> - Stephen Fuld
> (e-mail address disguised to prevent spam)

While this documents an abstract machine rather than an exact implementation, it still gives a lot of insight IMHO. https://docs.nvidia.com/deeplearning/performance/dl-performance-convolutional/index.html

Another couple papers that look interesting on first scan...
https://petewarden.com/2015/04/20/why-gemm-is-at-the-heart-of-deep-learning/
https://arxiv.org/pdf/1410.0759.pdf

Another direction might be programming for Apple's M1 Neural Engine -
https://developer.apple.com/documentation/accelerate/bnns
https://developer.apple.com/documentation/metalperformanceshaders

-JohnG

Stephen Fuld wrote:
> On 11/11/2021 10:38 PM, robf...@gmail.com wrote:
>> For the Thor 2021 version core neural network accelerator instructions
>> are included. Most of the instructions are for loading values into arrays
>> like the inputs or weights array, or variables like the bias value. The
>> neural network performs computations when triggered by software.
>> The network runs asynchronously and has a status register used to
>> indicate completion of a computation cycle. Eight neurons all compute
>> at the same time resulting in a sigmoid output for each neuron.
>> Thor’s neural network consists of eight neurons computing using 16.16
>> fixed point arithmetic. Each neuron may have up to 1024 inputs.
>> Activation values are computed serially with a fixed point multiply and
>> accumulate operation.
>> While there are only eight neurons in a single layer, multiple layers may
>> be built up by reading the output levels and using them as inputs for the
>> next round of calculations. The input / weights array may be partitioned
>> so that only part of it is used in any one calculation cycle.
>> I was wondering if there were any processors supporting neural networks
>> that I could study?
>
> There are some historical chips, and several current, either in use or
> under development.  I am not sure how much information you can get about
> their internal architecture.
>
> You might start with
>
> https://en.wikipedia.org/wiki/AI_accelerator
>
> and check out Google's Tensor Processing Unit, and for a different
> approach, IBM's True North.

Please include Tesla's custom chips as well, both the runtime engine
which is installed in all their cars (10K 8-bit MACs, presumably with a
32-bit accumulator) and the training chip they use in Dojo, which I
believe is using a more or less standard 16-bit fp format.

Tesla had much more severe power constraint than Google, so I believe
their in-car chips are leading in performance/watt.

Terje

--
- <Terje.Mathisen at tmsw.no>
"almost all programming can be viewed as an exercise in caching"

On Friday, November 12, 2021 at 6:20:32 AM UTC-6, Terje Mathisen wrote:
> Stephen Fuld wrote:
> > On 11/11/2021 10:38 PM, robf...@gmail.com wrote:
> >> For the Thor 2021 version core neural network accelerator instructions
> >> are included. Most of the instructions are for loading values into arrays
> >> like the inputs or weights array, or variables like the bias value. The
> >> neural network performs computations when triggered by software.
> >> The network runs asynchronously and has a status register used to
> >> indicate completion of a computation cycle. Eight neurons all compute
> >> at the same time resulting in a sigmoid output for each neuron.
> >> Thor’s neural network consists of eight neurons computing using 16.16
> >> fixed point arithmetic. Each neuron may have up to 1024 inputs.
> >> Activation values are computed serially with a fixed point multiply and
> >> accumulate operation.
> >> While there are only eight neurons in a single layer, multiple layers may
> >> be built up by reading the output levels and using them as inputs for the
> >> next round of calculations. The input / weights array may be partitioned
> >> so that only part of it is used in any one calculation cycle.
> >> I was wondering if there were any processors supporting neural networks
> >> that I could study?
> >
> > There are some historical chips, and several current, either in use or
> > under development. I am not sure how much information you can get about
> > their internal architecture.
> >
> > You might start with
> >
> > https://en.wikipedia.org/wiki/AI_accelerator
> >
> > and check out Google's Tensor Processing Unit, and for a different
> > approach, IBM's True North.
> Please include Tesla's custom chips as well, both the runtime engine
> which is installed in all their cars (10K 8-bit MACs, presumably with a
> 32-bit accumulator) and the training chip they use in Dojo, which I
> believe is using a more or less standard 16-bit fp format.
>
> Tesla had much more severe power constraint than Google, so I believe
> their in-car chips are leading in performance/watt.
>
> Terje
>
>
> --
> - <Terje.Mathisen at tmsw.no>
> "almost all programming can be viewed as an exercise in caching"

Dojo also supports 8-bit fp using a "block floating-point" exponent offset
https://cdn.motor1.com/pdf-files/535242876-tesla-dojo-technology.pdf
The Posit and NN literature have studies of shortened fp for NN weights.

Not sure if training uses larger fp and then application instances use shortened fp?

I think it is rather safe to say (at this point in time) the NN-accelerators
are in the 704 days of these kinds of architectures.

A bit more than barely function, and a long way to go.........

MitchAlsup wrote:
> I think it is rather safe to say (at this point in time) the NN-accelerators
> are in the 704 days of these kinds of architectures.
>
> A bit more than barely function, and a long way to go.........

The NN talked about mostly in the press and which most vendors
are trying to sell you are Convolution Neural Networks (CNN)
which are basically multiple layers of sums.
You can build a fancy pattern matcher out of it but
it will never make a good decision mechanism.
It will always be a mystery why it "decided" a certain way
because it is just calculating a great whacking polynomial.
As NN go, I have a gut feeling that is a dead end.

Real NN have feedback, called recurrent, and real neurons are spiky
which introduces signal timing and phase delays as attributes.

In particular signal phase timing adds a whole new dimension for
information storage. Feedback allows resonances to enhance or
suppress different combinations of inputs.

That is the basis for my suspicion that we will eventually find
that brains, all brains, are akin to _holograms_.

Clusters of neurons can build holographic modules and form
holographic modules of modules.

On 11/12/2021 4:52 PM, MitchAlsup wrote:
> I think it is rather safe to say (at this point in time) the NN-accelerators
> are in the 704 days of these kinds of architectures.
>
> A bit more than barely function, and a long way to go.........
>

Yeah, it is a bit hit or miss even what they should be doing exactly...

Some of the popular options seem to be in-effect doing large glorified
matrix multiplies using a truncated floating format (BF16, essentially
Binary32 with the low 16 bits cut off).

Not sure if there is a "good" reason for BF16, or if it is more a
workaround for popular / mainstream CPU architectures lacking support
for Binary16 SIMD.

It comes off like, say, if I proposed a new 32-bit floating point format
that was effectively just Double with the low 32 bits cut off (then try
to pass it off as "better" when it is really just a way to allow cheaper
format conversions).

Both Binary16 and BF16 could make sense for dedicated low-precision SIMD
ops, and also are small enough to be implemented reasonably affordably
on an FPGA.

For general use, Binary16 (S.E5.F10) probably makes more sense, though
one could debate whether BF16 (S.E8.F7) has enough value in the
general-case to make it worthwhile (short of trying to do a TensorFlow
port or similar, I have doubts).

More debatable, but one could argue for an 8x or 16x (S.E4.F3) vectors
for Neural-Net uses. Though, I suspect the gains would be small as the
relative cost of wrangling inputs would somewhat outweigh the possible
savings from such operators over operating on 16-bit elements (with
possible Packed FP8 <-> FP16 conversion operators).

Intermediate options are mostly those involving FP10 (S.E5.F4) or FP12
(S.E5.F6):
3x FP10 in 32 bits (Maps to 4x FP16, X/Y/Z/{0/-/1/-1});
6x FP10 in 64 bits (Maps to 8x FP16, X/Y/Z/0,P/Q/R/0);
4x FP12 in 48 bits;
...

If one expected to do a lot of FP16, this could justify the cost of have
dedicated SIMD units for these (rather than run them internally through
a slower but higher precision FPU).

EricP wrote:
>
> Real NN have feedback, called recurrent, and real neurons are spiky
> which introduces signal timing and phase delays as attributes.
>
> In particular signal phase timing adds a whole new dimension for
> information storage. Feedback allows resonances to enhance or
> suppress different combinations of inputs.
>
> That is the basis for my suspicion that we will eventually find
> that brains, all brains, are akin to _holograms_.
>
> Clusters of neurons can build holographic modules and form
> holographic modules of modules.

One mystery about this is why doesn't every brain immediately
collapse in a giant epileptic fit. Nature must have found a
way to detect and prevent it as the organism grows.
Natural selection would be a poor mechanism because there are
so many ways to fail and many fewer ways to succeed
that almost no brains would survive.

So there must be some mechanism that "drives" these self organizing
networks toward interconnections that do not have uncontrolled feedback.

I've said this before but I suspect _that_ is what nature discovered
at the Cambrian explosion 540 million years ago.

On Saturday, November 13, 2021 at 12:37:53 PM UTC-6, BGB wrote:
> On 11/12/2021 4:52 PM, MitchAlsup wrote:
> > I think it is rather safe to say (at this point in time) the NN-accelerators
> > are in the 704 days of these kinds of architectures.
> >
> > A bit more than barely function, and a long way to go.........
> >
> Yeah, it is a bit hit or miss even what they should be doing exactly...
>
>
> Some of the popular options seem to be in-effect doing large glorified
> matrix multiplies using a truncated floating format (BF16, essentially
> Binary32 with the low 16 bits cut off).
>
>
> Not sure if there is a "good" reason for BF16, or if it is more a
> workaround for popular / mainstream CPU architectures lacking support
> for Binary16 SIMD.
>
> It comes off like, say, if I proposed a new 32-bit floating point format
> that was effectively just Double with the low 32 bits cut off (then try
> to pass it off as "better" when it is really just a way to allow cheaper
> format conversions).
>
>
> Both Binary16 and BF16 could make sense for dedicated low-precision SIMD
> ops, and also are small enough to be implemented reasonably affordably
> on an FPGA.
>
> For general use, Binary16 (S.E5.F10) probably makes more sense, though
> one could debate whether BF16 (S.E8.F7) has enough value in the
> general-case to make it worthwhile (short of trying to do a TensorFlow
> port or similar, I have doubts).
<
A lot of the self-driving NNs can use as few as 1-bit weighting matrices
and 8-bit accumulators and achieve useful pattern recognition rates.
Indeed, much of the work, here, is centered around decreasing the storage
BW to feed to convolution engine than on making the convolution engine
faster or larger.
<
I am pretty sure this kind of experiments will continue for another decade.
>
>
> More debatable, but one could argue for an 8x or 16x (S.E4.F3) vectors
> for Neural-Net uses. Though, I suspect the gains would be small as the
> relative cost of wrangling inputs would somewhat outweigh the possible
> savings from such operators over operating on 16-bit elements (with
> possible Packed FP8 <-> FP16 conversion operators).
<
One big problem is how does one "express" such a matrix multiplication,
where the size of the containers in the weighting matrix change on a per
weight basis.
>
> Intermediate options are mostly those involving FP10 (S.E5.F4) or FP12
> (S.E5.F6):
> 3x FP10 in 32 bits (Maps to 4x FP16, X/Y/Z/{0/-/1/-1});
> 6x FP10 in 64 bits (Maps to 8x FP16, X/Y/Z/0,P/Q/R/0);
> 4x FP12 in 48 bits;
> ...
>
> If one expected to do a lot of FP16, this could justify the cost of have
> dedicated SIMD units for these (rather than run them internally through
> a slower but higher precision FPU).

EricP wrote:
> MitchAlsup wrote:
>> I think it is rather safe to say (at this point in time) the
>> NN-accelerators
>> are in the 704 days of these kinds of architectures.
>>
>> A bit more than barely function, and a long way to go.........
>
> The NN talked about mostly in the press and which most vendors
> are trying to sell you are Convolution Neural Networks (CNN)
> which are basically multiple layers of sums.
> You can build a fancy pattern matcher out of it but
> it will never make a good decision mechanism.
> It will always be a mystery why it "decided" a certain way
> because it is just calculating a great whacking polynomial.
> As NN go, I have a gut feeling that is a dead end.

This all reminds me of the fuzzy logic fad of the mid 1980's.
It was invented in the 1920's and the term was coined in 1965.
For some reason is became "a thing" in the tech magazines around 1985
for a while (though a quicky search still finds lots of references).
Everything was going fuzzy.
Fuzzy logic sometimes was and still is labeled artificial intelligence too.

On 11/13/2021 10:54 AM, EricP wrote:
> EricP wrote:
>>
>> Real NN have feedback, called recurrent, and real neurons are spiky
>> which introduces signal timing and phase delays as attributes.
>>
>> In particular signal phase timing adds a whole new dimension for
>> information storage. Feedback allows resonances to enhance or
>> suppress different combinations of inputs.
>>
>> That is the basis for my suspicion that we will eventually find
>> that brains, all brains, are akin to _holograms_.
>>
>> Clusters of neurons can build holographic modules and form
>> holographic modules of modules.
>
> One mystery about this is why doesn't every brain immediately
> collapse in a giant epileptic fit. Nature must have found a
> way to detect and prevent it as the organism grows.
> Natural selection would be a poor mechanism because there are
> so many ways to fail and many fewer ways to succeed
> that almost no brains would survive.
>
> So there must be some mechanism that "drives" these self organizing
> networks toward interconnections that do not have uncontrolled feedback.
>
> I've said this before but I suspect _that_ is what nature discovered
> at the Cambrian explosion 540 million years ago.
>

I thought the Cambrian invention was teeth?

On 11/13/2021 1:30 PM, MitchAlsup wrote:
> On Saturday, November 13, 2021 at 12:37:53 PM UTC-6, BGB wrote:
>> On 11/12/2021 4:52 PM, MitchAlsup wrote:
>>> I think it is rather safe to say (at this point in time) the NN-accelerators
>>> are in the 704 days of these kinds of architectures.
>>>
>>> A bit more than barely function, and a long way to go.........
>>>
>> Yeah, it is a bit hit or miss even what they should be doing exactly...
>>
>>
>> Some of the popular options seem to be in-effect doing large glorified
>> matrix multiplies using a truncated floating format (BF16, essentially
>> Binary32 with the low 16 bits cut off).
>>
>>
>> Not sure if there is a "good" reason for BF16, or if it is more a
>> workaround for popular / mainstream CPU architectures lacking support
>> for Binary16 SIMD.
>>
>> It comes off like, say, if I proposed a new 32-bit floating point format
>> that was effectively just Double with the low 32 bits cut off (then try
>> to pass it off as "better" when it is really just a way to allow cheaper
>> format conversions).
>>
>>
>> Both Binary16 and BF16 could make sense for dedicated low-precision SIMD
>> ops, and also are small enough to be implemented reasonably affordably
>> on an FPGA.
>>
>> For general use, Binary16 (S.E5.F10) probably makes more sense, though
>> one could debate whether BF16 (S.E8.F7) has enough value in the
>> general-case to make it worthwhile (short of trying to do a TensorFlow
>> port or similar, I have doubts).
> <
> A lot of the self-driving NNs can use as few as 1-bit weighting matrices
> and 8-bit accumulators and achieve useful pattern recognition rates.
> Indeed, much of the work, here, is centered around decreasing the storage
> BW to feed to convolution engine than on making the convolution engine
> faster or larger.
> <
> I am pretty sure this kind of experiments will continue for another decade.

Yeah.

I suspect in my case, some of the Block-Texture and Block-Audio ops
could also conceivably be applicable to NN weights, at least in very
specialized use-cases.

It is also possible (for cases involving vectors with constant weights),
that I could define encodings for, say:
PLDCM8SH Imm32, Rn //Load 4x FP8 into 4x Binary16 in Rn.
PLDCM8UH Imm32, Rn //Load 4x FP8 into 4x Binary16 in Rn.
PLDCH Imm32, Rn //Load 2x Binary16 into 2x Binary32

These could also make sense for vector-literals in C, but at the moment
is a lower priority due to the use of vector-literals in C being a
fairly niche feature.

Went and added a few possible encodings to the listing (in Op64 space),
which mostly add alternative forms to the existing "FLDCF Imm32, Rn"
encoding.

>>
>>
>> More debatable, but one could argue for an 8x or 16x (S.E4.F3) vectors
>> for Neural-Net uses. Though, I suspect the gains would be small as the
>> relative cost of wrangling inputs would somewhat outweigh the possible
>> savings from such operators over operating on 16-bit elements (with
>> possible Packed FP8 <-> FP16 conversion operators).
> <
> One big problem is how does one "express" such a matrix multiplication,
> where the size of the containers in the weighting matrix change on a per
> weight basis.

In many contexts, there is a trick that one can store a matrix in a
pre-transposed form and turn it into a bunch of vector-multiply and
accumulate operations.

Though, as can be noted, doing a net with naive matrix multiplies does
mean one is going to be doing a whole lot of meaningless multiplies with
zero.

I guess one possible trick (if doing a large matrix multiply with a
loop), would be using an RLE scheme to skip over vectors for runs where
all of the components are zeroes (and "pre-cooking" the model to
eliminate large sections of nearly-zero values).

Though, as can be noted:
I don't really use these sort of "huge matrix multiply" style nets in
any of my own projects.

My own uses have tended to be mostly things like using genetic
algorithms to build more specialized classifiers.

These would tend to be more things like "do some math on some vectors"
followed by comparing the results against some threshold biases, and
running the result into a decision tree. The "fancier" version would
likely be having some way to transform the vector-compare result into a
bit-mask which could be fed into a "switch()" block or similar (as-is,
this can theoretically be done for 4-element vector-compare results).

There is not currently a dedicated S-Curve operation, though when
needed, variations of the Heaviside function can be built using packed
compare and packed select (*1).

In these cases, the weight and bias vectors would typically be embedded
directly into the program logic (though, granted, this sort of thing
falls slightly outside the scope of "standard C").

*1:
const __vec4sf cv_n1={-1,-1,-1,-1}, cv_p1={1,1,1,1};
__vec4sf v, bi, wv;

... calculate v ...
wv = __vec4sf_pcsel_gt(v, bi, cv_p1, cv_n1);

Or, say, psuedo-asm:
PLDCM8SH 0x38383838, cv_p1 //possible encoding
PLDCM8SH 0xB8B8B8B8, cv_n1 //possible encoding
...
PCMPGT.H bi, v
PCSELT.W cv_p1, cv_n1, wv

Which, for each vector element, does effectively:
wv[i] = (v[i] > bi[i]) ? cv_p1[i] : cv_n1[i];
Or, alternately:
wv[i] = (v[i] > bi[i]) ? 1.0 : -1.0;

....

>>
>> Intermediate options are mostly those involving FP10 (S.E5.F4) or FP12
>> (S.E5.F6):
>> 3x FP10 in 32 bits (Maps to 4x FP16, X/Y/Z/{0/-/1/-1});
>> 6x FP10 in 64 bits (Maps to 8x FP16, X/Y/Z/0,P/Q/R/0);
>> 4x FP12 in 48 bits;
>> ...
>>
>> If one expected to do a lot of FP16, this could justify the cost of have
>> dedicated SIMD units for these (rather than run them internally through
>> a slower but higher precision FPU).

On Saturday, November 13, 2021 at 12:25:22 PM UTC-8, Ivan Godard wrote:
> On 11/13/2021 10:54 AM, EricP wrote:
> > EricP wrote:
> >>
> >> Real NN have feedback, called recurrent, and real neurons are spiky
> >> which introduces signal timing and phase delays as attributes.
> >>
> >> In particular signal phase timing adds a whole new dimension for
> >> information storage. Feedback allows resonances to enhance or
> >> suppress different combinations of inputs.
> >>
> >> That is the basis for my suspicion that we will eventually find
> >> that brains, all brains, are akin to _holograms_.
> >>
> >> Clusters of neurons can build holographic modules and form
> >> holographic modules of modules.
> >
> > One mystery about this is why doesn't every brain immediately
> > collapse in a giant epileptic fit. Nature must have found a
> > way to detect and prevent it as the organism grows.
> > Natural selection would be a poor mechanism because there are
> > so many ways to fail and many fewer ways to succeed
> > that almost no brains would survive.
> >
> > So there must be some mechanism that "drives" these self organizing
> > networks toward interconnections that do not have uncontrolled feedback..
> >
> > I've said this before but I suspect _that_ is what nature discovered
> > at the Cambrian explosion 540 million years ago.
> >
> I thought the Cambrian invention was teeth?

The internet says an early Cambrian innovation was hard parts, like shells and plates. The oldest toothed fossil dates to 410 Mya, but the Cambrian Era was 541 - 485.4 Mya. (Btw, more recent fossils suggest that the appearance of new body plans was more gradual than once believed, so not so "explosive.")

Arguing in favor of EricP's point, the cerebellum (which has been shown to at least be able to influence seizures) was a Cambrian "invention," according to Paul Cisek in "Resynthesizing behavior through phylogenetic refinement." (Great read!) However, Cisek also says image-forming eyes were already being used in visually guided approach and reinforcement learning by telencephalons during the Pre-Cambrian Era. To me that suggests that the solution for avoiding epileptic seizures was baked in even earlier.

On 11/13/2021 3:56 PM, Scott Smader wrote:
> On Saturday, November 13, 2021 at 12:25:22 PM UTC-8, Ivan Godard wrote:
>> On 11/13/2021 10:54 AM, EricP wrote:
>>> EricP wrote:
>>>>
>>>> Real NN have feedback, called recurrent, and real neurons are spiky
>>>> which introduces signal timing and phase delays as attributes.
>>>>
>>>> In particular signal phase timing adds a whole new dimension for
>>>> information storage. Feedback allows resonances to enhance or
>>>> suppress different combinations of inputs.
>>>>
>>>> That is the basis for my suspicion that we will eventually find
>>>> that brains, all brains, are akin to _holograms_.
>>>>
>>>> Clusters of neurons can build holographic modules and form
>>>> holographic modules of modules.
>>>
>>> One mystery about this is why doesn't every brain immediately
>>> collapse in a giant epileptic fit. Nature must have found a
>>> way to detect and prevent it as the organism grows.
>>> Natural selection would be a poor mechanism because there are
>>> so many ways to fail and many fewer ways to succeed
>>> that almost no brains would survive.
>>>
>>> So there must be some mechanism that "drives" these self organizing
>>> networks toward interconnections that do not have uncontrolled feedback.
>>>
>>> I've said this before but I suspect _that_ is what nature discovered
>>> at the Cambrian explosion 540 million years ago.
>>>
>> I thought the Cambrian invention was teeth?
>
> The internet says an early Cambrian innovation was hard parts, like shells and plates. The oldest toothed fossil dates to 410 Mya, but the Cambrian Era was 541 - 485.4 Mya. (Btw, more recent fossils suggest that the appearance of new body plans was more gradual than once believed, so not so "explosive.")
>
> Arguing in favor of EricP's point, the cerebellum (which has been shown to at least be able to influence seizures) was a Cambrian "invention," according to Paul Cisek in "Resynthesizing behavior through phylogenetic refinement." (Great read!) However, Cisek also says image-forming eyes were already being used in visually guided approach and reinforcement learning by telencephalons during the Pre-Cambrian Era. To me that suggests that the solution for avoiding epileptic seizures was baked in even earlier.

Why would you evolve shells if nobody has teeth?

On Saturday, November 13, 2021 at 4:01:16 PM UTC-8, Ivan Godard wrote:
> On 11/13/2021 3:56 PM, Scott Smader wrote:
> > On Saturday, November 13, 2021 at 12:25:22 PM UTC-8, Ivan Godard wrote:
> >> On 11/13/2021 10:54 AM, EricP wrote:
> >>> EricP wrote:
> >>>>
> >>>> Real NN have feedback, called recurrent, and real neurons are spiky
> >>>> which introduces signal timing and phase delays as attributes.
> >>>>
> >>>> In particular signal phase timing adds a whole new dimension for
> >>>> information storage. Feedback allows resonances to enhance or
> >>>> suppress different combinations of inputs.
> >>>>
> >>>> That is the basis for my suspicion that we will eventually find
> >>>> that brains, all brains, are akin to _holograms_.
> >>>>
> >>>> Clusters of neurons can build holographic modules and form
> >>>> holographic modules of modules.
> >>>
> >>> One mystery about this is why doesn't every brain immediately
> >>> collapse in a giant epileptic fit. Nature must have found a
> >>> way to detect and prevent it as the organism grows.
> >>> Natural selection would be a poor mechanism because there are
> >>> so many ways to fail and many fewer ways to succeed
> >>> that almost no brains would survive.
> >>>
> >>> So there must be some mechanism that "drives" these self organizing
> >>> networks toward interconnections that do not have uncontrolled feedback.
> >>>
> >>> I've said this before but I suspect _that_ is what nature discovered
> >>> at the Cambrian explosion 540 million years ago.
> >>>
> >> I thought the Cambrian invention was teeth?
> >
> > The internet says an early Cambrian innovation was hard parts, like shells and plates. The oldest toothed fossil dates to 410 Mya, but the Cambrian Era was 541 - 485.4 Mya. (Btw, more recent fossils suggest that the appearance of new body plans was more gradual than once believed, so not so "explosive.")
> >
> > Arguing in favor of EricP's point, the cerebellum (which has been shown to at least be able to influence seizures) was a Cambrian "invention," according to Paul Cisek in "Resynthesizing behavior through phylogenetic refinement." (Great read!) However, Cisek also says image-forming eyes were already being used in visually guided approach and reinforcement learning by telencephalons during the Pre-Cambrian Era. To me that suggests that the solution for avoiding epileptic seizures was baked in even earlier.
> Why would you evolve shells if nobody has teeth?

There are some mean suckers out there!

On Saturday, November 13, 2021 at 6:01:16 PM UTC-6, Ivan Godard wrote:
> On 11/13/2021 3:56 PM, Scott Smader wrote:
> > On Saturday, November 13, 2021 at 12:25:22 PM UTC-8, Ivan Godard wrote:
> >> On 11/13/2021 10:54 AM, EricP wrote:
> >>> EricP wrote:
> >>>>
> >>>> Real NN have feedback, called recurrent, and real neurons are spiky
> >>>> which introduces signal timing and phase delays as attributes.
> >>>>
> >>>> In particular signal phase timing adds a whole new dimension for
> >>>> information storage. Feedback allows resonances to enhance or
> >>>> suppress different combinations of inputs.
> >>>>
> >>>> That is the basis for my suspicion that we will eventually find
> >>>> that brains, all brains, are akin to _holograms_.
> >>>>
> >>>> Clusters of neurons can build holographic modules and form
> >>>> holographic modules of modules.
> >>>
> >>> One mystery about this is why doesn't every brain immediately
> >>> collapse in a giant epileptic fit. Nature must have found a
> >>> way to detect and prevent it as the organism grows.
> >>> Natural selection would be a poor mechanism because there are
> >>> so many ways to fail and many fewer ways to succeed
> >>> that almost no brains would survive.
> >>>
> >>> So there must be some mechanism that "drives" these self organizing
> >>> networks toward interconnections that do not have uncontrolled feedback.
> >>>
> >>> I've said this before but I suspect _that_ is what nature discovered
> >>> at the Cambrian explosion 540 million years ago.
> >>>
> >> I thought the Cambrian invention was teeth?
> >
> > The internet says an early Cambrian innovation was hard parts, like shells and plates. The oldest toothed fossil dates to 410 Mya, but the Cambrian Era was 541 - 485.4 Mya. (Btw, more recent fossils suggest that the appearance of new body plans was more gradual than once believed, so not so "explosive.")
> >
> > Arguing in favor of EricP's point, the cerebellum (which has been shown to at least be able to influence seizures) was a Cambrian "invention," according to Paul Cisek in "Resynthesizing behavior through phylogenetic refinement." (Great read!) However, Cisek also says image-forming eyes were already being used in visually guided approach and reinforcement learning by telencephalons during the Pre-Cambrian Era. To me that suggests that the solution for avoiding epileptic seizures was baked in even earlier.
<
> Why would you evolve shells if nobody has teeth?
<
The precursor to teeth is chitin (stuff of fingernails and claws). This is sufficient to "bite" through non-hardened skin (and also how the first hardened-skin evolved.
<
So after the evolution of chitin, somebody had to evolve a series of stuff even harder (for protection) and the arms race continues........even until today........

On Saturday, November 13, 2021 at 12:28:55 PM UTC-6, EricP wrote:
> MitchAlsup wrote:
> > I think it is rather safe to say (at this point in time) the NN-accelerators
> > are in the 704 days of these kinds of architectures.
> >
> > A bit more than barely function, and a long way to go.........
> The NN talked about mostly in the press and which most vendors
> are trying to sell you are Convolution Neural Networks (CNN)
> which are basically multiple layers of sums.
> You can build a fancy pattern matcher out of it but
> it will never make a good decision mechanism.
> It will always be a mystery why it "decided" a certain way
> because it is just calculating a great whacking polynomial.
> As NN go, I have a gut feeling that is a dead end.
>
> Real NN have feedback, called recurrent, and real neurons are spiky
> which introduces signal timing and phase delays as attributes.
>
> In particular signal phase timing adds a whole new dimension for
> information storage. Feedback allows resonances to enhance or
> suppress different combinations of inputs.
>
> That is the basis for my suspicion that we will eventually find
> that brains, all brains, are akin to _holograms_.
>
> Clusters of neurons can build holographic modules and form
> holographic modules of modules.

The Long Short Term Memory (LSTM) type of ANN has such feedback
connections and works well. Also see Recurrent Neural Network (RNN).

On 11/13/2021 10:28 AM, EricP wrote:
> MitchAlsup wrote:
>> I think it is rather safe to say (at this point in time) the
>> NN-accelerators
>> are in the 704 days of these kinds of architectures.
>>
>> A bit more than barely function, and a long way to go.........
>
> The NN talked about mostly in the press and which most vendors
> are trying to sell you are Convolution Neural Networks (CNN)
> which are basically multiple layers of sums.
> You can build a fancy pattern matcher out of it but
> it will never make a good decision mechanism.

I disagree with the last phrase. NNs have been used for decision
making, and I expect their use will continue, as they are better at
certain kinds of decisions than other, more "conventional" algorithms.
Of course, one may quibble about how "good" is good, but that is a
different question.

BTW, the brain seems to be a sophisticated pattern matcher, and it works
pretty well at many tasks.

> It will always be a mystery why it "decided" a certain way
> because it is just calculating a great whacking polynomial.

There has been, and continues to be work on this, and some progress is
being made. But, while it certainly would be nice, I am not sure it is
necessary for them to be useful.

> As NN go, I have a gut feeling that is a dead end.

Could be. Time will tell.

>
> Real NN have feedback, called recurrent, and real neurons are spiky
> which introduces signal timing and phase delays as attributes.

Sure. You have to distinguish between two different "uses" for neural
networks. One is "science", trying to figure out how the brain works.
These are usually research projects, and their emphasis is on biological
faithfulness. The other is "engineering", trying to find a better way
of solving some useful problem. For this purpose, biological
faithfulness isn't critical. That is where most of the money is, and all
of the hype.

--
- Stephen Fuld
(e-mail address disguised to prevent spam)

On 11/13/2021 5:26 PM, MitchAlsup wrote:
> On Saturday, November 13, 2021 at 6:01:16 PM UTC-6, Ivan Godard wrote:
>> On 11/13/2021 3:56 PM, Scott Smader wrote:
>>> On Saturday, November 13, 2021 at 12:25:22 PM UTC-8, Ivan Godard wrote:
>>>> On 11/13/2021 10:54 AM, EricP wrote:
>>>>> EricP wrote:
>>>>>>
>>>>>> Real NN have feedback, called recurrent, and real neurons are spiky
>>>>>> which introduces signal timing and phase delays as attributes.
>>>>>>
>>>>>> In particular signal phase timing adds a whole new dimension for
>>>>>> information storage. Feedback allows resonances to enhance or
>>>>>> suppress different combinations of inputs.
>>>>>>
>>>>>> That is the basis for my suspicion that we will eventually find
>>>>>> that brains, all brains, are akin to _holograms_.
>>>>>>
>>>>>> Clusters of neurons can build holographic modules and form
>>>>>> holographic modules of modules.
>>>>>
>>>>> One mystery about this is why doesn't every brain immediately
>>>>> collapse in a giant epileptic fit. Nature must have found a
>>>>> way to detect and prevent it as the organism grows.
>>>>> Natural selection would be a poor mechanism because there are
>>>>> so many ways to fail and many fewer ways to succeed
>>>>> that almost no brains would survive.
>>>>>
>>>>> So there must be some mechanism that "drives" these self organizing
>>>>> networks toward interconnections that do not have uncontrolled feedback.
>>>>>
>>>>> I've said this before but I suspect _that_ is what nature discovered
>>>>> at the Cambrian explosion 540 million years ago.
>>>>>
>>>> I thought the Cambrian invention was teeth?
>>>
>>> The internet says an early Cambrian innovation was hard parts, like shells and plates. The oldest toothed fossil dates to 410 Mya, but the Cambrian Era was 541 - 485.4 Mya. (Btw, more recent fossils suggest that the appearance of new body plans was more gradual than once believed, so not so "explosive.")
>>>
>>> Arguing in favor of EricP's point, the cerebellum (which has been shown to at least be able to influence seizures) was a Cambrian "invention," according to Paul Cisek in "Resynthesizing behavior through phylogenetic refinement." (Great read!) However, Cisek also says image-forming eyes were already being used in visually guided approach and reinforcement learning by telencephalons during the Pre-Cambrian Era. To me that suggests that the solution for avoiding epileptic seizures was baked in even earlier.
> <
>> Why would you evolve shells if nobody has teeth?
> <
> The precursor to teeth is chitin (stuff of fingernails and claws). This is sufficient to "bite" through non-hardened skin (and also how the first hardened-skin evolved.
> <
> So after the evolution of chitin, somebody had to evolve a series of stuff even harder (for protection) and the arms race continues........even until today........
>

And shells had precursors too. Call any offensive hard surface a
"tooth", and any defensive hard surface a "shell". I assert that teeth
precede shells.

Ivan Godard wrote:
> On 11/13/2021 10:54 AM, EricP wrote:
>> EricP wrote:
>>>
>>> Real NN have feedback, called recurrent, and real neurons are spiky
>>> which introduces signal timing and phase delays as attributes.
>>>
>>> In particular signal phase timing adds a whole new dimension for
>>> information storage. Feedback allows resonances to enhance or
>>> suppress different combinations of inputs.
>>>
>>> That is the basis for my suspicion that we will eventually find
>>> that brains, all brains, are akin to _holograms_.
>>>
>>> Clusters of neurons can build holographic modules and form
>>> holographic modules of modules.
>>
>> One mystery about this is why doesn't every brain immediately
>> collapse in a giant epileptic fit. Nature must have found a
>> way to detect and prevent it as the organism grows.
>> Natural selection would be a poor mechanism because there are
>> so many ways to fail and many fewer ways to succeed
>> that almost no brains would survive.
>>
>> So there must be some mechanism that "drives" these self organizing
>> networks toward interconnections that do not have uncontrolled feedback.
>>
>> I've said this before but I suspect _that_ is what nature discovered
>> at the Cambrian explosion 540 million years ago.
>>
>
> I thought the Cambrian invention was teeth?

Eyes developed then too, but so did legs, antenna, all complex life forms.
And probably teeth too, which need muscles to work them.
And all that requires a complex controller, particularly eyes.

In pre-Cambrian the most complex life was things like jellyfish which
are multicellular organisms and have nerves that allows them to swim,
and a few neurons are specialized to detect light or dark,
but no central controller, no complex signal processing or
decision making capability.

After the Cambrian line are arthropods and all the animals of today.
Eyes developed at this time which needs complex signal processing.

Clearly something changed at that boundary that allowed the assembly
of complex NN to control all of these new functions.

On Sunday, November 14, 2021 at 6:18:51 AM UTC-8, EricP wrote:
> Ivan Godard wrote:
> > On 11/13/2021 10:54 AM, EricP wrote:
> >> EricP wrote:
> >>>
> >>> Real NN have feedback, called recurrent, and real neurons are spiky
> >>> which introduces signal timing and phase delays as attributes.
> >>>
> >>> In particular signal phase timing adds a whole new dimension for
> >>> information storage. Feedback allows resonances to enhance or
> >>> suppress different combinations of inputs.
> >>>
> >>> That is the basis for my suspicion that we will eventually find
> >>> that brains, all brains, are akin to _holograms_.
> >>>
> >>> Clusters of neurons can build holographic modules and form
> >>> holographic modules of modules.
> >>
> >> One mystery about this is why doesn't every brain immediately
> >> collapse in a giant epileptic fit. Nature must have found a
> >> way to detect and prevent it as the organism grows.
> >> Natural selection would be a poor mechanism because there are
> >> so many ways to fail and many fewer ways to succeed
> >> that almost no brains would survive.
> >>
> >> So there must be some mechanism that "drives" these self organizing
> >> networks toward interconnections that do not have uncontrolled feedback.
> >>
> >> I've said this before but I suspect _that_ is what nature discovered
> >> at the Cambrian explosion 540 million years ago.
> >>
> >
> > I thought the Cambrian invention was teeth?
> Eyes developed then too, but so did legs, antenna, all complex life forms..
> And probably teeth too, which need muscles to work them.
> And all that requires a complex controller, particularly eyes.
>
> In pre-Cambrian the most complex life was things like jellyfish which
> are multicellular organisms and have nerves that allows them to swim,
> and a few neurons are specialized to detect light or dark,
> but no central controller, no complex signal processing or
> decision making capability.
>
> After the Cambrian line are arthropods and all the animals of today.
> Eyes developed at this time which needs complex signal processing.
>
> Clearly something changed at that boundary that allowed the assembly
> of complex NN to control all of these new functions.

The opinions being expressed here would do well to refer to contemporary research. A "tooth" means something specific in the fossil record. The Cambrian Era was not a "line" or "boundary." (https://www.nature.com/articles/s41559-019-0821-6) The neural tube had already yielded to archencephalon which had yielded to telencephalon as the most complex neural circuitry long before the Cambrian Era started. Tens of millions of years before the Cambrian, dopamine was already generating sophisticated foraging behavior that is today displayed by "microorganisms, insects, mollusks, reptiles, fish, birds, and even human hunter–gatherers" (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/).

Please consider reading this paper by an expert in the field: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/ or at least have a good look at Figure 2 from that paper: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/figure/Fig2/?report=objectonly

Then maybe we should start a separate thread?

Scott Smader wrote:
> On Sunday, November 14, 2021 at 6:18:51 AM UTC-8, EricP wrote:
>> Ivan Godard wrote:
>>> On 11/13/2021 10:54 AM, EricP wrote:
>>>> EricP wrote:
>>>>> Real NN have feedback, called recurrent, and real neurons are spiky
>>>>> which introduces signal timing and phase delays as attributes.
>>>>>
>>>>> In particular signal phase timing adds a whole new dimension for
>>>>> information storage. Feedback allows resonances to enhance or
>>>>> suppress different combinations of inputs.
>>>>>
>>>>> That is the basis for my suspicion that we will eventually find
>>>>> that brains, all brains, are akin to _holograms_.
>>>>>
>>>>> Clusters of neurons can build holographic modules and form
>>>>> holographic modules of modules.
>>>> One mystery about this is why doesn't every brain immediately
>>>> collapse in a giant epileptic fit. Nature must have found a
>>>> way to detect and prevent it as the organism grows.
>>>> Natural selection would be a poor mechanism because there are
>>>> so many ways to fail and many fewer ways to succeed
>>>> that almost no brains would survive.
>>>>
>>>> So there must be some mechanism that "drives" these self organizing
>>>> networks toward interconnections that do not have uncontrolled feedback.
>>>>
>>>> I've said this before but I suspect _that_ is what nature discovered
>>>> at the Cambrian explosion 540 million years ago.
>>>>
>>> I thought the Cambrian invention was teeth?
>> Eyes developed then too, but so did legs, antenna, all complex life forms..
>> And probably teeth too, which need muscles to work them.
>> And all that requires a complex controller, particularly eyes.
>>
>> In pre-Cambrian the most complex life was things like jellyfish which
>> are multicellular organisms and have nerves that allows them to swim,
>> and a few neurons are specialized to detect light or dark,
>> but no central controller, no complex signal processing or
>> decision making capability.
>>
>> After the Cambrian line are arthropods and all the animals of today.
>> Eyes developed at this time which needs complex signal processing.
>>
>> Clearly something changed at that boundary that allowed the assembly
>> of complex NN to control all of these new functions.
>
> The opinions being expressed here would do well to refer to contemporary research. A "tooth" means something specific in the fossil record. The Cambrian Era was not a "line" or "boundary." (https://www.nature.com/articles/s41559-019-0821-6) The neural tube had already yielded to archencephalon which had yielded to telencephalon as the most complex neural circuitry long before the Cambrian Era started. Tens of millions of years before the Cambrian, dopamine was already generating sophisticated foraging behavior that is today displayed by "microorganisms, insects, mollusks, reptiles, fish, birds, and even human hunter–gatherers" (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/)." rel="nofollow" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/).
>
> Please consider reading this paper by an expert in the field: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/ or at least have a good look at Figure 2 from that paper: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/figure/Fig2/?report=objectonly

Thanks, I just found the Cisek 2019 paper online this morning.
I'll have a look at the other too.

Problem is that when I started searching on this I came across
a whole bunch of papers that are also on-topic so it is easy to
get side tracked. E.G. I started looking at

[open access]
On the Independent Origins of Complex Brains and Neurons, 2019
https://www.karger.com/Article/FullText/258665

The problem is finding research that deals specifically with the
development neural interconnect and how it was able to scale out,
as opposed to say the evolution of different neural transmitters.

> Then maybe we should start a separate thread?

Seems on-topic to me (certainly more so than some of the other
threads discussions on the motivations of deities).

Understanding the origin of the wiring of biological NN (BNN)
is appropriate to discussion of NN Accelerators as we are
endeavoring to improve such simulators.

Hardware architectures like Tensorflow are likely not appropriate
for recurrent spiking NN (RSNN) and some researchers have been
exploring new hardware accelerators in this area.

On Sunday, November 14, 2021 at 10:46:40 AM UTC-6, EricP wrote:
> Scott Smader wrote:
> > On Sunday, November 14, 2021 at 6:18:51 AM UTC-8, EricP wrote:
> >> Ivan Godard wrote:
> >>> On 11/13/2021 10:54 AM, EricP wrote:
> >>>> EricP wrote:
> >>>>> Real NN have feedback, called recurrent, and real neurons are spiky
> >>>>> which introduces signal timing and phase delays as attributes.
> >>>>>
> >>>>> In particular signal phase timing adds a whole new dimension for
> >>>>> information storage. Feedback allows resonances to enhance or
> >>>>> suppress different combinations of inputs.
> >>>>>
> >>>>> That is the basis for my suspicion that we will eventually find
> >>>>> that brains, all brains, are akin to _holograms_.
> >>>>>
> >>>>> Clusters of neurons can build holographic modules and form
> >>>>> holographic modules of modules.
> >>>> One mystery about this is why doesn't every brain immediately
> >>>> collapse in a giant epileptic fit. Nature must have found a
> >>>> way to detect and prevent it as the organism grows.
> >>>> Natural selection would be a poor mechanism because there are
> >>>> so many ways to fail and many fewer ways to succeed
> >>>> that almost no brains would survive.
> >>>>
> >>>> So there must be some mechanism that "drives" these self organizing
> >>>> networks toward interconnections that do not have uncontrolled feedback.
> >>>>
> >>>> I've said this before but I suspect _that_ is what nature discovered
> >>>> at the Cambrian explosion 540 million years ago.
> >>>>
> >>> I thought the Cambrian invention was teeth?
> >> Eyes developed then too, but so did legs, antenna, all complex life forms..
> >> And probably teeth too, which need muscles to work them.
> >> And all that requires a complex controller, particularly eyes.
> >>
> >> In pre-Cambrian the most complex life was things like jellyfish which
> >> are multicellular organisms and have nerves that allows them to swim,
> >> and a few neurons are specialized to detect light or dark,
> >> but no central controller, no complex signal processing or
> >> decision making capability.
> >>
> >> After the Cambrian line are arthropods and all the animals of today.
> >> Eyes developed at this time which needs complex signal processing.
> >>
> >> Clearly something changed at that boundary that allowed the assembly
> >> of complex NN to control all of these new functions.
> >
> > The opinions being expressed here would do well to refer to contemporary research. A "tooth" means something specific in the fossil record. The Cambrian Era was not a "line" or "boundary." (https://www.nature.com/articles/s41559-019-0821-6) The neural tube had already yielded to archencephalon which had yielded to telencephalon as the most complex neural circuitry long before the Cambrian Era started. Tens of millions of years before the Cambrian, dopamine was already generating sophisticated foraging behavior that is today displayed by "microorganisms, insects, mollusks, reptiles, fish, birds, and even human hunter–gatherers" (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/)." rel="nofollow" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/).
> >
> > Please consider reading this paper by an expert in the field: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/ or at least have a good look at Figure 2 from that paper: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/figure/Fig2/?report=objectonly
> Thanks, I just found the Cisek 2019 paper online this morning.
> I'll have a look at the other too.
>
> Problem is that when I started searching on this I came across
> a whole bunch of papers that are also on-topic so it is easy to
> get side tracked. E.G. I started looking at
>
> [open access]
> On the Independent Origins of Complex Brains and Neurons, 2019
> https://www.karger.com/Article/FullText/258665
>
> The problem is finding research that deals specifically with the
> development neural interconnect and how it was able to scale out,
> as opposed to say the evolution of different neural transmitters.
> > Then maybe we should start a separate thread?
> Seems on-topic to me (certainly more so than some of the other
> threads discussions on the motivations of deities).
>
> Understanding the origin of the wiring of biological NN (BNN)
> is appropriate to discussion of NN Accelerators as we are
> endeavoring to improve such simulators.
<
It is pretty clear that NNs are "pattern matchers" where one does not
necessarily know the pattern a-priori.
<
The still open question is what kind of circuitry/algorithm is appropriate
to match the patterns one has never even dreamed up ??
<
>
> Hardware architectures like Tensorflow are likely not appropriate
> for recurrent spiking NN (RSNN) and some researchers have been
> exploring new hardware accelerators in this area.

MitchAlsup wrote:
> On Sunday, November 14, 2021 at 10:46:40 AM UTC-6, EricP wrote:
>> Scott Smader wrote:
>>> On Sunday, November 14, 2021 at 6:18:51 AM UTC-8, EricP wrote:
>>>> Ivan Godard wrote:
>>>>> On 11/13/2021 10:54 AM, EricP wrote:
>>>>>> EricP wrote:
>>>>>>> Real NN have feedback, called recurrent, and real neurons are spiky
>>>>>>> which introduces signal timing and phase delays as attributes.
>>>>>>>
>>>>>>> In particular signal phase timing adds a whole new dimension for
>>>>>>> information storage. Feedback allows resonances to enhance or
>>>>>>> suppress different combinations of inputs.
>>>>>>>
>>>>>>> That is the basis for my suspicion that we will eventually find
>>>>>>> that brains, all brains, are akin to _holograms_.
>>>>>>>
>>>>>>> Clusters of neurons can build holographic modules and form
>>>>>>> holographic modules of modules.
>>>>>> One mystery about this is why doesn't every brain immediately
>>>>>> collapse in a giant epileptic fit. Nature must have found a
>>>>>> way to detect and prevent it as the organism grows.
>>>>>> Natural selection would be a poor mechanism because there are
>>>>>> so many ways to fail and many fewer ways to succeed
>>>>>> that almost no brains would survive.
>>>>>>
>>>>>> So there must be some mechanism that "drives" these self organizing
>>>>>> networks toward interconnections that do not have uncontrolled feedback.
>>>>>>
>>>>>> I've said this before but I suspect _that_ is what nature discovered
>>>>>> at the Cambrian explosion 540 million years ago.
>>>>>>
>>>>> I thought the Cambrian invention was teeth?
>>>> Eyes developed then too, but so did legs, antenna, all complex life forms..
>>>> And probably teeth too, which need muscles to work them.
>>>> And all that requires a complex controller, particularly eyes.
>>>>
>>>> In pre-Cambrian the most complex life was things like jellyfish which
>>>> are multicellular organisms and have nerves that allows them to swim,
>>>> and a few neurons are specialized to detect light or dark,
>>>> but no central controller, no complex signal processing or
>>>> decision making capability.
>>>>
>>>> After the Cambrian line are arthropods and all the animals of today.
>>>> Eyes developed at this time which needs complex signal processing.
>>>>
>>>> Clearly something changed at that boundary that allowed the assembly
>>>> of complex NN to control all of these new functions.
>>>
>>> The opinions being expressed here would do well to refer to contemporary research. A "tooth" means something specific in the fossil record. The Cambrian Era was not a "line" or "boundary." (https://www.nature.com/articles/s41559-019-0821-6) The neural tube had already yielded to archencephalon which had yielded to telencephalon as the most complex neural circuitry long before the Cambrian Era started. Tens of millions of years before the Cambrian, dopamine was already generating sophisticated foraging behavior that is today displayed by "microorganisms, insects, mollusks, reptiles, fish, birds, and even human hunter–gatherers" (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/)." rel="nofollow" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/).
>>>
>>> Please consider reading this paper by an expert in the field: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/ or at least have a good look at Figure 2 from that paper: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/figure/Fig2/?report=objectonly
>> Thanks, I just found the Cisek 2019 paper online this morning.
>> I'll have a look at the other too.
>>
>> Problem is that when I started searching on this I came across
>> a whole bunch of papers that are also on-topic so it is easy to
>> get side tracked. E.G. I started looking at
>>
>> [open access]
>> On the Independent Origins of Complex Brains and Neurons, 2019
>> https://www.karger.com/Article/FullText/258665
>>
>> The problem is finding research that deals specifically with the
>> development neural interconnect and how it was able to scale out,
>> as opposed to say the evolution of different neural transmitters.
>>> Then maybe we should start a separate thread?
>> Seems on-topic to me (certainly more so than some of the other
>> threads discussions on the motivations of deities).
>>
>> Understanding the origin of the wiring of biological NN (BNN)
>> is appropriate to discussion of NN Accelerators as we are
>> endeavoring to improve such simulators.
> <
> It is pretty clear that NNs are "pattern matchers" where one does not
> necessarily know the pattern a-priori.
> <
> The still open question is what kind of circuitry/algorithm is appropriate
> to match the patterns one has never even dreamed up ??

That is the pattern of "I have never seen this pattern before! I wonder
why?" which is the basis for most new research & inventions, right?

Terje

--
- <Terje.Mathisen at tmsw.no>
"almost all programming can be viewed as an exercise in caching"