On 11/14/2021 12:12 PM, MitchAlsup wrote:

snip

> It is pretty clear that NNs are "pattern matchers" where one does not
> necessarily know the pattern a-priori.
> <
> The still open question is what kind of circuitry/algorithm is appropriate
> to match the patterns one has never even dreamed up ??

There are algorithms that extract some pattern from whatever data is
presented to it. They generally fall into the area of "unsupervised
learning". Whether the pattern extracted is of value is, of course, a
totally different question. :-)

But these are probably not appropriate for mass implementation in
circuitry, as they don't have as widespread use as matching a previously
known pattern.

--
- Stephen Fuld
(e-mail address disguised to prevent spam)

EricP wrote:
> Scott Smader wrote:
>> On Sunday, November 14, 2021 at 6:18:51 AM UTC-8, EricP wrote:
>>> Ivan Godard wrote:
>>>> On 11/13/2021 10:54 AM, EricP wrote:
>>>>> EricP wrote:
>>>>>> Real NN have feedback, called recurrent, and real neurons are
>>>>>> spiky which introduces signal timing and phase delays as attributes.
>>>>>> In particular signal phase timing adds a whole new dimension for
>>>>>> information storage. Feedback allows resonances to enhance or
>>>>>> suppress different combinations of inputs.
>>>>>> That is the basis for my suspicion that we will eventually find
>>>>>> that brains, all brains, are akin to _holograms_.
>>>>>> Clusters of neurons can build holographic modules and form
>>>>>> holographic modules of modules.
>>>>> One mystery about this is why doesn't every brain immediately
>>>>> collapse in a giant epileptic fit. Nature must have found a way to
>>>>> detect and prevent it as the organism grows. Natural selection
>>>>> would be a poor mechanism because there are so many ways to fail
>>>>> and many fewer ways to succeed that almost no brains would survive.
>>>>> So there must be some mechanism that "drives" these self organizing
>>>>> networks toward interconnections that do not have uncontrolled
>>>>> feedback.
>>>>> I've said this before but I suspect _that_ is what nature
>>>>> discovered at the Cambrian explosion 540 million years ago.
>>>> I thought the Cambrian invention was teeth?
>>> Eyes developed then too, but so did legs, antenna, all complex life
>>> forms.. And probably teeth too, which need muscles to work them. And
>>> all that requires a complex controller, particularly eyes.
>>> In pre-Cambrian the most complex life was things like jellyfish which
>>> are multicellular organisms and have nerves that allows them to swim,
>>> and a few neurons are specialized to detect light or dark, but no
>>> central controller, no complex signal processing or decision making
>>> capability.
>>> After the Cambrian line are arthropods and all the animals of today.
>>> Eyes developed at this time which needs complex signal processing.
>>> Clearly something changed at that boundary that allowed the assembly
>>> of complex NN to control all of these new functions.
>>
>> The opinions being expressed here would do well to refer to
>> contemporary research. A "tooth" means something specific in the
>> fossil record. The Cambrian Era was not a "line" or "boundary."
>> (https://www.nature.com/articles/s41559-019-0821-6) The neural tube
>> had already yielded to archencephalon which had yielded to
>> telencephalon as the most complex neural circuitry long before the
>> Cambrian Era started. Tens of millions of years before the Cambrian,
>> dopamine was already generating sophisticated foraging behavior that
>> is today displayed by "microorganisms, insects, mollusks, reptiles,
>> fish, birds, and even human hunter–gatherers"
>> (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/)." rel="nofollow" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/).
>>
>> Please consider reading this paper by an expert in the field:
>> https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/ or at least have
>> a good look at Figure 2 from that paper:
>> https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6848052/figure/Fig2/?report=objectonly
>>
>
> Thanks, I just found the Cisek 2019 paper online this morning.
> I'll have a look at the other too.

Ok I've read them quickly.
Yes there are some burrowing animals with a central nervous system
in the Ediacaran but both papers acknowledge the development of
limbs and vision across the Ediacaran–Cambrian boundary.

Burrowing doesn't strike me as complicated behavior,
perhaps just a variation on the jellyfish nerve system
and doesn't require any signal processing capability.

And jellyfish today have single neuron light sensors.
Again it doesn't require any signal processing capability to move
towards light, move away from dark, eat whatever you bump into.

I saw nothing in those to contradict the idea that something important
happened to allow the construction far more complex NN around the
Ediacaran–Cambrian boundary as an enabling technology to all the
other changes.

> Problem is that when I started searching on this I came across
> a whole bunch of papers that are also on-topic so it is easy to
> get side tracked. E.G. I started looking at
>
> [open access]
> On the Independent Origins of Complex Brains and Neurons, 2019
> https://www.karger.com/Article/FullText/258665
>
> The problem is finding research that deals specifically with the
> development neural interconnect and how it was able to scale out,
> as opposed to say the evolution of different neural transmitters.

I came across this paper later.
It is more on target for what I'm thinking of as it addresses
the change from simple nerve nets to centralized brains.

[open access]
Of Circuits and Brains: The Origin and Diversification
of Neural Architectures, 2020
https://www.frontiersin.org/articles/10.3389/fevo.2020.00082/full

In the above papers' section "How Do These Neural Circuits Evolve?"
it references to the one below which I haven't read yet but also
looks relevant (see what I mean about getting side tracked) as it
has a section titled "2.1. Evolution of connectivity":

[open access]
Developmental and genetic mechanisms of neural circuit evolution, 2017
https://www.sciencedirect.com/science/article/pii/S0012160617301495

But I don't see anything yet in either one that addresses how
they construct NN correctly (non-epileptic).

Terje Mathisen <terje.mathisen@tmsw.no> schrieb:

> That is the pattern of "I have never seen this pattern before! I wonder
> why?" which is the basis for most new research & inventions, right?

My personal experience is that a lot of inventions is due to
transfer, recognizing a pattern in another set of circumstances
than the one you originally saw them, and applying it.

I've read recently that there is a high correlation between the
inventiveness of people and their tendency to make bad jokes.
The latter requires taking things out of their usual context, just
as invention des.

On Sunday, November 14, 2021 at 4:29:05 PM UTC-6, Thomas Koenig wrote:
> Terje Mathisen <terje.m...@tmsw.no> schrieb:
> > That is the pattern of "I have never seen this pattern before! I wonder
> > why?" which is the basis for most new research & inventions, right?
<
> My personal experience is that a lot of inventions is due to
> transfer, recognizing a pattern in another set of circumstances
> than the one you originally saw them, and applying it.
>
> I've read recently that there is a high correlation between the
> inventiveness of people and their tendency to make bad jokes.
> The latter requires taking things out of their usual context, just
> as invention des.
<
Guilty as charged............

On Sunday, November 14, 2021 at 1:56:27 PM UTC-8, EricP wrote:
> EricP wrote:

> Ok I've read them quickly.
> Yes there are some burrowing animals with a central nervous system
> in the Ediacaran but both papers acknowledge the development of
> limbs and vision across the Ediacaran–Cambrian boundary.
>
> Burrowing doesn't strike me as complicated behavior,
> perhaps just a variation on the jellyfish nerve system
> and doesn't require any signal processing capability.
>
> And jellyfish today have single neuron light sensors.
> Again it doesn't require any signal processing capability to move
> towards light, move away from dark, eat whatever you bump into.
>
> I saw nothing in those to contradict the idea that something important
> happened to allow the construction far more complex NN around the
> Ediacaran–Cambrian boundary as an enabling technology to all the
> other changes.

> > Problem is that when I started searching on this I came across
> > a whole bunch of papers that are also on-topic so it is easy to
> > get side tracked. E.G. I started looking at
> >
> > [open access]
> > On the Independent Origins of Complex Brains and Neurons, 2019
> > https://www.karger.com/Article/FullText/258665
> >
> > The problem is finding research that deals specifically with the
> > development neural interconnect and how it was able to scale out,
> > as opposed to say the evolution of different neural transmitters.
> I came across this paper later.
> It is more on target for what I'm thinking of as it addresses
> the change from simple nerve nets to centralized brains.
>
> [open access]
> Of Circuits and Brains: The Origin and Diversification
> of Neural Architectures, 2020
> https://www.frontiersin.org/articles/10.3389/fevo.2020.00082/full
>
> In the above papers' section "How Do These Neural Circuits Evolve?"
> it references to the one below which I haven't read yet but also
> looks relevant (see what I mean about getting side tracked) as it
> has a section titled "2.1. Evolution of connectivity":
>
> [open access]
> Developmental and genetic mechanisms of neural circuit evolution, 2017
> https://www.sciencedirect.com/science/article/pii/S0012160617301495
>
> But I don't see anything yet in either one that addresses how
> they construct NN correctly (non-epileptic).

Thank you for those references. I look forward to reading them.

I apologize for leading you to believe that the Cisek and Wood, et al, papers denied increasing neural complexity during the Cambrian Era or explained how epileptic synchrony was prevented. I sought simply to point out that sophisticated behavior and teeth did not "explode" in at a moment in time.

At the risk of burdening you with more possibly unfruitful reading, may I recommend the work of Ramin Hasani at MIT with others from various institutions? His group have built robotic control systems based ultimately on the careful measurements they had made previously on the 302 neurons and 5000 synapses of C. elegans. I don't think they have published specifically about the epileptic problem, but their work seems like it might at least be another interesting tangent for you. They do show that the hidden state of every neuron in one of their networks is bounded, but I doubt that that directly relates to epileptic avoidance.

Liquid Time Constant Networks https://arxiv.org/abs/2006.04439
This paper describes their LTC networks in some detail and compares performance against LSTM, CT-RNN, Neural ODE and CT-GRU.

Hasani also gave this talk in March about: Liquid Time Constant Networks https://simons.berkeley.edu/talks/tbd-296
Images and sequences shown starting at about 31 minutes are quite impressive, especially considering the much smaller number of neurons required and the robustness against noise. He also mentions cascading LTCs into what he calls Neural Circuit Policies which he then shows are Dynamic Causal Models.

I have only skimmed most of his publications: http://www.raminhasani.com/publications/
which document his journey from worms to LTCNs, but perhaps it may help you sort out whether the epileptic avoidance solution is earlier or later than the C elegans brain!

Best wishes!

> I've read recently that there is a high correlation between the
> inventiveness of people and their tendency to make bad jokes.

Sadly, it's only a correlation.
I think I'm pretty good(?) at making bad jokes, but it doesn't seem to
carry over to inventiveness,

Stefan

On 11/14/2021 1:55 PM, EricP wrote:
> EricP wrote:

<snip>

> But I don't see anything yet in either one that addresses how
> they construct NN correctly (non-epileptic).

A feedback system can go into oscillation (i.e. epilepsy) and get eaten.
Or it can simply damp down and get eaten. Over evolutionary time it will
hunt to the chaotic boundary; that's us.

Thomas Koenig wrote:
> Terje Mathisen <terje.mathisen@tmsw.no> schrieb:
>
>> That is the pattern of "I have never seen this pattern before! I wonder
>> why?" which is the basis for most new research & inventions, right?
>
> My personal experience is that a lot of inventions is due to
> transfer, recognizing a pattern in another set of circumstances
> than the one you originally saw them, and applying it.
>
> I've read recently that there is a high correlation between the
> inventiveness of people and their tendency to make bad jokes.
> The latter requires taking things out of their usual context, just
> as invention des.

I thought that was pretty well established?

Punning is a well-known form of this, some people supposedly hate puns,
I tend to love them, the more groan-inducing the better.

Terje

--
- <Terje.Mathisen at tmsw.no>
"almost all programming can be viewed as an exercise in caching"

MitchAlsup wrote:
> On Sunday, November 14, 2021 at 10:46:40 AM UTC-6, EricP wrote:
>>
>> Understanding the origin of the wiring of biological NN (BNN)
>> is appropriate to discussion of NN Accelerators as we are
>> endeavoring to improve such simulators.
> <
> It is pretty clear that NNs are "pattern matchers" where one does not
> necessarily know the pattern a-priori.
> <
> The still open question is what kind of circuitry/algorithm is appropriate
> to match the patterns one has never even dreamed up ??

The artificial convolution NN are basically fancy curve fit algorithms
that adjust a polynomial with tens or hundreds of thousands of terms
to some number of inputs after millions of examples.

Biological NN perform associative learning after just a few examples
with just a few neurons.

Both are suitable for sorting fish but only one
can fit inside and control a fruit fly.

Scott Smader wrote:
> On Sunday, November 14, 2021 at 1:56:27 PM UTC-8, EricP wrote:
>> EricP wrote:
>
>> Ok I've read them quickly.
>> Yes there are some burrowing animals with a central nervous system
>> in the Ediacaran but both papers acknowledge the development of
>> limbs and vision across the Ediacaran–Cambrian boundary.
>>
>> Burrowing doesn't strike me as complicated behavior,
>> perhaps just a variation on the jellyfish nerve system
>> and doesn't require any signal processing capability.
>>
>> And jellyfish today have single neuron light sensors.
>> Again it doesn't require any signal processing capability to move
>> towards light, move away from dark, eat whatever you bump into.
>>
>> I saw nothing in those to contradict the idea that something important
>> happened to allow the construction far more complex NN around the
>> Ediacaran–Cambrian boundary as an enabling technology to all the
>> other changes.
>
>
>>> Problem is that when I started searching on this I came across
>>> a whole bunch of papers that are also on-topic so it is easy to
>>> get side tracked. E.G. I started looking at
>>>
>>> [open access]
>>> On the Independent Origins of Complex Brains and Neurons, 2019
>>> https://www.karger.com/Article/FullText/258665
>>>
>>> The problem is finding research that deals specifically with the
>>> development neural interconnect and how it was able to scale out,
>>> as opposed to say the evolution of different neural transmitters.
>> I came across this paper later.
>> It is more on target for what I'm thinking of as it addresses
>> the change from simple nerve nets to centralized brains.
>>
>> [open access]
>> Of Circuits and Brains: The Origin and Diversification
>> of Neural Architectures, 2020
>> https://www.frontiersin.org/articles/10.3389/fevo.2020.00082/full
>>
>> In the above papers' section "How Do These Neural Circuits Evolve?"
>> it references to the one below which I haven't read yet but also
>> looks relevant (see what I mean about getting side tracked) as it
>> has a section titled "2.1. Evolution of connectivity":
>>
>> [open access]
>> Developmental and genetic mechanisms of neural circuit evolution, 2017
>> https://www.sciencedirect.com/science/article/pii/S0012160617301495
>>
>> But I don't see anything yet in either one that addresses how
>> they construct NN correctly (non-epileptic).
>
> Thank you for those references. I look forward to reading them.
>
> I apologize for leading you to believe that the Cisek and Wood, et al, papers denied increasing neural complexity during the Cambrian Era or explained how epileptic synchrony was prevented. I sought simply to point out that sophisticated behavior and teeth did not "explode" in at a moment in time.

Not at all, they were both interesting. And I have seen others
questioning when the Cambrian Explosion started and what caused it.

> At the risk of burdening you with more possibly unfruitful reading, may I recommend the work of Ramin Hasani at MIT with others from various institutions? His group have built robotic control systems based ultimately on the careful measurements they had made previously on the 302 neurons and 5000 synapses of C. elegans. I don't think they have published specifically about the epileptic problem, but their work seems like it might at least be another interesting tangent for you. They do show that the hidden state of every neuron in one of their networks is bounded, but I doubt that that directly relates to epileptic avoidance.

Yes, this is the track I'm thinking of.
He uses the word "liquid" to mean time-varying.

Designing Worm-inspired Neural Networks for
Interpretable Robotic Control, 2019
https://publik.tuwien.ac.at/files/publik_287624.pdf

"In this paper, we design novel liquid time-constant recurrent neural
networks for robotic control, inspired by the brain of the nematode,
C. elegans. In the worm’s nervous system, neurons communicate through
nonlinear time-varying synaptic links established amongst them by their
particular wiring structure. This property enables neurons to express
liquid time-constants dynamics and therefore allows the network to
originate complex behaviors with a small number of neurons."
....
"We evaluate their performance in controlling mobile and arm robots"
....
"The C. elegans nematode, with a rather simple nervous system composed
of 302 neurons and 8000 synapses, exhibits remarkable controllability
in it’s surroundings; it expresses behaviors such as processing complex
chemical input stimulations, sleeping, realizing adaptive behavior,
performing mechano-sensation, and controlling 96 muscles.
How does C. elegans perform so much with so little?"

> Liquid Time Constant Networks https://arxiv.org/abs/2006.04439
> This paper describes their LTC networks in some detail and compares performance against LSTM, CT-RNN, Neural ODE and CT-GRU.

This sounds like what I was trying to speculate earlier, that information
in the NN is encoded not only in the location of connections and
in their weights, but also in the phase delay of the signal arrivals.

An analogy would be an asynchronous logic circuit with feedback pathways
where propagation delay on the interconnect wire encodes part of the
signal processing logic.

I see they say the networks are stable and bounded but its not clear
to me yet why they are. I've searched for terms like "meta stability".

> Hasani also gave this talk in March about: Liquid Time Constant Networks https://simons.berkeley.edu/talks/tbd-296
> Images and sequences shown starting at about 31 minutes are quite impressive, especially considering the much smaller number of neurons required and the robustness against noise. He also mentions cascading LTCs into what he calls Neural Circuit Policies which he then shows are Dynamic Causal Models.
>
> I have only skimmed most of his publications: http://www.raminhasani.com/publications/
> which document his journey from worms to LTCNs, but perhaps it may help you sort out whether the epileptic avoidance solution is earlier or later than the C elegans brain!
>
> Best wishes!

Thanks for the pointers.

On 11/15/2021 8:18 AM, EricP wrote:

snip

> This sounds like what I was trying to speculate earlier, that information
> in the NN is encoded not only in the location of connections and
> in their weights, but also in the phase delay of the signal arrivals.

The weights in Artificial Neural Networks are a stand in for the timing
of real neurons. Weighted amount of firing doesn't really exist in real
neurons.

Real neurons are, of course, analogue. When the pre-synaptic neuron
fires, it send a relatively fixed amount of a neurotransmitter into the
synapse. This causes the receptors in the post synaptic neuron to open
a channel that allows ions into the cell. The cell has ion pumps that
continually try to maintain the potential across the cell membrane, so
over time, the effect of the depolarization from the synapse is
dissipated. But if enough synaptic receptors open before the ion pumps
can cope with the effects, the whole neuron depolarizes (i.e. "fires").

But the amount of depolarization from any one synapse is relatively
fixed. It is the number and timing of the firings of the presynaptic
neurons, not their "weight" that determines when the post synaptic
neuron fires. And, of course there is no "clock". ANNs use weights and
defined update times to simulate this.

--
- Stephen Fuld
(e-mail address disguised to prevent spam)

On Monday, November 15, 2021 at 8:18:50 AM UTC-8, EricP wrote:

> Thanks for the pointers.
Well, quoting Bobcat Goldthwait, "Thank you for encouraging my behavior."

> This sounds like what I was trying to speculate earlier, that information
> in the NN is encoded not only in the location of connections and
> in their weights, but also in the phase delay of the signal arrivals.
>
> An analogy would be an asynchronous logic circuit with feedback pathways
> where propagation delay on the interconnect wire encodes part of the
> signal processing logic.

That is very much in haromony with the thinking in this paper which documents the use of varying myelination of axons to produce precise timing in birdsong.
Local axonal conduction delays underlie precise timing of a neural sequence
https://www.biorxiv.org/content/10.1101/864231v1

This 2009 paper directly states your proposition: "[T]he visual detection threshold fluctuates over time along with the phase of ongoing EEG activity. The results support the notion that ongoing oscillations shape our perception, possibly by providing a temporal reference frame for neural codes that rely on precise spike timing."
The Phase of Ongoing EEG Oscillations Predicts Visual Perception
https://www.jneurosci.org/content/29/24/7869

And the criticality of phase-related information is also suggested by this:
Intracranial recordings reveal ubiquitous in-phase and in- antiphase functional connectivity between homologous brain regions in humans
https://www.biorxiv.org/content/10.1101/2020.06.19.162065v2

Possibly related, this paper claims that brain signaling is divided into frequency bands:
Causal evidence of network communication in whole-brain dynamics through a multiplexed neural code
https://doi.org/10.1101/2020.06.09.142695
I don't believe the paper addresses this, but multiple bands could be used simultaneously for individual phase-synchronization signals in separated functional networks.

In line with Ivan's insightful comment about evolution optimizing control systems to the edge of chaos, it also makes sense that given enough time, evolution would find an (approximate) implementation of almost every possible signal processing technique.

And maybe even back-propagation, too, as speculated in this very recent paper about some simulations they did:
Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits
https://www.nature.com/articles/s41593-021-00857-x
I'm too cheap to buy the article, but this Wired article describes it:
https://www.wired.com/story/neuron-bursts-can-mimic-a-famous-ai-learning-strategy/
Numenta forum has a discussion about it: https://discourse.numenta.org/t/burst-as-a-local-learning-rule-in-apical-but-not-basal-dendrites/9093

Fun stuff!

Stephen Fuld wrote:
> On 11/15/2021 8:18 AM, EricP wrote:
>
> snip
>
>> This sounds like what I was trying to speculate earlier, that information
>> in the NN is encoded not only in the location of connections and
>> in their weights, but also in the phase delay of the signal arrivals.
>
> The weights in Artificial Neural Networks are a stand in for the timing
> of real neurons. Weighted amount of firing doesn't really exist in real
> neurons.
>
> Real neurons are, of course, analogue. When the pre-synaptic neuron
> fires, it send a relatively fixed amount of a neurotransmitter into the
> synapse. This causes the receptors in the post synaptic neuron to open
> a channel that allows ions into the cell. The cell has ion pumps that
> continually try to maintain the potential across the cell membrane, so
> over time, the effect of the depolarization from the synapse is
> dissipated. But if enough synaptic receptors open before the ion pumps
> can cope with the effects, the whole neuron depolarizes (i.e. "fires").
>
> But the amount of depolarization from any one synapse is relatively
> fixed. It is the number and timing of the firings of the presynaptic
> neurons, not their "weight" that determines when the post synaptic
> neuron fires. And, of course there is no "clock". ANNs use weights and
> defined update times to simulate this.

I don't think this is correct, or maybe we are thinking of
different types of neurons.

It is my understanding that after a neuron fires the action potential
is constant down the axon (no-fire or fire), but there are different
numbers of synaptic vesicles that release neurotransmitters and receptors
to receive them and this controls the strength of individual connections.
The number of vesicles and/or receptors are adjusted over time
to increase or decrease the individual connection weights.

Adjusting the number of vesicles or receptors and thereby the weights of
connections is thought to be one of the mechanisms for long term memory,
but also I was told to addiction to some drugs like cocaine
and possibly how LSD can cause flashbacks.

https://en.wikipedia.org/wiki/Chemical_synapse#Synaptic_strength

https://en.wikipedia.org/wiki/Synaptic_plasticity

https://en.wikipedia.org/wiki/Long-term_potentiation

Also in real neurons some connections are excitatory and others inhibitory
but that can be modeled by using signed weights.

One thing that the classic artificial "sum of multiplied weights" neuron
can't model is an XOR gate - it can only do AND and OR.

On Monday, November 15, 2021 at 10:39:01 AM UTC-8, EricP wrote:
> Stephen Fuld wrote:
> > On 11/15/2021 8:18 AM, EricP wrote:
> >
> > snip
> >
> >> This sounds like what I was trying to speculate earlier, that information
> >> in the NN is encoded not only in the location of connections and
> >> in their weights, but also in the phase delay of the signal arrivals.
> >
> > The weights in Artificial Neural Networks are a stand in for the timing
> > of real neurons. Weighted amount of firing doesn't really exist in real
> > neurons.
> >
> > Real neurons are, of course, analogue. When the pre-synaptic neuron
> > fires, it send a relatively fixed amount of a neurotransmitter into the
> > synapse. This causes the receptors in the post synaptic neuron to open
> > a channel that allows ions into the cell. The cell has ion pumps that
> > continually try to maintain the potential across the cell membrane, so
> > over time, the effect of the depolarization from the synapse is
> > dissipated. But if enough synaptic receptors open before the ion pumps
> > can cope with the effects, the whole neuron depolarizes (i.e. "fires").
> >
> > But the amount of depolarization from any one synapse is relatively
> > fixed. It is the number and timing of the firings of the presynaptic
> > neurons, not their "weight" that determines when the post synaptic
> > neuron fires. And, of course there is no "clock". ANNs use weights and
> > defined update times to simulate this.
> I don't think this is correct, or maybe we are thinking of
> different types of neurons.
>
> It is my understanding that after a neuron fires the action potential
> is constant down the axon (no-fire or fire), but there are different
> numbers of synaptic vesicles that release neurotransmitters and receptors
> to receive them and this controls the strength of individual connections.
> The number of vesicles and/or receptors are adjusted over time
> to increase or decrease the individual connection weights.
>
> Adjusting the number of vesicles or receptors and thereby the weights of
> connections is thought to be one of the mechanisms for long term memory,
> but also I was told to addiction to some drugs like cocaine
> and possibly how LSD can cause flashbacks.
>
> https://en.wikipedia.org/wiki/Chemical_synapse#Synaptic_strength
>
> https://en.wikipedia.org/wiki/Synaptic_plasticity
>
> https://en.wikipedia.org/wiki/Long-term_potentiation
>
> Also in real neurons some connections are excitatory and others inhibitory
> but that can be modeled by using signed weights.
>

It's not just the number of pre-synaptic neurons; there are typically multiple synapses between connected neurons, and varying numbers of vesicles on synapses.
Pretty video from Sebastian Seung's lab in 2013:
How to map neurons in 3D
https://www.youtube.com/watch?v=_iKrE2A2Vx4
The red and green neurons can be seen contacting at two separated areas.

> One thing that the classic artificial "sum of multiplied weights" neuron
> can't model is an XOR gate - it can only do AND and OR.

Um, that was true for the original Perceptron, but XOR is a long-solved problem for networks. If you've got AND, OR, and NOT with multiple levels, you can do anything!

On Monday, November 15, 2021 at 10:39:01 AM UTC-8, EricP wrote:
> Stephen Fuld wrote:
> > On 11/15/2021 8:18 AM, EricP wrote:
> >
> > snip
> >
> >> This sounds like what I was trying to speculate earlier, that information
> >> in the NN is encoded not only in the location of connections and
> >> in their weights, but also in the phase delay of the signal arrivals.
> >
> > The weights in Artificial Neural Networks are a stand in for the timing
> > of real neurons. Weighted amount of firing doesn't really exist in real
> > neurons.
> >
> > Real neurons are, of course, analogue. When the pre-synaptic neuron
> > fires, it send a relatively fixed amount of a neurotransmitter into the
> > synapse. This causes the receptors in the post synaptic neuron to open
> > a channel that allows ions into the cell. The cell has ion pumps that
> > continually try to maintain the potential across the cell membrane, so
> > over time, the effect of the depolarization from the synapse is
> > dissipated. But if enough synaptic receptors open before the ion pumps
> > can cope with the effects, the whole neuron depolarizes (i.e. "fires").
> >
> > But the amount of depolarization from any one synapse is relatively
> > fixed. It is the number and timing of the firings of the presynaptic
> > neurons, not their "weight" that determines when the post synaptic
> > neuron fires. And, of course there is no "clock". ANNs use weights and
> > defined update times to simulate this.
> I don't think this is correct, or maybe we are thinking of
> different types of neurons.
>
> It is my understanding that after a neuron fires the action potential
> is constant down the axon (no-fire or fire), but there are different
> numbers of synaptic vesicles that release neurotransmitters and receptors
> to receive them and this controls the strength of individual connections.
> The number of vesicles and/or receptors are adjusted over time
> to increase or decrease the individual connection weights.
>
> Adjusting the number of vesicles or receptors and thereby the weights of
> connections is thought to be one of the mechanisms for long term memory,
> but also I was told to addiction to some drugs like cocaine
> and possibly how LSD can cause flashbacks.
>
> https://en.wikipedia.org/wiki/Chemical_synapse#Synaptic_strength
>
> https://en.wikipedia.org/wiki/Synaptic_plasticity
>
> https://en.wikipedia.org/wiki/Long-term_potentiation
>
> Also in real neurons some connections are excitatory and others inhibitory
> but that can be modeled by using signed weights.
>
t's not just the number of pre-synaptic neurons; there are typically multiple synapses between connected neurons, and varying numbers of vesicles on synapses.
Pretty video from Sebastian Seung's lab in 2013:
How to map neurons in 3D
https://www.youtube.com/watch?v=_iKrE2A2Vx4
The red and green neurons can be seen contacting at two separated areas.

> One thing that the classic artificial "sum of multiplied weights" neuron
> can't model is an XOR gate - it can only do AND and OR.

Um, that was true for the original Perceptron, but XOR is a long-solved problem for networks. If you've got AND and NOT, or OR and NOT, with multiple levels, you can do anything!

On 11/15/2021 10:37 AM, EricP wrote:
> Stephen Fuld wrote:
>> On 11/15/2021 8:18 AM, EricP wrote:
>>
>> snip
>>
>>> This sounds like what I was trying to speculate earlier, that
>>> information
>>> in the NN is encoded not only in the location of connections and
>>> in their weights, but also in the phase delay of the signal arrivals.
>>
>> The weights in Artificial Neural Networks are a stand in for the
>> timing of real neurons. Weighted amount of firing doesn't really exist
>> in real neurons.
>>
>> Real neurons are, of course, analogue.  When the pre-synaptic neuron
>> fires, it send a relatively fixed amount of a neurotransmitter into
>> the synapse.  This causes the receptors in the post synaptic neuron to
>> open a channel that allows ions into the cell.  The cell has ion pumps
>> that continually try to maintain the potential across the cell
>> membrane, so over time, the effect of the depolarization from the
>> synapse is dissipated.  But if enough synaptic receptors open before
>> the ion pumps can cope with the effects, the whole neuron depolarizes
>> (i.e. "fires").
>>
>> But the amount of depolarization from any one synapse is relatively
>> fixed.  It is the number and timing of the firings of the presynaptic
>> neurons, not their "weight" that determines when the post synaptic
>> neuron fires.  And, of course there is no "clock".  ANNs use weights
>> and defined update times to simulate this.
>
> I don't think this is correct, or maybe we are thinking of
> different types of neurons.

>
> It is my understanding that after a neuron fires the action potential
> is constant down the axon (no-fire or fire), but there are different
> numbers of synaptic vesicles that release neurotransmitters and receptors
> to receive them and this controls the strength of individual connections.
> The number of vesicles and/or receptors are adjusted over time
> to increase or decrease the individual connection weights.

You are right, of course. I obviously had a malfunction in my neural
network. :-( I apologize.

snip

> One thing that the classic artificial "sum of multiplied weights" neuron
> can't model is an XOR gate - it can only do AND and OR.

That was a failure of the original perceptron. Minsky and Papert showed
this (for any number of layers), and that was what led to the "dark
winter" of NN research. The realization that non-linear activation
functions could get around this problem is what led to their "renascence".

--
- Stephen Fuld
(e-mail address disguised to prevent spam)

On Tuesday, November 16, 2021 at 5:02:39 PM UTC-6, Stephen Fuld wrote:
> On 11/15/2021 10:37 AM, EricP wrote:
> > Stephen Fuld wrote:
> >> On 11/15/2021 8:18 AM, EricP wrote:
> >>
> >> snip
> >>
> >>> This sounds like what I was trying to speculate earlier, that
> >>> information
> >>> in the NN is encoded not only in the location of connections and
> >>> in their weights, but also in the phase delay of the signal arrivals.
> >>
> >> The weights in Artificial Neural Networks are a stand in for the
> >> timing of real neurons. Weighted amount of firing doesn't really exist
> >> in real neurons.
> >>
> >> Real neurons are, of course, analogue. When the pre-synaptic neuron
> >> fires, it send a relatively fixed amount of a neurotransmitter into
> >> the synapse. This causes the receptors in the post synaptic neuron to
> >> open a channel that allows ions into the cell. The cell has ion pumps
> >> that continually try to maintain the potential across the cell
> >> membrane, so over time, the effect of the depolarization from the
> >> synapse is dissipated. But if enough synaptic receptors open before
> >> the ion pumps can cope with the effects, the whole neuron depolarizes
> >> (i.e. "fires").
> >>
> >> But the amount of depolarization from any one synapse is relatively
> >> fixed. It is the number and timing of the firings of the presynaptic
> >> neurons, not their "weight" that determines when the post synaptic
> >> neuron fires. And, of course there is no "clock". ANNs use weights
> >> and defined update times to simulate this.
> >
> > I don't think this is correct, or maybe we are thinking of
> > different types of neurons.
>
>
> >
> > It is my understanding that after a neuron fires the action potential
> > is constant down the axon (no-fire or fire), but there are different
> > numbers of synaptic vesicles that release neurotransmitters and receptors
> > to receive them and this controls the strength of individual connections.
> > The number of vesicles and/or receptors are adjusted over time
> > to increase or decrease the individual connection weights.
> You are right, of course. I obviously had a malfunction in my neural
> network. :-( I apologize.
>
> snip
<
> > One thing that the classic artificial "sum of multiplied weights" neuron
> > can't model is an XOR gate - it can only do AND and OR.
<
> That was a failure of the original perceptron. Minsky and Papert showed
> this (for any number of layers), and that was what led to the "dark
> winter" of NN research. The realization that non-linear activation
> functions could get around this problem is what led to their "renascence"..
<
This leads to a short story about microcode.........
<
Basically, microcode is a ROM built from a PLA--a PLA is simply 2 NOT planes
back to back. There are a lot of things a PLA cannot do easily, but the addition
of a row of XOR gates between the NOR-planes significantly increases the kinds
of things a PLA can compute (sequence...)
<
But, computer NNs are built around × and +, they could just as easily be built
around × and ± ; if the weights (and/or coefficients) were signed..
<
It is just arithmetic..........
<
Getting nice signmoid functions is not that hard with look-up-tables..........
> --
> - Stephen Fuld
> (e-mail address disguised to prevent spam)

MitchAlsup wrote:
> <
> This leads to a short story about microcode.........
> <
> Basically, microcode is a ROM built from a PLA--a PLA is simply 2 NOT planes

NOR planes or course.

> back to back. There are a lot of things a PLA cannot do easily, but the addition
> of a row of XOR gates between the NOR-planes significantly increases the kinds
> of things a PLA can compute (sequence...)

The planes are both dynamic logic because CMOS doesn't allow static
wired-OR and I saw some mention that there are some design fiddly bits
ensuring the second NOR plane doesn't discharge the first plane too soon.

Hmmm... XOR's between NOR planes... interesting, I never thought of that.

> <
> But, computer NNs are built around × and +, they could just as easily be built
> around × and ± ; if the weights (and/or coefficients) were signed..
> <
> It is just arithmetic..........

That is how it is usually represented in the equations,
as the sum of a series of synapses state multiplies times the weights,
and an unsigned compare to a trigger value.

The sum and trigger have enough bit to not overflow.
For 1024 8-bit integer synapse weights the parallel adder looks
like it requires 2048 adders varying in size from 8 to 8+10 bits
producing an 18 bit total. Does that sound correct?

Note that the synapse state is 0 or 1 so the multiply is unnecessary.
I would consider replacing the above neuron mechanism with a PHI or MUX
function to select between 0 or a signed weight for each synapse based on
the 0 or 1 state, the sum done with signed arithmetic without overflow,
and a signed compare to the trigger level.

MitchAlsup <MitchAlsup@aol.com> schrieb:

> Basically, microcode is a ROM built from a PLA--a PLA is simply
> 2 NOT planes back to back. There are a lot of things a PLA cannot
> do easily, but the addition of a row of XOR gates between the
> NOR-planes significantly increases the kinds of things a PLA can
> compute (sequence...)

Sound interesting.

Do you have a reference for that, by any chance?

On Wednesday, November 17, 2021 at 11:37:45 AM UTC-6, Thomas Koenig wrote:
> MitchAlsup <Mitch...@aol.com> schrieb:
> > Basically, microcode is a ROM built from a PLA--a PLA is simply
> > 2 NOT planes back to back. There are a lot of things a PLA cannot
> > do easily, but the addition of a row of XOR gates between the
> > NOR-planes significantly increases the kinds of things a PLA can
> > compute (sequence...)
> Sound interesting.
>
> Do you have a reference for that, by any chance?
<
Probably not:: we did use this trick on the 68000, 68010 and 68020 microcode
stores.
<
Use cases: Say you have a term that is asserted by 98% of all microcode
"instructions", you can save power by only computing it on the 2% that
don't need it and then use XOR to flip the polarity.
<
Another use case: Say you have a calculation and an available function
unit and, as long as "blah" does not happen, you can use it, so you setup
microcode to assume you can do it, and then use the XORs to cancel
it on those "special" occasions when someone else consumes that function
unit.

On Wednesday, November 17, 2021 at 11:37:45 AM UTC-6, Thomas Koenig wrote:
> MitchAlsup <Mitch...@aol.com> schrieb:
> > Basically, microcode is a ROM built from a PLA--a PLA is simply
> > 2 NOT planes back to back. There are a lot of things a PLA cannot
> > do easily, but the addition of a row of XOR gates between the
> > NOR-planes significantly increases the kinds of things a PLA can
> > compute (sequence...)
> Sound interesting.
>
> Do you have a reference for that, by any chance?
wikipedia has a writeup on "CPLD" most of which have the XOR of a product term with other sum-of-product terms.
Digikey has them in stock along with data sheets
"Embedded - CPLDs (Complex Programmable Logic Devices)"

I've used the XC9536X series.
It has a schematic of the "Macrocell"
Totally obsoleted by FPGAs which are bigger, faster and lower power.

On 11/15/2021 10:15 AM, EricP wrote:
> MitchAlsup wrote:
>> On Sunday, November 14, 2021 at 10:46:40 AM UTC-6, EricP wrote:
>>>
>>> Understanding the origin of the wiring of biological NN (BNN) is
>>> appropriate to discussion of NN Accelerators as we are endeavoring to
>>> improve such simulators.
>> <
>> It is pretty clear that NNs are "pattern matchers" where one does not
>> necessarily know the pattern a-priori.
>> <
>> The still open question is what kind of circuitry/algorithm is
>> appropriate to match the patterns one has never even dreamed up ??
>
> The artificial convolution NN are basically fancy curve fit algorithms
> that adjust a polynomial with tens or hundreds of thousands of terms
> to some number of inputs after millions of examples.
>

A lot of what people are doing with NNs could be done with
auto-correlation and FIR filters.

Though, in the overly loose definitions often used, one could also try
to classify auto-correlation and FIR filters as a type of NN.

> Biological NN perform associative learning after just a few examples
> with just a few neurons.
>
> Both are suitable for sorting fish but only one
> can fit inside and control a fruit fly.
>

There is one reason I like genetic algorithms and genetic programming
for some tasks:
While the training process is itself fairly slow, and the results are
(rarely) much better than something one could come up with themselves
(in usually a fraction the time and effort), one can at least use it to
generate results that are fairly cheap to run within the constraints of
the target machine (unlike CNN's or "Deep Learning" models).

So, one can in theory set up a GP evolver to be able to use a simple set
of vector-arithmetic operators and a simplified register machine
(generally simpler register models seem to work out better, and are
simpler to implement, than a GP evolver which works in terms of ASTs).

The weighting algorithm can also impose a penalty for the number of
(Non-NOP) operations used, favoring smaller and simpler solutions (and
causing non-effective operations to tend to mutate into NOPs; which can
be removed when generating the final output).

Say, for example, for each "program":
Has between 64 and 1024 instruction words to work with;
Usually this is a fixed parameter in the tests.
Has 32 or 64 bits per instruction word;
Has 16 or 32 registers;
May or may not have control-flow operations (depending on the task);
...

An example GP-ISA design might have:
64-bit instruction words;
16 or 32 registers, encoded in a padded form (ECC style, *);
Opcode bits may or may not also have a padded encoding;
Most invalid operations are treated as NOP;
There is a way to encode things like vector loads, ...;
Most operators are 3R form, eg: "OP Rs, Rt, Rd"
...

*: Multiple encodings may map to the same logical register, and using
ECC bits makes the register ID more resistant to random bit-flips
(caused by the mutator).

So, a register may be encoded as:
4 bits (abcd): Register Number, Gray-Coded
3 bits: Parity (a^b^c, b^c^d, c^d^a)
R0: 000-0000
R1: 011-0001
R2: 100-0011
R3: 111-0010
R4: 001-0110
...

Similarly, could do a 5 bit register in 8 bits, eg:
{ a^b^c, b^c^d, c^d^e, e, a, b, c, d }

The ECC (~ Hamming(7,4)) may try to "correct" the register on decode.

Opcode may encode an 8 or 10-bit opcode number in 16 bits.

Encodings which fall into the "unrecoverable" or "disallowed" parts of
the encoding space are interpreted as NOPs.

Vector immediate values may be encoded as 48-bits, such as four 12-bit
floating-point values (S.E5.M6), which may also be stored in gray-coded
form. There might also be 2x 24-bit bit (truncated gray-coded single),
or 1x 48-bit (truncated gray-coded double).

It may also make sense to have integer operators available (depends on
the task).

....

The GP evolver basically consists of:
Test data, which is fed into the program in some form;
Say, the test data is presented as input registers.
An interpreter, which runs each GP program;
Output is one or more registers.
A heuristic to rank its performance;
...

For breeding the top-performing programs:
Pick instruction words randomly from each parent;
Randomly flip bits in each child produced.

The initial state would fill the programs with "random garbage" though,
using NOP encodings for the operators.

If one allows for control-flow, the interpreter will automatically
terminate after a certain number of instructions, and impose a fairly
severe penalty value.

Result (after a certain number of runs) would be dumped out in an ASM
style notation ("disassembled" from the internal format).

Not really developed any of this into a cohesive library or tool, partly
as it tends to be fairly ad-hoc, and I am not sure if anyone besides
myself would find something like this all that useful. These sorts of
small-scale tests were usually done via "copy-pasting something together".

....

Actually, thinking of it, it isn't exactly that huge of a stretch that
someone could also run such a GP evolver on an FPGA (as opposed to
running it on the CPU on a PC). It is possible that an FPGA could be
significantly faster at this task (if one had a good way to move results
and data between the FPGA and a PC).

....

Scott Smader wrote:
> On Monday, November 15, 2021 at 8:18:50 AM UTC-8, EricP wrote:
>
>> Thanks for the pointers.
> Well, quoting Bobcat Goldthwait, "Thank you for encouraging my behavior."
>
>> This sounds like what I was trying to speculate earlier, that information
>> in the NN is encoded not only in the location of connections and
>> in their weights, but also in the phase delay of the signal arrivals.
>>
>> An analogy would be an asynchronous logic circuit with feedback pathways
>> where propagation delay on the interconnect wire encodes part of the
>> signal processing logic.
>
> That is very much in haromony with the thinking in this paper which documents the use of varying myelination of axons to produce precise timing in birdsong.
> Local axonal conduction delays underlie precise timing of a neural sequence
> https://www.biorxiv.org/content/10.1101/864231v1
>
> This 2009 paper directly states your proposition: "[T]he visual detection threshold fluctuates over time along with the phase of ongoing EEG activity. The results support the notion that ongoing oscillations shape our perception, possibly by providing a temporal reference frame for neural codes that rely on precise spike timing."
> The Phase of Ongoing EEG Oscillations Predicts Visual Perception
> https://www.jneurosci.org/content/29/24/7869
>
> And the criticality of phase-related information is also suggested by this:
> Intracranial recordings reveal ubiquitous in-phase and in- antiphase functional connectivity between homologous brain regions in humans
> https://www.biorxiv.org/content/10.1101/2020.06.19.162065v2
>
> Possibly related, this paper claims that brain signaling is divided into frequency bands:
> Causal evidence of network communication in whole-brain dynamics through a multiplexed neural code
> https://doi.org/10.1101/2020.06.09.142695
> I don't believe the paper addresses this, but multiple bands could be used simultaneously for individual phase-synchronization signals in separated functional networks.
>
> In line with Ivan's insightful comment about evolution optimizing control systems to the edge of chaos, it also makes sense that given enough time, evolution would find an (approximate) implementation of almost every possible signal processing technique.
>
> And maybe even back-propagation, too, as speculated in this very recent paper about some simulations they did:
> Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits
> https://www.nature.com/articles/s41593-021-00857-x
> I'm too cheap to buy the article, but this Wired article describes it:
> https://www.wired.com/story/neuron-bursts-can-mimic-a-famous-ai-learning-strategy/

Google Scholar finds

https://www.biorxiv.org/content/10.1101/2020.03.30.015511v2.full

> Numenta forum has a discussion about it: https://discourse.numenta.org/t/burst-as-a-local-learning-rule-in-apical-but-not-basal-dendrites/9093
>
> Fun stuff!
>

Thanks.

With respect to phase shifting of signals I was thinking it might be
viewed as a dynamic mechanism.
Viewed statically neuron A triggers if B AND (C OR D) inputs are present.
But viewed dynamically signal C arrives early, D arrives later,
so they can be seen as causing a phase shift in A's output,
C advances A's output, D retards A's output.

Networks of such dynamic circuits connected together with feedback
reminded me of a hologram where the associative memory is distributed
across the whole of the circuit. That would be a big advantage as it
means that no single neuron is responsible for an individual memory.
Also the storage capacity is much higher than just the number
of synapses implies. Also no back propagation is required.

https://en.wikipedia.org/wiki/Holographic_associative_memory

I finally found the links I was looking for and it seems I am not
the first to note a similarity between neural networks and holograms
as it was proposed by Pribram in 1969. Apparently these are called
"Holographic Recurrent Networks" or "Holographic Reduced Representations"

Holographic Recurrent Networks, Plate, 1992
https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.17.6991&rep=rep1&type=pdf

Holographic Reduced Representations, Plate, 1995
https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.33.4546&rep=rep1&type=pdf

Plugging those titles into Google Scholar leads to
great many papers on holographic neural memory like

Encoding Structure in Holographic Reduced Representations, 2013
https://www.researchgate.net/profile/Douglas-Mewhort/publication/233836706_Encoding_Structure_in_Holographic_Reduced_Representations/links/57adb76608ae0932c976b72e/Encoding-Structure-in-Holographic-Reduced-Representations.pdf

Dynamically Structured Holographic Memory 2014
https://psyarxiv.com/pw93e/download/?format=pdf

Towards holographic brain memory based on randomization and
Walsh-Hadamard transformation 2016
https://www.researchgate.net/profile/Shlomi-Dolev/publication/289487007_Holographic_Brain_Memory_and_Computation/links/5d23fd2e92851cf4407280d0/Holographic-Brain-Memory-and-Computation.pdf

My gut tells me that somehow all of the above ties together
and the path forward to really useful artificial NN's is in
systems that combine all of the above ideas.
Which is why I thought convolution NN look like a dead end.

On Wednesday, November 17, 2021 at 1:00:55 PM UTC-8, EricP wrote:
<snip>
> With respect to phase shifting of signals I was thinking it might be
> viewed as a dynamic mechanism.
> Viewed statically neuron A triggers if B AND (C OR D) inputs are present.
> But viewed dynamically signal C arrives early, D arrives later,
> so they can be seen as causing a phase shift in A's output,
> C advances A's output, D retards A's output.
>

Subutai Ahmad at Numenta has proposed that sub-threshold dendritic pre-charge can prime a neuron to fire enough sooner than competitive neurons in a voting circuit to allow it to inhibit their firing. It's certainly possible that there are asynchronous races between portions of neural circuits as they compete for their own axonal discharge behavior (fire/not fire/burst/partial depolarization) to get a reward from the system for that behavior (eg, nourishment that allows new synapses to be made; maybe some other hygienic activity by astrocytes). But if the system is phase-locked (with some controllable variability) to a reference, then isn't it easier to pick winners and losers? (Presumably, neurons that don't meet the next-cycle deadline don't get rewarded and gradually disconnect from that circuit.)

And it's possible that networks of asynchronous circuits might fire synchronously at chaotically stable frequencies without being influenced to do so, but that seems pretty unlikely to me, especially given the ubiquity of alpha, theta, etc. waves.

> Networks of such dynamic circuits connected together with feedback
> reminded me of a hologram where the associative memory is distributed
> across the whole of the circuit. That would be a big advantage as it
> means that no single neuron is responsible for an individual memory.
> Also the storage capacity is much higher than just the number
> of synapses implies. Also no back propagation is required.
>
> https://en.wikipedia.org/wiki/Holographic_associative_memory
>
> I finally found the links I was looking for and it seems I am not
> the first to note a similarity between neural networks and holograms
> as it was proposed by Pribram in 1969. Apparently these are called
> "Holographic Recurrent Networks" or "Holographic Reduced Representations"
>
> Holographic Recurrent Networks, Plate, 1992
> https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.17.6991&rep=rep1&type=pdf
>
> Holographic Reduced Representations, Plate, 1995
> https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.33.4546&rep=rep1&type=pdf
>
> Plugging those titles into Google Scholar leads to
> great many papers on holographic neural memory like
>
> Encoding Structure in Holographic Reduced Representations, 2013
> https://www.researchgate.net/profile/Douglas-Mewhort/publication/233836706_Encoding_Structure_in_Holographic_Reduced_Representations/links/57adb76608ae0932c976b72e/Encoding-Structure-in-Holographic-Reduced-Representations.pdf
>
> Dynamically Structured Holographic Memory 2014
> https://psyarxiv.com/pw93e/download/?format=pdf
>
> Towards holographic brain memory based on randomization and
> Walsh-Hadamard transformation 2016
> https://www.researchgate.net/profile/Shlomi-Dolev/publication/289487007_Holographic_Brain_Memory_and_Computation/links/5d23fd2e92851cf4407280d0/Holographic-Brain-Memory-and-Computation.pdf
>
> My gut tells me that somehow all of the above ties together
> and the path forward to really useful artificial NN's is in
> systems that combine all of the above ideas.
> Which is why I thought convolution NN look like a dead end.

I agree that the original CNNs are inferior to however networks are configured in human brains, and I mean no disrespect to your gut, but there are other ways to survive noise and component failure in an associative memory. One such is sparse data representation, also described by Numenta. SDRs have other neat characteristics, like the ability to store multiple entries in one group of neurons while retaining the ability to access them individually, and the correspondence between the sparsity of neurons required for a representation in an SDR and the fact that only about 2% of neurons are active at any given moment. Numenta offer a suite of tools to explore SDRs.

Subutai Ahmad has also shown how a single neuron can learn hundreds of contexts or sequence of its inputs. The Numenta group has done some really innovative and useful work.

One thing I don't like about Numenta's approach is their assumption that intelligence is neocortical. That seems kinda chauvinist coming from the species with the most blown-out neocortexx, and a whole lot of animals survived and are surviving without neocortices. Besides, humans had bigger brains before 3,000 years ago, so were people smarter then?
https://www.frontiersin.org/articles/10.3389/fevo.2021.742639/full

And now I have some learning to do about the biological basis for holographic memory. I had dismissed it years ago, as reports of hippocampal localization of memory, place and grid cells, visual cortex specificity - esp. layer 1, etc., grew more frequent. I had not realized it's actively being pursued.

Thank you.

On Wednesday, November 17, 2021 at 5:32:18 PM UTC-8, Yoga Man wrote:
> On Wednesday, November 17, 2021 at 1:00:55 PM UTC-8, EricP wrote:
> <snip>
> > With respect to phase shifting of signals I was thinking it might be
> > viewed as a dynamic mechanism.
> > Viewed statically neuron A triggers if B AND (C OR D) inputs are present.
> > But viewed dynamically signal C arrives early, D arrives later,
> > so they can be seen as causing a phase shift in A's output,
> > C advances A's output, D retards A's output.
> >
> Subutai Ahmad at Numenta has proposed that sub-threshold dendritic pre-charge can prime a neuron to fire enough sooner than competitive neurons in a voting circuit to allow it to inhibit their firing. It's certainly possible that there are asynchronous races between portions of neural circuits as they compete for their own axonal discharge behavior (fire/not fire/burst/partial depolarization) to get a reward from the system for that behavior (eg, nourishment that allows new synapses to be made; maybe some other hygienic activity by astrocytes). But if the system is phase-locked (with some controllable variability) to a reference, then isn't it easier to pick winners and losers? (Presumably, neurons that don't meet the next-cycle deadline don't get rewarded and gradually disconnect from that circuit.)
>
> And it's possible that networks of asynchronous circuits might fire synchronously at chaotically stable frequencies without being influenced to do so, but that seems pretty unlikely to me, especially given the ubiquity of alpha, theta, etc. waves.
> > Networks of such dynamic circuits connected together with feedback
> > reminded me of a hologram where the associative memory is distributed
> > across the whole of the circuit. That would be a big advantage as it
> > means that no single neuron is responsible for an individual memory.
> > Also the storage capacity is much higher than just the number
> > of synapses implies. Also no back propagation is required.
> >
> > https://en.wikipedia.org/wiki/Holographic_associative_memory
> >
> > I finally found the links I was looking for and it seems I am not
> > the first to note a similarity between neural networks and holograms
> > as it was proposed by Pribram in 1969. Apparently these are called
> > "Holographic Recurrent Networks" or "Holographic Reduced Representations"
> >
> > Holographic Recurrent Networks, Plate, 1992
> > https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.17.6991&rep=rep1&type=pdf
> >
> > Holographic Reduced Representations, Plate, 1995
> > https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.33.4546&rep=rep1&type=pdf
> >
> > Plugging those titles into Google Scholar leads to
> > great many papers on holographic neural memory like
> >
> > Encoding Structure in Holographic Reduced Representations, 2013
> > https://www.researchgate.net/profile/Douglas-Mewhort/publication/233836706_Encoding_Structure_in_Holographic_Reduced_Representations/links/57adb76608ae0932c976b72e/Encoding-Structure-in-Holographic-Reduced-Representations..pdf
> >
> > Dynamically Structured Holographic Memory 2014
> > https://psyarxiv.com/pw93e/download/?format=pdf
> >
> > Towards holographic brain memory based on randomization and
> > Walsh-Hadamard transformation 2016
> > https://www.researchgate.net/profile/Shlomi-Dolev/publication/289487007_Holographic_Brain_Memory_and_Computation/links/5d23fd2e92851cf4407280d0/Holographic-Brain-Memory-and-Computation.pdf
> >
> > My gut tells me that somehow all of the above ties together
> > and the path forward to really useful artificial NN's is in
> > systems that combine all of the above ideas.
> > Which is why I thought convolution NN look like a dead end.
> I agree that the original CNNs are inferior to however networks are configured in human brains, and I mean no disrespect to your gut, but there are other ways to survive noise and component failure in an associative memory. One such is sparse data representation, also described by Numenta. SDRs have other neat characteristics, like the ability to store multiple entries in one group of neurons while retaining the ability to access them individually, and the correspondence between the sparsity of neurons required for a representation in an SDR and the fact that only about 2% of neurons are active at any given moment. Numenta offer a suite of tools to explore SDRs.
>
> Subutai Ahmad has also shown how a single neuron can learn hundreds of contexts or sequence of its inputs. The Numenta group has done some really innovative and useful work.
>
> One thing I don't like about Numenta's approach is their assumption that intelligence is neocortical. That seems kinda chauvinist coming from the species with the most blown-out neocortexx, and a whole lot of animals survived and are surviving without neocortices. Besides, humans had bigger brains before 3,000 years ago, so were people smarter then?
> https://www.frontiersin.org/articles/10.3389/fevo.2021.742639/full
>
> And now I have some learning to do about the biological basis for holographic memory. I had dismissed it years ago, as reports of hippocampal localization of memory, place and grid cells, visual cortex specificity - esp. layer 1, etc., grew more frequent. I had not realized it's actively being pursued.
>
> Thank you.
Oops. I was logged in with my other Google ID. I'm usually Yoga Man in other contexts. I'll try to keep it Scott Smader here.