Viewing a single comment thread. View all comments

Blakut t1_jdo7dkp wrote

Reply to comment by _AlreadyTaken_ in The two retinas are tied/linked together in the brain. Are they tied 1:1, so that each retinal point corresponds to the same retinal point in the other eye? I.e., each retinal point from one eye shares the same binocular neuron with its counterpoint in the other eye? by ch1214ch

afaik the first convolutional neural networks in AI were modelled to mimic the retina (cows in particular? idk)

8

Fenrisvitnir t1_jdpqicw wrote

No. Convolutional networks are simply fully connected all-combinations of every pixel in the image (under a sliding window, usually). They are not modeled after any brain, they are modeled after signal processing convolution filters (pre-neural network) for 2D signals. The learning epochs of the convolution network teach the network which pixels to pay attention to at the meta level (features), and the further levels combine those features.

5

Blakut t1_jdpy8xc wrote

> Convolutional networks are simply fully connected a

uhm no.

https://en.wikipedia.org/wiki/Convolutional_neural_network

Convolutional networks were inspired by biological processes[10][11][12][13] in that the connectivity pattern between neurons resembles the organization of the animal visual cortex. Individual cortical neurons respond to stimuli only in a restricted region of the visual field known as the receptive field. The receptive fields of different neurons partially overlap such that they cover the entire visual field.

2

Fenrisvitnir t1_jdq1qvn wrote

Um no. References 10-13 don't establish the fact, if you look at them. Convolution kernels long predate their use in neural networks as a convolution layer.

The sliding NxM convolution window is the "receptive field" but it isn't analogous to the field in the eye. The kernel matrix existed long before it was used in NNs, and is the mapping mechanism to the fully connected convolution input layer.

https://en.wikipedia.org/wiki/Kernel_(image_processing)

Thanks for being interested, but there is a lot of fluffery in ML discussions. The neurons of a NN are not remotely the same as biological neurons - the only thing they share in common is the activation function, and even then they are only symbolically similar.

11

adventuringraw t1_jdrnbmp wrote

Um no (we have to keep the comment chain going).

You're actually being overly dismissive of what they're saying I think. The key word they used was 'inspired'. I tried to dig up the origin of convolutional image kernels, and while I couldn't find much in five minutes of digging, I'm sure you're right, that they predate deep learning certainly, and possibly even digital computing entirely given that their historical origin was probably in signal processing.

Their comment though wasn't whether or not CNNs directly imitate biology, or that the way they did it was entirely novel... They were just pointing out that biology was an inspiration for trying it this way, and that part's unambiguously true. To my knowledge, the first paper introducing the phrase 'convolutional neural network' was from Yann LeCun. This one I believe, from 1989. If you look at the references, you'll note Hubel and Wiesel's 1962 paper introducing a crude model of biological vision processing is in the references. More importantly, Fukushima, 1980 is referenced (and mentioned in the text as a direct inspiration). This 'Neocognitron' is generally accepted to be the first proto-CNN. The architecture is a bit different than we're used to, but it's where things started... And as the author puts it in the abstract:

> A neural network model for a mechanism of visual pattern recognition is proposed in this paper. The network is self-organized by "learning without a teacher", and acquires an ability to recognize stimulus patterns based on the geometrical similarity (Gestalt) of their shapes without affected by their positions. This network is given a nickname "neocognitron". After completion of self-organization, the network has a structure similar to the hierarchy model of the visual nervous system proposed by Hubel and Wiesel.

So... Yes. CNNs weren't inspired by cow vision or something... Hubel and Wiesel's most famous work involved experiments on kittens. but CNN origins are unambiguously tied into Hubel and Wiesel's work in biological visual processing, so the person you're responding to is actually the one that was right. I just noticed even, some of the papers referenced from Wikipedia that you said didn't show biological inspiration are the same ones I mentioned even, so they were the correct papers to cite.

If I may be a bit rude for my own Sunday morning amusement: 'Thanks for being interested, but there is a lot of fluffery in ML discussions.'

Seriously though, it's an interesting topic for sure, and historical image processing techniques are certainly equally important to the history of CNNs... They were the tool reached for given the biological inspiration, so in all seriousness you're not entirely wrong from another perspective, even if you're not justified in shooting down a biological inspiration.

7

Coomb t1_jdrsw29 wrote

Did anyone say that the matrix operation of convolution, or the idea of smearing it across an image, was invented via inspiration from experimental explication of image processing as performed by animals? I don't think they did, and I would be surprised if that were true. But those references do show that the "neocognitron" was explicitly inspired by actual physical neural networks used by animals for image processing, because among other things they include the original neocognitron paper, which is very clear about its inspiration. This is relevant because review papers of convolutional neural networks like this one from University College London almost universally identify the neocognitron as the direct precursor to modern convolutional neural networks.

2

Fenrisvitnir t1_jdsp8nn wrote

https://glassboxmedicine.com/2019/04/13/a-short-history-of-convolutional-neural-networks/

"The popular press often talks about how neural network models are “directly inspired by the human brain.” In some sense, this is true, as both CNNs and the human visual system follow a “simple-to-complex” hierarchical structure. However, the actual implementation is totally different; brains are built using cells, and neural networks are built using mathematical operations."

2

Coomb t1_jdsre81 wrote

Is this supposed to be responsive to my point?

2