In deep learning, neurons are not represented as a linear function. The output of a neuron is implemented by taking a linear combination of the inputs and then feeding that into a non-linear function, e.g. ReLU. The non-linearity is critical, because without it, you can't approximate non-linear functions well, even with deep networks.

1. Layered linear nodes can model non-linear behaviours.
2. Computational complexity. Its more efficient to use the aforementioned layers of linears than it is to use non-linear functions directly

From the edited question I can see that you understand that it is possible but unnecessary, yet you are still wondering whether it has any drawbacks or inertia.

One thing to consider is that modern deep learning relies on very efficient parallel hardwares such as GPUs. They are usually made to carry out simple instructions in a massively parallel manner. A widely known metaphor is that CPU is several highly educated people, while GPU is thousand of ten year olds. If it is the best we have, we as well may should utilize what they can do - performing simple instructions(matrix mult etc).

If using polynomial neurons or such stuff has added benefits such as extending theoretical results, it might have been considered. However, we have Univ. Approx Thm and rich results that makes such effort less exciting.

So yes, I think if you can somehow design some next generation GPU which can evaluate cubic polynomials extremely fast, by all means ML will learn to adapt using cubics as building blocks.

This is a bit like why computers happen to use binary and not ternary. Everything has been tried before.

There's a long and rich history for artificial neural networks but everybody seems to gravitate toward fewer moving parts in their already uncertain and difficult work.

I guess eventually the paved road of today's MLPs with GPUs became so convenient to use that very few have the time or means to try something radically different without good reason.

This is a fun read by the way: https://stats.stackexchange.com/questions/468606/why-arent-neural-networks-used-with-rbf-activation-functions-or-other-non-mono

Assuming you mean how the weights/biasses are summed up, we don't. In neural networks the sum of weights is passed through a (usually) nonlinear activation function. Because of the nonlinearity we get the universal approximation theorem which says a 1 layer neural network is suffecient to model any continuous function, with some caveats.

Basically the nonlinearity in the activations is assumed to be suffecient for all applications, therefore it is not necessary to have a more complex structure.

However, when in comes to discontinuous functions things get more interesting, 1 layer NNs cannot, for example, model the XOR function. But with a more complex or Nonlinear construction this might be possible.

Thank you for the replies.

I understand that neural networks can represent non-linear complex functions.

To clarify more,

My question is that a single neuron still computes F(X) = WX + b, which is a linear function.

Why not use a higher order function F(X) = WX^n + W1 X^(n-1) + ... +b.

I can imagine the increase in computational needed to implement this, but neural networks were considered to be time-consuming until we started using GPUs for parallel computations.

So if we ignore the implementation details to accomplish this for large networks, are there any inherent advantages to using higher-order neurons?

A common theme in these topics is people observe the status quo design decisions, such as linear layers connected by relu, and then try and backwards rationalize it with relatively hand-wavy mathematical justification. Citing things such as the universal approximation theorem which are not particularly relevant.

The reality is that this field is heavily driven by empirical results, and I would be highly skeptical of anyone saying that "xyz is the clear best way to do it".

Here is my answer: First, let’s ask why not an exponential or logarithmic function instead of a quadratic or a higher order polynomial? Or maybe a sinusoid function? The thing is, we might be needing one of such nonlinearities, or maybe another kind of nonlinearity, based on the problem, and, we don’t know it. The idea with neural networks is that it combines many simple neurons that can learn any of these nonlinearities inside it during training. If such a quadratic relationship is a relevant feature for your problem, it will learn it, meaning that some of the neurons will end up with simulating that quadratic relationship. This is a much more flexible way than hard coding the nonlinearity right away from the beginning.

First it is not linear. Relu functions (rectified linear functions) are piecewise linear model. By combining them you can make approximate any non linear function with a finite number of discontinuities as close as you want. Think of it as some form of tesselation. If you add more face, you can match the function better. Note that it is note mandatory to use piecewise linear function. You may use other nonlinearity such as tanh. But using such bounded linearities, the learning process becomes subject to vanishing gradient that makes the learning process very difficult. As ReLu are unbounded they avoid this problem.

Is this going to make the function approximation strength of NNs any better than what it already is?

Probably not. It is already quite good. I don't think not using polynomial building blocks is the bottleneck.

I have seen very complicated high dimensional manifolds in different settings (physics, finance etc.) being learnt by simple but sometimes huge MLPs. Unless there is a strong inductive reason/bias in how a polynomial layer can help contribute towards in an ML problem, there isn't any strong reason to use it. Indeed, overfitting isn't function approximation gone bad rather the opposite as anyone who has trained a NN will know.

Non-linear doesn’t mean polynomial. There are many non-linear functions polynomials cannot approximate well, such as softmax() or even just sin(). Ultimately, it’s enough to approximate complex functions with simple functions without having to worry about picking the correct complex function.

Spiking Neural Networks are a thing too, if you want to explore a different approach.

As many others have mentioned, the decision boundaries from piecewise linear models are actually quite smooth in the end, given a sufficient amount of layers.

But to get to the core of your question, why would you prefer many stupid neurons over few smart ones. I believe there is a relatively simple explanation why the former is better. Having more complex neurons would mean that the computational complexity goes up while the number of parameters stays the same. I.e. with the same compute, you can train bigger (number of params) models if the neurons are simple. A high number of parameters is important for optimization as extra dimensions can be helpful in getting out of local minima. Not sure if this has been fully explained, but it is in part the reason why pruning works so well: we wouldnt need that many parameters to represent a good fit, but it is much easier to find in high dimensions, from where we can prune down to simpler models (only 5% of parameters with almost same performance).

Well, probably this is one reason: https://en.wikipedia.org/wiki/Universal_approximation_theorem

Nobody else has mentioned resnets, yet. They have something like higher order weights with f(x) = σ(W1σ(W0x+b0)+b1) + σ(W0x+b0). Highway networks take it a step further with f(x) = σ(W0x+b0)σ(W1x+b1) + xσ(W2x+b2). However, both are done to resolve gradient issues.

The outputs of neurons are passed through a nonlinearity which is essential to any (complex) learning process. If we didn't do this, then the NN would be a composition of linear functions which is itself a linear function (pretty boring).

As for why we choose to operate on inputs with an affine transformation before putting them through a nonlinearity, I see two reasons. The first is that linear transformations are well understood and succinct to use theoretically. The second is that computers (in particular GPUs) are very good with matrix multiplication, so we do a lot of "heavy lifting" with them and then just pass the result through a nonlinearity so we don't get a boring learning process.

Just my 2 cents, happy for input/feedback!

A linear system is NOT necessarily composed of linear functions. Nonlinear systems are very difficult and sometimes impossible to solve numerically.

overfitting

F(x) = s(WX+b) isn’t all of deep learning.

You may have heard of transformers, which are closer to s(X W X^T) (but actually more involved than that). They’re an extremely popular model right now.