Artificial Neural Networks

Figure 3.2: An example graphical representation of a Multi-layer Feed Forward Neural Network.

Artificial Neural Networks (ANNs) have multiple representational forms. The most common are the mathematical (equation 3.2) and graphical form (figure 3.2). For each Artificial Neuron, the mathematical form consists of a function $g(x)$ of the input vector $x$, where $x = (x_1, x_2, \ldots, x_i)$. Each input $x_i$ is weighted according to its weight $w = (w_1, w_2, \ldots, w_i)$. $K$ is the post-processing function that is finally applied. This results in the following equation for a single neuron:

$$g (x) = K \left(\sum_i w_i x_i)\right)$$ (3.1)

Neural Networks consist of multiple artificial neurons like these. The output of one neuron is connected to the input of another neuron. Mathematically, $g(x)$ (the result of equation 3.1) is an input to a neuron with function$f(x)$:

$$f (x) = K \left(\sum_i w_i g_i(x)\right)$$ (3.2)

To introduce non-linearity, a hyperbolic tangent or sigmoid (S-shaped) function is commonly used for $K$. Non-linearity is deliberately analogous to biological neurons, and responsible for its versatile information processing properties.

A reason for its popularity [Wik07b] in neural networks is because the sigmoid function satisfies the differential equation $y' = y(1-y)$. The right hand side is a low order polynomial. Furthermore, the polynomial has factors y and $1 - y$, both of which are simple to compute. Given $y = {\rm sig}(t)$ at a particular $t$, the derivative of the sigmoid function at that $t$ can be obtained by multiplying the two factors together. These relationships result in simplified implementations of artificial neural networks with artificial neurons.