What is fully connected neural network

Neural network

Human intelligence is based on a biological neural network with approx. 100,000,000,000 neurons in the brain, some of which are connected to a high degree in parallel with only a few, but also with many thousands of other neurons. In about a millisecond, an activated neuron can activate other neurons to which it is connected via axons and synapses.

One possibility to realize artificial intelligence is to simulate the biological neural network by means of an artificial neural network (short: KNN, English ANN = Artificial Neural Network) in the computer. The biological processes of human thinking and learning (activation of neurons, chemical change of synapses, etc.) are described mathematically as well as possible and modeled in software or hardware. The first successful neuro-computer, the Mark I Perceptron, was developed in 1957/58 by Frank Rosenblatt, Charles Wightman and colleagues at MIT (see Fig. 1, source: author's post-doctoral thesis).


Fig. 1: Frank Rosenblatt with his Mark I Perceptron 1958


Specialized in pattern recognition problems, this analog computer can recognize simple digits with a 20 by 20 pixel image sensor. The Mark I Perceptron works with the help of 512 motorized potentiometers, i. H. a potentiometer for each of the variable, trainable weights in the artificial neural network, which has a so-called perceptron topology (Fig. 2, source: author's habilitation thesis). To date, it is not nearly possible to simulate a human brain in a computer using artificial neural networks, because on the one hand the number of artificial neurons is several powers of ten less and on the other hand the highly parallel connection of many artificial neurons is not yet possible. The connections between the artificial neurons are called ANN weights and the type of activation and networking as ANN topology designated.

For the training of an artificial neural network, two main process classes must be distinguished: The supervised learning and the unsupervised learning. In supervised learning, input / output relationships, the patterns, are presented to an ANN. An error vector is calculated, the components of which are the differences between the current ANN outputs and the target outputs for the inputs. Ideally, the artificial neural network is supported by suitable Weight and topology optimization successively trained the ability to independently and sufficiently accurately deliver the target output belonging to an input, cf. B. So-called 3- and 4-layer perceptrons and radial base networks as well as Fig. 2. "E" stands for input, "A" for activation, "+" for the sum of the incoming activations multiplied by the weights: The Evaluation of the artificial neural networks takes place from left to right following the arrows. “Bias” stands for a neuron with constant activation, which provides a threshold value for the activation of subsequent neurons.



Fig. 2: Fully connected, three-layer perceptrons without (left) and with (right) direct connections


A well-trained artificial neural network should as often as possible also provide the correct target output for similar, distorted, noisy or incomplete inputs (Generalization property). This corresponds to the requirements of business decision-making processes, which often only contain incomplete, imprecise or uncertain information. In unsupervised learning, only inputs are presented to an artificial neural network. The ANN should then adapt itself automatically by making the weight vectors more similar to the input vectors. This process is also known as self-organization. In this case, the neurons are representatives of typical input data and form what are known as clusters, cf. B. so-called self-organizing maps, associative memory and Kohonen networks.

For most practical problems in business informatics, supervised learning with multi-layer perceptrons is of particular importance. Important areas of application are e.g. B.

  • Time series analyzes and forecasts,

  • Image and character recognition, especially OCR, as well as

  • Rating, control and warning systems with pattern recognition.


Modern neurosimulators, e.g. B. use the Hanoverian neurosimulator FAUN (Fast approximation with universal neural networks)

  • modern high-performance optimization processes, e.g. B. SQP method to achieve acceptable computing times,

  • Approaches of global optimization in order to avoid the problem of the extremely large number of bad local minima when minimizing the error,

  • linear and non-linear constraints on the weights, including to optimize convergence and to comply with curvature or convexity restrictions,

  • graphical user interfaces with various online and offline visualizations of the training and the ANN weights,

  • automatic division of the patterns into training, cross-validation and generalization data.


Some useful links:

http://www.snn.ru.nl/enns (European Neural Network Society)

http://ieee-cis.org/about_cis (IEEE Computational Intelligence Society)

http://www.gi-ev.de/gliederungen/fachgebiete/kuenstliche-intellektiven-ki (Society for Computer Science, Department of AI)

http://www.nips.cc (Neural Information Processing Systems)




Prof. Dr. Michael H. Breitner, Leibniz University Hanover, Institute for Information Systems, Königsworther Platz 1, 30167 Hanover

Author info

Item Actions