The Complete Guide
All you need to know about Joone
17. January 2007
by Paolo Marrone
http://www.joone.org
Java Object Oriented Neural Engine
Java Object Oriented Neural Engine
Joone Core Engine The Complete Guide
Table of Contents
1 Introduction...................................................................................6
1.1 Intended Audience........................................................................................6
1.2 What is Joone................................................................................................6
1.2.1 Custom systems.....................................................................................7
1.2.2 Embedded systems................................................................................7
1.2.3 Mobile Devices.......................................................................................7
1.3 About this Guide...........................................................................................7
1.4 Acknowledgements.......................................................................................9
2 Getting and Installing Joone..........................................................10
2.1 Platform and requirements.........................................................................10
2.2 Installing the binary distribution................................................................10
2.2.1 The Core Engine...................................................................................10
2.2.2 The GUI Editor......................................................................................12
2.3 Building from the source distribution.........................................................17
2.3.1 Prerequisites.........................................................................................17
2.3.2 Getting the last released source code.................................................17
2.3.3 Getting the CVS sources......................................................................17
2.3.4 Compiling..............................................................................................18
3 Inside the Core Engine..................................................................19
3.1 Basic Concepts...........................................................................................19
3.2 The Transport Mechanism..........................................................................20
3.3 The Processing Elements............................................................................22
3.3.1 The Layers............................................................................................22
3.3.1.1 The Linear Layer............................................................................22
3.3.1.2 The Biased Linear Layer................................................................23
3.3.1.3 The Sigmoid Layer.........................................................................23
3.3.1.4 The Tanh Layer...............................................................................24
3.3.1.5 The SoftMax Layer.........................................................................24
3.3.1.6 The Logarithmic Layer...................................................................24
3.3.1.7 The Sine Layer...............................................................................25
3.3.1.8 The Delay Layer.............................................................................25
3.3.1.9 The Context Layer..........................................................................26
3.3.1.10 The GaussLayer...........................................................................27
3.3.1.11 The RBFGaussianLayer................................................................27
3.3.1.12 The WinnerTakeAll Layer..............................................................28
3.3.1.13 The Gaussian Layer.....................................................................29
3.3.2 The Synapses.......................................................................................30
3.3.2.1 The Direct Synapse........................................................................30
3.3.2.2 The Full Synapse............................................................................31
3.3.2.3 The Delayed Synapse....................................................................31
3.3.2.4 The Kohonen Synapse....................................................................32
3.3.2.5 The Sanger Synapse......................................................................33
3.4 The Monitor: a central point to control the neural network.......................34
3.4.1 The Monitor as a container of the Network Parameters......................35
3.4.2 The Monitor as the Network Controller................................................35
3.4.3 Managing the events............................................................................37
http://www.joone.org 2
Joone Core Engine The Complete Guide
3.4.4 How the patterns and the internal weights are represented .............40
3.4.4.1 The Pattern.....................................................................................40
3.4.4.2 The Matrix......................................................................................40
3.5 Technical details.........................................................................................41
3.5.1 The Layer abstract class......................................................................41
3.5.1.1 The Recall Phase............................................................................42
3.5.1.2 The Learning Phase........................................................................42
3.5.2 Connecting a Synapse to a Layer........................................................43
3.5.3 The Synapse abstract class..................................................................44
4 I/O components: a link with the external world..............................47
4.1 The Input mechanism.................................................................................47
4.1.1 The FileInputSynapse...........................................................................49
4.1.2 The URLInputSynapse..........................................................................49
4.1.3 The ExcelInputSynapse........................................................................50
4.1.4 The JDBCInputSynapse.........................................................................50
4.1.5 The ImageInputSynapse......................................................................51
4.1.6 The YahooFinanceInputSynapse...........................................................52
4.1.7 The MemoryInputSynapse...................................................................53
4.1.8 The InputConnector..............................................................................53
4.2 The Output: using the outcome of a neural network.................................56
4.3 The Switching Mechanism..........................................................................56
4.3.1 The InputSwitch....................................................................................57
4.3.2 The OutputSwitchSynapse...................................................................57
4.4 The Validation Mechanism..........................................................................58
4.5 Technical details.........................................................................................60
4.5.1 The StreamInputSynapse.....................................................................60
4.5.2 The StreamOutputSynapse..................................................................63
4.5.3 The Switching mechanism’s object model..........................................63
4.5.3.1 The InputSwitchSynapse................................................................64
4.5.3.2 The OutputSwitchSynapse.............................................................64
4.5.3.3 The LearningSwitch........................................................................64
5 Teaching a neural network: the supervised learning......................65
5.1 The Teacher component.............................................................................65
5.2 Comparing the desired with the output patterns......................................67
5.3 The Supervised Learning Algorithms.........................................................67
5.3.1 The basic On-Line BackProp algorithm................................................69
5.3.2 The Batch BackProp algorithm............................................................70
5.3.3 The Resilient BackProp algorithm (RPROP)..........................................70
5.3.4 How to set the learning algorithm.......................................................70
5.4 Technical details.........................................................................................72
5.4.1 The learning components object model..............................................72
5.4.2 The Learners object model..................................................................74
5.4.3 The Extensible Learning Mechanism...................................................75
6 The Plugin based expansibility mechanism....................................78
6.1 The Input Plugins........................................................................................78
6.2 The Output Plugins.....................................................................................78
6.3 The Monitor Plugins....................................................................................79
6.4 The Scripting Mechanism...........................................................................81
6.5 Technical details.........................................................................................82
6.5.1 The Input/Output Plugins object model...............................................82
http://www.joone.org 3
Joone Core Engine The Complete Guide
6.5.2 The Monitor Plugin object model.........................................................84
6.5.3 The Scripting mechanism object model..............................................86
7 Using the Neural Network as a Whole............................................88
7.1 The NeuralNet object..................................................................................88
7.2 The NestedNeuralLayer object...................................................................89
7.3 Technical details.........................................................................................92
8 Common Architectures..................................................................94
8.1 Modular Neural Networks...........................................................................94
8.1.1 The Parity Problem...............................................................................94
8.2 Temporal Feed Forward Neural Networks..................................................97
8.2.1 Time Series Forecasting.......................................................................97
8.2.1.1 Preprocessing.................................................................................97
8.2.1.2 Trend prediction:..........................................................................100
8.2.1.3 Dynamic control of the training parameters...............................103
8.3 Unsupervised Neural Networks................................................................107
8.3.1 Kohonen Self Organized Maps...........................................................107
8.3.1.1 Example: a character recognition system...................................107
9 Applying Joone............................................................................114
9.1 Build your own first neural network.........................................................114
9.2 The standard API.......................................................................................114
9.2.1 A simple (but useless) neural network..............................................114
9.2.2 A real implementation: the XOR problem. ........................................115
9.3 Saving and restoring a neural network....................................................119
9.4 Using the outcome of a neural network...................................................121
9.4.1 Writing the results to an output file...................................................121
9.4.2 Getting the results into an array........................................................122
9.4.2.1 Using multiple input patterns......................................................122
9.4.2.2 Using only one input pattern.......................................................124
9.5 Controlling the training of a neural network............................................125
9.5.1 Controlling the RMSE..........................................................................125
9.5.2 Cross Validation..................................................................................126
9.6 The JooneTools helper class......................................................................130
9.6.1 Building & runnning a simple feed forward neural network ............130
9.6.2 The JooneTools I/O helper methods...................................................132
9.6.3 Testing the performances of a network.............................................134
9.6.4 Building unsupervised (SOM) networks.............................................135
9.6.5 Loading and saving a network with JooneTools.................................135
10 The LGPL Licence......................................................................136
http://www.joone.org 4
Joone Core Engine The Complete Guide
I
I would like to present the objectives that I had in mind when I started to write the first lines of code
of Joone.
My dream was (and still is) to create a framework to implement a new approach the use of neural
networks. I felt this necessity because the biggest (and unresolved until now) problem is to find the
fittest network for a given problem, without falling into local minima, thus finding the best
architecture.
Okay - you'll say - this is what we can do simply by training some randomly initialized neural
network with a supervised or unsupervised algorithm.
Yes, it's true, but this is just scholastic theory, because training only one neural network, especially
for hard problems of the real life, is not enough.
To find the best neural network is a really hard task because we need to determine many parameters
of the net such as the number of the layers, how many neurons for each layer, the transfer function,
the value of the learning rate, the momentum, etc... often causing frustrating failures.
The basic idea is to have an environment to easily train many neural networks in parallel, initialised
with different weights, parameters, or different architectures, so the user can find the best NN simply
by selecting the fittest neural network after the training process.
Not only that, but this process could continue retraining the selected NNs until some final parameter
is reached (i.e. a low RMSE value) like a distillation process. The best architecture can be discovered
by Joone, not by the user! Many programs today exist that permit selection of the fittest neural
network applying a genetic algorithm. I want to go beyond this, because my goal is to build a flexible
environment programmable by the end user, so any existing or newly discovered global optimisation
algorithm can be implemented. This is why Joone has its own distributed training environment and
why it is based on a cloneable engine.
My dreams aren't finished, because another one is to make a trained NN easily usable and
distributable by the end user. For example, I'm imagining an assurance company that continuously
trains many neural networks on customer’s risk evaluation
1
(using the results of historical cases),
distributing the best ‘distilled’ (or genetically evolved) neural network to its sales force, so that they
can use it on their mobile devices.
This is why a neural network built with Joone is serializable and remotely transportable using any
wired or wireless protocol, and it is easily runnable using a simple, small and generalized program.
Moreover, my dream can become a more solid reality thanks to the advent of handheld devices like
mobile phones and PDA having inside a java virtual machine. Joone is ready to run on them, too.
Hoping you’ll find our work interesting and useful, thank you for your interest in Joone.
Paolo Marrone
and the Joone team
1
The ethics (and the law in many countries) forbids to make racial, sexual, religious (and others) discriminations.
Consequently, a decisional system based on such personal characteristics cannot be built.
http://www.joone.org 5
