The Task

Let's try a multilayer neural network for time series prediction.

First load matrix library version 3.4W. I have added a couple of things to make your life easier for this assignment.

Modify your multilayer neural network to learn to predict a "next element in the series". This is not a classification problem. See section 4.4.4 in your book for more background. You will read in 3 numbers followed by a column matrix. The later will be read in using the read() method for the Matrix object or equivalent. 

#step #stride #hidden
#rows 1 
row1
row2
 ...
lastrow
#rows 1
row1
row2
 ...
lastrow

#step is the number of steps using a stride of #stride. The true input to NN will be a series of overlapping windows of size #step + 1. The first #step values are the input and the last value is the expected output. The method seriesSampleCol in the matrix library will resample a column from a matrix using step and stride to create the desired matrix. Read the comments that go with the method for details. This is a function in the version 3.4 library.

The resulting matrix from series sample method is used to train your NN. Adjust the parameters of you NN for the number of inputs and outputs using this created matrix for training.

Now you need to make a few small changes to your multilayer NN.

Normalize your input so it is between 0 and 1. Use the number of hidden nodes given as the third number read in. Remove the sigmoid function in the FINAL layer since that will try to give you an answer of either 0 or 1. You want the answer to anywhere in the space between 0 and 1. Your final layer is now just the raw sum of weighted inputs with no transfer function. This means in the backprop phase the first delta back to the hidden layer is no longer the fancy but rather just the simple difference between your prediction and the target. This gives a simple signed value into the backprop. Be sure to scale this value by dividing by number of samples (rows) as seen in the author's example.

Next add a momentum parameter to your backprop. Every pass your weights W and V are updated by update matrix. Remember the update matrix. You should now have W and V updated not just by the newly computed update but add in a momentum factor (a number between 0 and 1) times the last update you did. A momentum factor of .9 tends to maintain "direction" while a momentum factor of 0 gives you the old update with no memory of previous direction.

All the Python code for the book is available on-line. Here is the multilayer NN code from the author. Notice the linear option to the mlptrain procedure. This is the adjustment we are making. We are not doing the logistic option for this problem. You can also see the use of the momentum factor.

Output

The test data for this exercise is just the training data. For output you should:

  1. print the normalized time series matrix you will train on.
  2. print a 2 column matrix with your prediction followed by the target. Don't forget to unnormalize.
  3. the square of the Euclidean distance between the prediction and target.
The test data has some simple sine curves plus the notorious sunspot data by year. The test files specify step, stride, and number of hidden nodes, but play around with these values and you can see what makes a problem harder and easier. This is not part of the assignment but just personal entertainment and education. Time series are not easy.

Your code must compile and run on the class unix machine. If it does not compile or fails to run (e.g. gets seg faults) or in other ways produces no output that is a very serious fault and will result in a very poor score. In 4xx/5xx CS classes your code should at least run and produce reasonable output.

makefile

Your makefile must make the program nn.

Submission

Tar up all the code necessary along with a makefile to build the program named nn that reads the sample data from stdin as described above. Homework should be submitted as an uncompressed tar file to the homework submission page linked from the class web page. You can submit as many times as you like. The LAST file you submit BEFORE the deadline will be the one graded. For all submissions you will receive email at your uidaho.edu mail address giving you some automated feedback on the unpacking and compiling and running of code and possibly some other things that can be autotested.

Have fun.