This post describes the approach used for digit recognition in my Android application Sudoku Capture. It’s based on the image recognition plug-in for Neuroph, which is a Java neural network framework.
Neuroph
The easyNeurons graphical editor provides tools for creating and training neural networks with Neuroph. The image recognition tool described in this post was used to create a multilayer perceptron neural network for pattern recognition.
The latest version of Neuroph (2.4) also includes tools for text and handwriting recognition. However, simply using the image recognition tool allowed me to train a neural network with exactly the characteristics of the inputs that would be encountered at run time.
Character recognition
The neural network distinguishes between the patterns of the digits 1-9. It’s possible that the same approach could be used for more general purpose OCR but I suspect that larger alphabets would require more pre-processing than was necessary in this example. Skeletonization or other normalisation of the inputs might be useful in that case.
Training data
I’m starting with a set of 15×15 .png training images. How I arrive at a set of digits to recognise in my application is descibed in this post. I used over 2000 training images with various typefaces. A pre-processing stage transforms them into a square so that the digit patterns occur in the same locations across similar images.
Each digit is essentially represented as 225 pixel positions which are either on or off (1 or 0, foreground or background).
One directory contains all training digits. Backpropagation neural networks use supervised training, so training images must be hand classified. Each training image is labelled with the number it represents. There are many training images for each digit so they are labelled as one_1, one_2 one_3 … nine_1, nine_2 etc. This tells the image recognition tool that many images belong to the same label.
Another directory contains a solid black and solid white square of the same size for the junk field.
Structure
As each input pattern is pre-processed into a 15×15 square, the input layer of the neural network has 225 neurons with values 0 or 1. This is specified by the width and height. The black and white setting is selected because we’re really using the tool to create a neural network for monochrome pattern recognition.
(There is a great guide for doing actual image recognition with this tool here.)
Any given input could be 1 of 9 digits so there are 9 neurons in the output layer. The output layer is the size of the alphabet. So if you wanted to recognise the characters a-z there would be 26 output neurons. This is handled automatically by the tool based on the labels in the training directory.
The Sigmoid transfer function is selected. Only one hidden layer is needed. Finding the optimal number of hidden neurons was simply a process of experimentation and elimination.
Results
It took around 8 hours of training (using Neuroph 2.3.1 – training is faster in 2.4) for a total network error of 0.01. Eight different hidden neuron counts were tried. 15 hidden neurons produced the best result for my data. A lot of the other variations became stuck during training.
The overall result was good. An evaluation of the application in the hands of real first time users recorded a correct recognition rate of 97%.
Android
The neural network created by this process is serialized Java code. It can run on Android but the image recognition plug-in must be removed and the neural network accessed directly. This is because the plug-in used to train the network uses some AWT classes that aren’t available on Android.
The simple solution is to write a program that loads the neural network, removes the plug-in and saves it again. This means you lose the output neuron mapping so be prepared for some trial and error recovery.
Sudoku Capture
This image is taken from the last stage of the testing tool, which I mentioned in an earlier post about how the Sudoku Capture app works.
It demonstrates how the resulting neural network was tested on a large number of real and unseen inputs prior to user testing.
Hallo! I am a student from Poland. I’m writing Image Recognition Application on Android. Firstly, I used easyNeurons and tool for creating and removing neural networks. Then I wrote a program that loads neural network and removes OCR plugin. At the end, the program saves it again. (standard java)
Could you tell me how I can load file .nnet in my Andorid project now?
The code below does’t work in my project:
NeuralNetwork nnet = NeuralNetwork.load(inputStream);
or NeuralNetwork nnet = NeuralNetwork.load(“pathString”);
LogCat:
java.lang.StackOverflowError
at java.io.DataInputStream.readByte(DataInputStream.java:127)
at java.io.ObjectInputStream.nextTC(ObjectInputStream.java:650)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:837)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:2299)
…
Neural network file is in ‘assets’ directory. Maybe there is another way of doing it?
I will be glad if you could give me some tips.
I store the serialized neural network as a raw resource and read it into memory when my app starts.
Resources res = getResources();
InputStream input = res.openRawResource(R.raw.neuralnet);
I read the input stream into a file and then call NeuralNetwork.load() on that file path. As far as I can remember anyway.
Thank you for your response. Unfortunately, it didn’t help.
How did you load the neural network, removed the plugin and save it again?
I tried to replace the awt api with and.awt, but it only makes problems.
Hi,
the reason for the StackOverflowError is the small standard stack size on android.
You have to create a new thread with a bigger stack size (max is 256k on android).
Hope it will help.
I also get a stack java.lang.StackOverflowError error. I would be nice to see your code:)
Hey dude, could you provide the code snippet for loading a .nnet file and retrieving results from the plug in?
I also get the StackOverflowError as Artur, plus I do not understand this part:
“The simple solution is to write a program that loads the neural network, removes the plug-in and saves it again.”
Thanks!
The following Java class can be used to remove the image recognition plug-in. This allows you to use a neural network created with the image recognition plug-in on Android, but you need to access the output neurons directly.
public class ConvertNeuralNet {
/**
* @param args[0] : Serialised Neuroph neural network .nnet file.
*/
public static void main(String args[]) {
NeuralNetwork nnet = NeuralNetwork.load(args[0]);
nnet.removePlugin(ImageRecognitionPlugin.IMG_REC_PLUGIN_NAME);
String out = args[0].substring(0, args[0].lastIndexOf(‘.’)) + “safe.nnet”;
nnet.save(out);
}
}
Till is correct about the cause of StackOverflowErrors. You need a thread with a large stack size to de-serialise the neural network. You should only need to run it once on load.
I realise that there is a bit of interest in the intricacies of running this type of neural network on the Android platform. I’m going to look at open-sourcing this application when I have some time to clean and comment.
Im also have trouble with stackOverflow errors. Whatever I try I cant deserialize a file bigger than a few KBs, even after creating a new Thread with a stack size of 1MB. Seems to not have any effect at all. Your NerualNetwork loading could would be much much appreciated.
And for the rest, well done, thanks for the help, and nice website =]
Nice article! It really helped me getting started with Neuroph and number recognition. I have a question… Are you willing to make your trained neural network available to the public?