Measure Classification Performance

Measure the accuracy of a digit recognizer trained on the MNIST database of handwritten digits.

First obtain the training and validation data.

In[1]:=

resource = ResourceObject["MNIST"]; trainingData = ResourceData[resource, "TrainingData"]; testData = ResourceData[resource, "TestData"];

In[2]:=

RandomSample[trainingData, 5]

Out[2]=

Define a convolutional neural network that takes in 28x28 grayscale images as input.

In[3]:=

lenet = NetChain[{ ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2], ConvolutionLayer[50, 5], Ramp, PoolingLayer[2, 2], FlattenLayer[], 500, Ramp, 10, SoftmaxLayer[]}, "Output" -> NetDecoder[{"Class", Range[0, 9]}], "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}] ]

Out[3]=

Train the network for three training rounds.

In[4]:=

lenet = NetTrain[lenet, trainingData, ValidationSet -> testData, MaxTrainingRounds -> 3]

Out[4]=

Evaluate the trained network directly on images randomly sampled from the validation set.

In[5]:=

imgs = Keys @ RandomSample[testData, 5]; Thread[imgs -> lenet[imgs]]

Out[5]=

Create a ClassifierMeasurements object from the trained network and the validation set.

In[6]:=

cm = ClassifierMeasurements[lenet, testData]

Out[6]=

Obtain the accuracy of the network on the validation set.

In[7]:=

cm["Accuracy"]

Out[7]=

List 3s that were misclassified as 8s.

In[8]:=

cm[{"Examples", 3 -> 8}]

Out[8]=

Obtain a plot of the confusion matrix of the network predictions on the validation set.

In[9]:=

cm["ConfusionMatrixPlot"]

Out[9]=