Learning From Data – A Short Course: Exercise 7.18

Since the input x is an image it is convenient to represent it as a mtrix [x_{ij}] of its pixels which are black (x_{ij} = 1) or white (x_{ij} = 0). The basic shape of  \phi_{k} identifies a set of these pixels which are black.

(a) Show that feature  \phi_{k} can be computed by the neural network node

    \[ \phi_{k}(x) = \tanh{\left ( w_{0} + \sum_{ij}w_{ij}x_{ij} \right )} \]

Set  w_{ij} = 1 if the pixel (i, j) is black in the basic shape and  w_{ij} = 0 otherwise. w_{0} is the threshold that we want to decide how many correct black pixel needed for the input image to be considered to have that basic shape, w_{0} = \frac{1}{2} \text{ number of black pixels of the basic shape} for example.

However, what if the input image has only black color? Then maybe we want to set w_{ij} = -1 if the pixel (i, j) is white in the basic shape, so that if the pixel (i, j) in the input image is black then its total score will be penalized by 1 point. But our input image may contain several basic shapes so it might not be smart to blindly penalize “incorrect” pixels like that, the weights should be adjusted in some ways to achieve the best performance and I believe that’s what Neural Network does (actually, through some Neural Network algorithms, the basic shapes may look pretty different).

(b) What are the inputs to the neural network node?

After the dicussion in (a), I think we all know what the inputs are.

(c) What do you choose as values for the weights? [Hint: consider separately the weights of the pixels for those x_{ij} \in \phi_{k} and those x_{ij} \notin \phi_{k} ]

Dicussed in (a).

(d) How would you choose w_{0}? (Not all digits are written identically, and so a basic shape may not always be exactly represented in the image.)

Dicussed in (a).

(e) Draw the final network, filling in as many details as you can.

As implicitly mentioned in the e-chapter, the units z_{1} and z_{2} should be two AND implementations and by that their weights are defined, the output unit should output + 1 when the digit is 1 and - 1 when the digit is 5 and  0 when the digit is neither 1 nor 5 (I feel like we are no longer in binary classification problem).


Leave a Reply

Your email address will not be published. Required fields are marked *