## Learning From Data – A Short Course: Problem 2.22

Page 74: When there is noise in the data, , where . If is a zero-mean noise random variable with variance , show that the bias-variance decomposition becomes     We have: We split the above expression in two sub-expressions, in which: Together, we can derive that:

## Learning From Data – A Short Course: Problem 1.3

Page 33: Prove that the PLA eventually converges to a linear separator for separable data. The following steps will guide you through the proof. Let be an optimal set of weights (one which separates the data). The essential idea in this proof is to show that the PLA weights get “more aligned” with with every [...]

## Learning From Data – A Short Course: Exercise 1.3

Page 8: The weight update rule in (1.3) has the nice interpretation that it moves in the direction of classifying correctly. (a) Show that . [Hint: x(t) is misclassified by .] Because is misclassified by so: : . Hence: . : . Hence: . So: . (b) Show that . [Hint: Use (1.3).] My solution for (b) is wrong. [...]

## Learning From Data – A Short Course: Exercise 3.12

Page 103: We know that in the Euclidean plane, the perceptron model cannot implement all 16 dichotomies on 4 points. That is . Take the feature transform in (3.12). (a) Show that . We have proved (in Exercise 2.4) that the hypothesis set of perceptron model in Euclidean plane has , and by the definition of [...]

## Learning From Data – A Short Course: Exercise 3.10

Page 98: (a) Define an error for a single data point to be     Argue that PLA can be viewed as SGD on with learning rate . when   means that agrees with (no error at that point): . when  and  disagrees (that point is misclassified): . Hence: When there is no error at the [...]

## Learning From Data – A Short Course: Exercise 3.8

Page 94: The claim that is the direction which gives largest decrease in only holds for small . Why? is small and ignorable only when is small. P/s: I’m bored. That’s why I’m posting separate post.

## Learning From Data – A Short Course: Exercise 3.4

Page 87: Consider a noisy target for generating the data, where is a noise term with zero mean and variance, independently generated for every example . The expected error of the best possible linear fit to this target is thus . For the data , denote the noise in as and let , assume that [...]