Learning From Data – A Short Course: Exercise 4.3

Page 125:

Deterministic noise depends on H, as some models approximate f better than others.

(a) Assume H is fixed and we increase the complexity of f. Will deterministic noise in general go up or down? Is there a higher or lower tendency to overfit?

Deterministic noise in general will go up because it is harder for any hypothesis in H to approximate f, so the bias and var components of expected out-of-sample error will go up. If expected in-sample error is low then the model is overfitting, the expected in-sample error is high then the model is underfitting.

(b) Assume f is fixed and we decrease the complexity of H. Will deterministic noise in general go up or down? Is there a higher or lower tendency to overfit? [Hint: There is a race between two factors that affect overfitting in opposite ways, but one wins.]

Deterministic noise in general will go up because H has lower chance to approximate f, so the bias component of expected out-of-sample error will go up, however the var component will go down. I am not sure about the overfitting tendency in this case (though it looks like the var will eventually wins), however we have this quote from the author:

So getting back to the point, if you make H more complex, you will decrease the det. noise (bias) but you will increase the var (its indirect impact). Usually the latter dominates (overfitting, not because of the direct impact of the noise, but because of its indirect impact) … unless you are in the underfitting regime when the former dominates.

There is also a dicussion about this on the forum.


Facebooktwittergoogle_plusredditpinterestlinkedinmail

Leave a Reply

Your email address will not be published. Required fields are marked *