• Page 458: I’m not sure if I interpret the statement right but here is my guess: One of the risk function in the example should look like this: .
• Page 459:
• The cost when an is refused to decide is: .
• The following arguments are applied for the Algorithm 15.1:
• If then we will rather to fail than to give it a thought whether we should decide or not. If then we prefer to make no decision at all. Hence .
• If we choose type then the cost will be . Then we will have to compare the costs of choosing type and choosing not to decide:
• We will choose type if: .
• We will choose not to decide if: .
• Reference: Bayesian Decision Theory.
• Page 461:
• , same goes for .
• Notes that is sigmoid function.
• For MLE stuffs you can take a look at Page 91 of Learning From Data – A Short Course. For further insight, Wikipedia also offers some insight at the end of the Principles section.
• For this formula: .
• When you will get: • Same goes for .
• It looks like the equality is not true in general except the case where or . Try the case and then you will see the contradiction.
• Wikipedia: “However, the logistic loss function does not assign zero penalty to any points. Instead, functions that correctly classify points with high confidence (i.e., with high values of ) are penalized less.”
• “but also due to examples that have near zero”: To be more specific, if and has the same sign and is near zero then the loss will be also very large. Sep 23, 2016: I wonder if this note is correct?
• Page 462:
• “i.e., the example is close to the decision boundary”: Notice that we have: (same goes for ). Also remember that a point is on the boundary if and only if .
• Robust machine learning.
• Hinge Lost. Solver.com: “A non-convex optimization problem is any problem where the objective or any of the constraints are non-convex, as pictured below. Such a problem may have multiple feasible regions and multiple locally optimal points within each region. It can take time exponential in the number of variables and constraints to determine that a non-convex problem is infeasible, that the objective function is unbounded, or that an optimal solution is the “global optimum” across all feasible regions.” Bookmark: NIPS 2015 workshop on non-convex optimization, ResearchGate.