**Prediction and Inference:**

Prediction's f is like a black-box while inference seeks the inner relationships and therefore needs f in exact form.

**Estimation of f:**

**Parametric method:**

linear; overfitting

**Non-parametric methods:**

In contrast, non-parametric approaches completely avoid this danger, since essentially no assumption about the form offis made. Since they do not reduce the problem of estimatingfto a small number of parameters, a very large number of observations (far more than is typically needed for a parametric approach) is required in order to obtain an accurate estimate for f.

**Trade-Off Between Prediction Accuracy and Model Interpretability**

Still depends on the preference to prediction or inference.

We will often obtain more accurate predictions using a less flexible method. - "overfitting"

**Supervised Versus Unsupervised Learning**

Unsupervised Learning: We can seek to understand the relationships between the variables or between the observations.

Example: classification v.s. clustering

Semi-supervised learning: m cases with supervised model and n - m the other.

**Regression Versus Classification Problems**

quantitive: numerical values and qualitive: classes/categories

quantitive: regression, qualitive: classification

Since it(logistic regression) estimates class probabilities, it can be thought of as a regression method as well.

**Measuring the Quality of Fit**

As model flexibility increases, training MSE will decrease, but the test MSE may not. When a given method yields a small training MSE but a large test MSE, we are said to be overfitting the data.