Parameter Tuning

Most learning algorithms have some parameters that need to be set, which is commonly referred to as parameter tuning, and the performance of the learned model often varies significantly depending on the parameter configuration.

Many parameters of the learning algorithm are taken in the real range, so it is not feasible to train a model for each parameter. It is common to select a range and step $\lambda$ for each parameter, which makes the learning process feasible.

For example, assuming that the algorithm has 3 parameters, each considering only 5 candidate values, there are $5^3$ = $125$ models to examine for each training/test set.

It is notable that once the model and paramters have been set, we need to retrain the model using the initial dataset $D$. This means that the test set initially divided for evaluation is also learned by the model to enhance the learning effct.