Hyperparameter Tuning

Hyperparameters are the knobs you turn before training begins: learning rate, batch size, number of layers, hidden dimensions, dropout rate, weight decay, and optimizer settings. Unlike model parameters (weights) that are learned from data, hyperparameters must be set by the practitioner and can dramatically affect model performance. A 10x difference in learning rate can mean the difference between convergence and divergence.

Tuning approaches range from manual experimentation (informed by experience and intuition) to systematic methods like grid search (trying all combinations), random search (sampling random configurations, often more efficient than grid search), and Bayesian optimization (using previous results to intelligently choose the next configuration to try). Tools like Optuna, Ray Tune, and Weights & Biases Sweeps automate the process.

For production ML, the practical approach is to start with established defaults for your architecture (Adam optimizer, learning rate 1e-4, batch size 32), tune the most impactful hyperparameters first (learning rate is almost always the highest priority), and use early stopping to avoid wasting compute on unpromising configurations. For LLM fine-tuning specifically, learning rate, number of epochs, and LoRA rank are the three most important hyperparameters to tune.