Training-Time Optimization of AI

How is a modern foundation model actually made? Not simply by choosing an optimizer, but through a sequence of coupled decisions about data, trajectories, parameterization, scale, post-training, and compute. This graduate-level text develops the mathematics behind those decisions and shows how optimization theory can illuminate the manufacturing process of contemporary AI systems.

Beginning with nonconvex landscapes and training dynamics, the book moves from solution sets, gradient flows, conservation laws, and implicit bias to data-mixture optimization, deduplication, synthetic-data feedback, and algorithmic selection. It then examines solvable large-width limits, neural tangent kernels, feature learning, µP and hyperparameter transfer, before separating genuine critical phenomena from sharp but ordinary crossovers. Later chapters connect the optimizer to generalization through stability, PAC-Bayes and information-theoretic bounds, and develop a unified mathematical view of post-training in policy space, including supervised fine-tuning, reward models, PPO, DPO, and group-relative methods. The final part treats compute, memory, precision, and other training constraints through value functions and shadow prices.

Written for graduate students, researchers, and technically oriented practitioners, Training-Time Optimization for AI emphasizes what each theorem actually establishes, where idealized models stop transferring to frontier-scale training, and which quantities can be measured in real training runs. The result is not a catalog of optimization algorithms, but a mathematical framework for understanding why training procedures deliver the models they do.