In the formulation of practical optimization methods, it is often the case that the choice of numerical linear algebra method used in some inherent calculation can determine the choice of the whole optimization algorithm. The numerical linear algebra is particularly relevant in large-scale optimization, where the linear equation solver has a dramatic effect on both the robustness and the efficiency of the optimization.

We review some of the principal linear algebraic issues associated with the design of modern optimization algorithms. Much of the discussion will concern the use of direct and iterative linear solvers for large-scale optimization. Particular emphasis will be given to some recent developments in the use of regularization.