In this book, approximate dynamic programming (ADP) designs based on adaptive critic structures are developed to solve the discrete-time optimal control problems in which the state and action spaces are continuous. This work considers linear discrete-time systems as well as nonlinear discrete- time systems that are affine in the input. This work resulted in forward-in-time reinforcement learning algorithms that converge to the solution of the Generalized Algebraic Riccati Equation (GARE) for linear systems. For the nonlinear case, a forward-in-time reinforcement learning algorithm is presented that converges to the solution of the associated Hamilton-Jacobi Bellman equation (HJB).