This book explores the usage of Reinforcement Learning for Multi-Agent Coordination. Chapter 1 introduces fundamentals of the multi-robot coordination. Chapter 2 offers two useful properties, which have been developed to speed-up the convergence of traditional multi-agent Q-learning (MAQL) algorithms in view of the team-goal exploration, where team-goal exploration refers to simultaneous exploration of individual goals. Chapter 3 proposes the novel consensus Q-learning (CoQL), which addresses the equilibrium selection problem. Chapter 4 introduces a new dimension in the literature of the traditional correlated Q-learning (CQL), in which correlated equilibrium (CE) is computed partly in the learning and the rest in the planning phases, thereby requiring CE computation once only. Chapter 5 proposes an alternative solution to the multi-agent planning problem using meta-heuristic optimization algorithms. Chapter 6 provides the concluding remarks based on the principles and experimental results acquired in the previous chapters. Possible future directions of research are also examined briefly at the end of the chapter.