In the last few decades, we have been witnessing an information revolution that can be traced back to the invention of integrated circuits and computer chips. The widespread use of electronic devices and computing resources has tremendously increased the rate at which we generate, gather and store data in every fields of science, business and engineering. Machine learning takes center stage in our quest for analyzing this vast amount of data and extracting latent knowledge from it. However, as the datasets grow, increased running time and memory constraints may become prohibitive. Additionally, real-world data can have noise and imbalanced class distribution, which adversely affect the generalization accuracy of learning algorithms. In order to cope with these challenges, science in the 21st century requires a different set of computational techniques and algorithms. This book presents methodologies that address these issues with the goal of improving scalability, computational and data efficiency and generalization performance of machine learning algorithms in the context of online and active learning with a particular focus on Support Vector Machines.