Video context analysis is an active and vibrant research area, which provides means for extracting, analyzing and understanding behavior of a single target and multiple targets. Over the last few decades, computer vision researchers have been working to improve the accuracy and robustness of algorithms to analyse the context of a video automatically. In general, the research work in this area can be categorized into three major topics: 1) counting number of people in the scene 2) tracking individuals in a crowd and 3) understanding behavior of a single target or multiple targets in the scene. This book focusses on tracking individual targets and detecting abnormal behavior of a crowd in a complex scene. Firstly, this book surveys the state-of-the-art methods for tracking multiple targets in a complex scene and describes the authors' approach for tracking multiple targets. The proposed approach is to formulate the problem of multi-target tracking as an optimization problem of finding dynamic optima (pedestrians) where these optima interact frequently. A novel particle swarm optimization (PSO) algorithm that uses a set of multiple swarms is presented. Through particles and swarms diversification, motion prediction is introduced into the standard PSO, constraining swarm members to the most likely region in the search space. The social interaction among swarm and the output from pedestrians-detector are also incorporated into the velocity-updating equation. This allows the proposed approach to track multiple targets in a crowded scene with severe occlusion and heavy interactions among targets. The second part of this book discusses the problem of detecting and localising abnormal activities in crowded scenes. We present a spatio-temporal Laplacian Eigenmap method for extracting different crowd activities from videos. This method learns the spatial and temporal variations of local motions in an embedded space and employs representatives of different activities to construct the model which characterises the regular behavior of a crowd. This model of regular crowd behavior allows for the detection of abnormal crowd activities both in local and global context and the localization of regions which show abnormal behavior. The last chapter suggests a number of research directions to be pursued for future work.