Video surveillance is of increasing importance to many applications. In this book, we investigate several key problems and develop new technologies for video surveillance systems. Two basic problems are investigated: automatic recognition of activities and optimization of video codec systems. Firstly, automatic activity recognition plays a key role in video surveillance. We develop new algorithms that can recognize human activities with good accuracy. The problems of recognition with limited training data and flexibility for recognition algorithms are also addressed. Furthermore, we develop a group activity recognition algorithm for recognizing interactions among groups of people. The algorithm can deal with the problems of recognition with varying numbers of group members as well as with hierarchical interactions among people. Secondly, many surveillance systems require the video codec to deliver videos in real time with low power in order for battery-based sensors to work as long as possible. This makes the codec optimization very important. Thus, we also develop multiple codec optimization techniques to address this key problem.