Because of increasing terrorist activities, the resolution of video cameras and the number of cameras deployed for surveillance are increasing exponentially - producing huge amount of video data. Manual analysis of this large volume of video data by human operators for crime scene and forensic analysis is neither reliable nor scalable. This has generated enormous interest in research activities related to automation of video surveillance systems which allows real-time automatic extraction and analysis of information from live incoming video streams and enables automatic detection and tracking of targets without human intervention. To meet the real-time requirements of automated video surveillance systems, very different technologies and design methodologies have been used in literature. These range from use of General Purpose Processors (GPPs) or special purpose Digital Signal Processors (DSPs) or Graphics Processing Units (GPUs) to Application Specific Integrated Circuits (ASICs) or Applications Specific Instruction Set Processors (ASIPs) or even programmable logic devices like Field Programmable Gate Arrays (FPGAs). FPGAs provide real-time performance that is hard to achieve with GPPs/DSPs, limit the extensive design work, time, and cost required for ASICs, and allow algorithmic changes in later stages of system development. Due to these features FPGAs are being increasingly used for prototyping automated video surveillance system quickly. In this paper we present the top level description of a complete automated video surveillance system along with the elaboration of different challenges/issues involved in its design and implementation, a comparative analysis of design methodologies and existing FPGA platforms, complete design flow for prototyping the FPGA-based automated video surveillance system, and details of various primary input/output interfaces required for designing smart automated video surveillance systems for future.