Image as well as video processing applications like visual surveillance, vehicle tracking, traffic monitoring etc. have real time constraints to identify moving object in the given scene. Several hardware as well as software approaches are proposed which suits the real time moving object detection. However due to low cost and high performance, hardware based approaches are preferred over software based approaches. In this paper, the authors propose a FPGA based hardware architecture which is capable of detecting moving object using background subtraction algorithm.