One of the broad applications of scientific fields is real time moving object tracking and its implementation on a high speed reconfigurable hardware. This implementation has a positive effect on speed, accuracy, efficiency and reduces design cost. This paper presents an optimized design of moving object tracking based on color and edge fusion by using mean shift algorithm. The partitioned synthesizable design simulate written by VHDL carry out on Xilinx ISE14.2 and implemented on XC5SX35 FPGA. Results show the implemented tracking system has a good efficiency on speed and accuracy.