Fabric defect detection is a key part to ensure quality of product in textile industry. The paper addressed on structure design and implementation of real-time fabric defect detection system based on Gabor filtering. The system mainly consists of three parts of image acquisition, defect detection and defect classification. Defect detection is core of whole system, the basic structure of which is PCI-based image data transmission as well as process system based on FPGA and external RAM. The Gabor filtering algorithm is used for defect detection, which is implemented with high performance FPGA assisted by external RAM to buffer image data.