In the past few years, Graphics Processing Unit (GPU) has evolved into a unified powerful many-core processor. The modern GPUs are well suited for compute-intensive tasks and massively parallel computation. As one of the most common and important matrix problems, solving the large-scale linear system can be significantly accelerated if corresponding algorithms can be mapped well to the structure of the GPU and be accord with SIMD (Single-Instruction, Multiple-Data) pattern. In this paper, the authors focus on the problem of solving the Poisson equation. The equation arises in many applications such as computational fluid dynamics, electrostatics, magneto statics, etc.