As the size of FPGA devices grows following Moore's law, it becomes possible to put a complete manycore system onto a single FPGA chip. The centralized memory hierarchy on typical embedded systems in which both data and instructions are stored in the off-chip global memory will introduce the bus contention problem as the number of processing cores increases. In this paper, the authors present their exploration into how distributed multi-tiered memory hierarchies can effect the scalability of manycore systems. They use the Xilinx Virtex FPGA devices as the testing platforms and the buses as the interconnect. Several variances of the centralized memory hierarchy and the distributed memory hierarchy are compared by running various benchmarks, including matrix multiplication, IDEA encryption and 3D FFT.