Re-ranking, as an effective way to improve the results of web-based search, has been adopted by current commercial search engines. Given a query keyword, a pool is first retrieved by the search engine based on textual information. By asking the user to select a query image and text from the pool, the remaining images and text are re-ranked. A major challenge is that the similarities of visual image features do not well correlate with images’ semantic meanings which interpret users’ search intention.