In recent years, researchers have leveraged computer algorithms for a host of applications across industries, including the arts. In 2018, the “Edmond de Belamy, from La Famille de Belamy,” an original artwork created by an artificial intelligence (AI) computer algorithm sold at Christie’s for $432,500. Now, a team of researchers are leveraging algorithms to shed light on similarities in art across cultures, styles, and mediums. Researchers at MIT have developed an image retrieval system to comb through a vast art collection and pinpoint analogous artworks.

Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) in partnership with Microsoft created the algorithm known as “MosAIc” to sift through the Metropolitan Museum of Art and Amsterdam’s Rijksmuseum. The image retrieval system was then leveraged to determine the best match, or “analogous” work, for a given piece using these two collections.

SEE: TechRepublic Premium editorial calendar: IT policies, checklists, toolkits, and research for download (TechRepublic Premium)

Using a particular piece as the standard for a given analysis, the system can be dialed in to identify a similar piece within a vast set of filter parameters. This allows the team to use a particular image and find the closest object either filtered through style or media. The algorithm can then deliver the closest glassware match, for example, or Egyptian piece for a selected artwork.

As the team explained in a recent release, if the image retrieval system was prompted to answer “which musical instrument is closest to this painting of a blue and white dress,” MosAIc would retrieve a photo of a white and blue porcelain violin. The authors of the report note that, while the two items share similar form and stylistic patterning, these objects also “draw their roots from a broader cultural exchange of porcelain between the Dutch and Chinese.”

“Image retrieval systems let users find images that are semantically similar to a query image, serving as the backbone of reverse image search engines and many product recommendation engines,” says MIT CSAIL Ph.D. student Mark Hamilton, the lead author on a paper about MosAIc. “Restricting an image retrieval system to particular subsets of images can yield new insights into relationships in the visual world. We aim to encourage a new level of engagement with creative artifacts.”


A deep dive into the mind of the networks

To understand how these deep networks perceived the similarities between artworks, the researchers needed to analyze the network activations. Per the report, the proximity of these activations, also known as “features,” is how the researcher’s determined image similarity.

The team also created a “conditional KNN Tree” data structure with similar pieces grouped into portions of particular branches within the larger framework. According to the report, “the data structure improves on its predecessors by allowing the tree to quickly “prune” itself to a particular culture, artist, or collection, quickly yielding answers to new types of queries.”

The researchers found that this data structure could also be used for purposes other than comparative analysis between the two art collections.

SEE: Robotics in the enterprise (free PDF) (TechRepublic)

Future applications of these findings

Today, Generative Adversarial Networks (GANs) are routinely used to develop so-called deepfake images. This data structure framework of groupings can be leveraged to pinpoint where these probabilistic models excel at deepfake image creation and areas where these models are less refined.

“The idea is that instead of filling this tree with art, you fill this tree with deepfakes and real images. If you look where the deepfakes cluster, that’s the areas where these algorithms [GANs] are particularly good at making images,” said Hamilton.


While these systems are particularly skilled in some areas, at other times, these models create rather peculiar images. The researchers dubbed the areas where these models are less sophisticated as “blind spots.” The blind spots can “give us insight into how GANs can be biased,” per the report.

“Going forward, we hope this work inspires others to think about how tools from information retrieval can help other fields like the arts, humanities, social science, and medicine,” Hamilton said. “These fields are rich with information that has never been processed with these techniques and can be a source for great inspiration for both computer scientists and domain experts. This work can be expanded in terms of new datasets, new types of queries, and new ways to understand the connections between works.”