For their final project in associate professor Sergio Alvarez’s “Artificial Intelligence” course in the computer science department, Andrew Francl ’16 and Jesse Mu ’17 chose to adapt the work of computational neuroscientists at Germany's University of Tübingen, who in September 2015 reported on an algorithm they’d developed that extracts the brush-stroke style of a painting and applies it to another image. Francl, a double major in computer science and biology with a minor in mathematics, and Mu, a computer science major with a minor in mathematics, reconstructed and extended the Tübingen algorithm to produce, among other “artwork,” this scene—a photograph of Gasson Hall at sunset—suggesting the distinctive textures of Vincent van Gogh’s The Starry Night (1889).
Unlike computer apps that are programmed to impose specific painterly effects—an impressionist haze, say—onto another image, the algorithm that made this image can autonomously extract the style of any artist. The process, known as deep learning, involves algorithms, modeled on human neural systems, that enable a computer to learn from experience, modifying its behavior in response to mathematical rules as it takes on more data. The network Francl and Mu worked with, called a convolutional neural network, was modeled on the visual cortex, the part of the brain primarily involved with processing visual information.
Using a programming language called Python and an algorithm template titled Caffe, Francl and Mu translated mathematical formulations from the Tübingen paper into some 650 lines of code, sitting side by side in the Fulton Hall computer science lab. The work required hours of “trial-and-error and debugging,” said Francl. “We had to make sure our computational intuitions aligned with the mathematical equations,” added Mu. The image above is the 512th step by the computer in improving its faithfulness to van Gogh’s brush-strokes (the quality plateaued thereafter), and represents some six hours of computing time.
The students tested the algorithm on a range of brush styles, from the proto-impressionism of J. M. W. Turner to the pointillism of Georges Seurat, transposing those styles onto color photographs of, for instance, the Back Bay skyline. In doing so, they encountered shortcomings in their program: Notably, it failed to make sense of art that lacked a distinctive surface texture—for example, Salvador Dalí’s sleek The Persistence of Memory (1931).
To capture van Gogh’s use of color, Francl and Mu wrote an extension to the Tübingen algorithm resulting in the sky shown here, but its application proved limited. The pair also attempted to extract an artist’s overall style, using 45 cubist works by Picasso. Their conclusion: For that artist at least, “the average becomes meaningless.”
Francl will study cognitive sciences at MIT in the fall. Mu is a 2016 Goldwater Scholar.