How Instagram’s Viral Face Filters Work

Computer vision is incredibly valuable to Facebook, and Pokémon, Disney princesses, and ‘Friends’ characters help us see why

What Pokémon are you? Which Disney character or Friends friend? You’ve probably seen these face filters invading Instagram over the past few weeks. Pull up the camera, point it at your face, and the app will generate a random selection for you. Naturally, you post the result — “Snorlax??” — your friends see it, and then they make their own; a viral sensation is born.

All of this is the result of Instagram opening up its platform last year, giving anyone with a computer and a free program called Spark AR the ability to make their own simple face filter in just a few minutes. You don’t even need technical expertise: A simple YouTube tutorial will show you how it’s done.

The filter community is flourishing. There are those “which X are you?” filters, but some creators have gone much deeper. Instagram filter artist Dvoshanky has created side-scrolling games inside filters, where a user can play a Flappy Bird clone, or as an alien invading Area 51 who has to jump over obstacles. All of it stems from computer vision technology that allows the software to understand, in a very basic sense, what it’s “seeing” in a photograph or video. In recent years, it’s become a crucial feature in social apps like Instagram and Snapchat, which are always hungry for innovative ways to hook users.

There are around 80,000 members in Facebook’s “Spark AR Creators” group, and the subset of those who actually make effects are given a relatively small set of tools. The Spark AR software allows filters to detect a face, identify three expressions—smiling, kissing, or surprised—and track a person’s hand. They can also allow a user to place a digital object in their surroundings.

Making a filter through the Spark AR interface is incredibly simple and visual. Creators are given a two-dimensional workspace and place virtual blocks based on what they want the filter to do. If you want your filter to react to the camera being turned on, add a camera block and create a digital switch to be flipped when the camera is detected. Adding a face detector is as simple as dragging and dropping it…