In my twenty-first century, first-world, millennial view, few things in life are as annoying and yet tragically ubiquitous as the face-altering filters on Snapchat. I think they were cool at first, solely on account of their novelty, and there have even been some neat ones, like the X-Men ones from last summer. In my opinion, however, they've become increasingly overused and ridiculous.
A filter that makes your head into a tomato and then has a stream of more tomatoes shoot out of your mouth with a revolting sound when you open your mouth? An evil, screaming rabbit? Seriously, why? People think this sort of thing is cute, but it's not. I think My friend even tells me that half of all the girls he swipes right for on Tinder have those iconic dog ears and nose plastered over their faces in their photos, which really annoys me him because how is he even supposed to get a real sense of what someone looks like?
Anyway, partly because I'm curious as to how this bane of my social media consumption works and also because I was in desperate need of a blog topic, I did a little research on what's really behind the augmented reality of Snapchat filters. Here's what I found:
Snapchat acquired the technology from a Ukrainian company called Looksery for $150 million in September of last year. It turns out that Snapchat has been pretty reluctant to share the details of its face filter secrets, but you can find the patent for the technology can be found online, so it kind of doesn't matter.
The first step in the filtering process is called detection. How does the Snapchat app recognize that the image it's getting from your phone's front-facing camera is a face? The tech is somewhat similar to how you can scan VR codes or how primitive robots can see where they're going (see my post from last week). They call it computer vision.
But seeing as human faces are markedly more complex than barcodes or wall corners, you'd think that the method to get a computer to recognize individual ones would be more complicated, no?
Recall that a computer cannot truly recognize color--it only reads binary assignments for specific color values of each individual pixel of the image. Essentially, the computer looks for the contrast of light and dark color values to discern whether or not the image is a face. For example, the bridge of your nose typically appears as a lighter shade than the sides of your nose, so the computer will pick up on that and tell itself, "Yep, that's a face." In this way, the computer uses something called the
Viola-Jones algorithm.
Next, the computer needs to figure out your facial structure so it knows where to situate the virtual flower crown on your head or where it should put those god-awful dog ears. It does this with an active shape model. In succinct terms, the computer knows where your eyes, forehead, chin, etc. should be because the programmers manually marked the location of such features on a plethora of models' faces and then ran them through the computer as examples. So since humans have more or less the same facial structure (think broadly), the computer has a pretty good repertoire of faces to go off of.
After plotting the location of your features like a map, the computer creates a mesh--that is, an augmented reality mask that will move as your face does an adjust itself when you, say, open your mouth to let tomatoes spill out.
That's the gist of it. I must say that after reading about how it works I've garnered a bit more respect for the creators of these Snapchat filters. It is pretty intriguing once you see how it works.
This facial recognition software is pretty cool--but there is a dark side. Facebook, for example, has begun living up to its name by amassing a database of millions of faces based off of when people tag their friends in photos. That's just a bit creepy. Even worse, the federal government can do the same thing, which should be more than a little troubling.
Sources
