Snapchat's Facial Recognition and Image Processing

In my twenty-first century, first-world, millennial view, few things in life are as annoying and yet tragically ubiquitous as the face-altering filters on Snapchat. I think they were cool at first, solely on account of their novelty, and there have even been some neat ones, like the X-Men ones from last summer. In my opinion, however, they've become increasingly overused and ridiculous. 

A filter that makes your head into a tomato and then has a stream of more tomatoes shoot out of your mouth with a revolting sound when you open your mouth? An evil, screaming rabbit? Seriously, why? People think this sort of thing is cute, but it's not. I think My friend even tells me that half of all the girls he swipes right for on Tinder have those iconic dog ears and nose plastered over their faces in their photos, which really annoys me him because how is he even supposed to get a real sense of what someone looks like?

Anyway, partly because I'm curious as to how this bane of my social media consumption works and also because I was in desperate need of a blog topic, I did a little research on what's really behind the augmented reality of Snapchat filters. Here's what I found:

Snapchat acquired the technology from a Ukrainian company called Looksery for $150 million in September of last year. It turns out that Snapchat has been pretty reluctant to share the details of its face filter secrets, but you can find the patent for the technology can be found online, so it kind of doesn't matter.

The first step in the filtering process is called detection. How does the Snapchat app recognize that the image it's getting from your phone's front-facing camera is a face? The tech is somewhat similar to how you can scan VR codes or how primitive robots can see where they're going (see my post from last week). They call it computer vision.

But seeing as human faces are markedly more complex than barcodes or wall corners, you'd think that the method to get a computer to recognize individual ones would be more complicated, no?

Recall that a computer cannot truly recognize color--it only reads binary assignments for specific color values of each individual pixel of the image. Essentially, the computer looks for the contrast of light and dark color values to discern whether or not the image is a face. For example, the bridge of your nose typically appears as a lighter shade than the sides of your nose, so the computer will pick up on that and tell itself, "Yep, that's a face." In this way, the computer uses something called the Viola-Jones algorithm.

Next, the computer needs to figure out your facial structure so it knows where to situate the virtual flower crown on your head or where it should put those god-awful dog ears. It does this with an active shape model. In succinct terms, the computer knows where your eyes, forehead, chin, etc. should be because the programmers manually marked the location of such features on a plethora of models' faces and then ran them through the computer as examples. So since humans have more or less the same facial structure (think broadly), the computer has a pretty good repertoire of faces to go off of.

After plotting the location of your features like a map, the computer creates a mesh--that is, an augmented reality mask that will move as your face does an adjust itself when you, say, open your mouth to let tomatoes spill out.

That's the gist of it. I must say that after reading about how it works I've garnered a bit more respect for the creators of these Snapchat filters. It is pretty intriguing once you see how it works.

This facial recognition software is pretty cool--but there is a dark side. Facebook, for example, has begun living up to its name by amassing a database of millions of faces based off of when people tag their friends in photos. That's just a bit creepy. Even worse, the federal government can do the same thing, which should be more than a little troubling.

Sources

• Kenyan Muslim Antichrist Who Wants to Take Our Guns w/ Dog Filter
• The Incredible Underlying Technology of Snapchat's Selfie Lenses--DigitalTrends.com
• "How Snapchat's filters work" by Vox--YouTube.com

How Robots See

I can see you.

The idea that the field of robotics might one day become so advanced that robots can function virtually the same way as living organisms has long been the subject of a plethora of science fiction films and novels. While robotics has indeed made significant strides, one impediment to its further advancement is the fact that robots are as of yet still unable to truly see the world, at least totally in the sense that humans can. But let's take a look at how most robots are able to process the world with our current technology.

So, just how do humans see? In abridged layman's terms, we use our eyes to collect light that reflects off of the matter around us. The eyes then convert that light into electric signals that travel to the brain via the optic nerves. Obviously, the brain does the heavy lifting here--and some researchers have postulated that up to 50% of our brain mass is involved, one way or another, in the process of seeing. The brain, then, processes those electric signals into valuable information about our surroundings for us.

Therefore, it is no surprise that enabling a robot to gather information about the world in this way, just as animals do, would be largely beneficial to advancing robotics.

Currently, technology allows for robots "see" the way you probably think they might: A video camera is used to collect a constant stream of images, which is then passed to the computer inside the robot. From there, a few different things can happen.

Roboticists use features in the stream of images--say, corners, lines, or unique textures-- to let the robot "see." The features are then stored in a library. They then write code that will recognize the patterns in these features to help the robot comprehend what's around it.

This code forces the robot to evaluate the information it receives from its cameras and compare its features with those it has stored in its library. So if a robot has a feature that looks like the corner of a room in its library, then it ought to be able to interpret another corner for what it is.

It's a somewhat laborious and complicated process, but it is definitely efficient.

Sources

• How do robots 'see' the world?--the conversation.com
• C-3PO--Flickr.com

How Fitbit Works

It's the brand new craze that's sweeping the nation--Fitbit! I'm sure most have, one way or another, come across these personal fitness trackers. Marketed as a way for the average consumer to keep a close watch on their daily activity, a Fitbit is a watch or wearable clip that monitors the steps you take, the calories you burn, and (depending on the version you own) your heart rate. Downloading the Fitbit app for your smartphone takes things a step further: If you sync the app with your Fitbit, you can scan the bar codes of your food and count the calories you consume, log in what you do when you go to the gym, and even challenge friends to walk more steps than you. More recently, Fitbit has come out with a sleep tracking feature.

Now, I know humanity's made some pretty decent technological progress in the few hundreds of thousands of years we've been ambling around this planet. We're in the Digital Age. We've put men on the moon, split the atom, and invented Hot Pockets. But for a long while, I was skeptical about Fitbits. How could a piece of plastic count how many calories I've burned? So I looked it up, and here's what I found.

To track steps, Fitbits use something called a three-dimensional accelerometer to track the user's movement as well as the intensity of that movement. It's quite similar to what's used in Wii remotes. This, simply, is how Fitbit tracks steps. However, this raw accelerometer data is pretty useless on its own. Fitbit relies on special algorithms to interpret it into something useful--the caloric and perambulatory information we so crave.

It seems that the engineers of Fitbit had to resort to plain old trial and error in order to finely tune the algorithm that the devices use. They compared Fitbit's algorithm to that of other, more established test machines in order to see how well it worked. For example, when the engineers were developing the feature assessing how many calories the user burns, they compared Fitbit's results with those of a portable telemetric gas analysis system. (The gas analysis system, which is so great an assessor of calorie use that googling it yields mostly scholarly articles I haven't got the time to peruse, analyzes gas composition as we exhale.) Fitbit then takes into account your basic metabolic rate (BMR), which includes heart rate and brain activity, and adds it to the data collected from the accelerometer to calculate the calories you burn.

Fitbit's sleep tracker is similar to its step tracker; it merely logs whether or not you're moving. So, if you're wearing your Fitbit while you're in bed but you can't fall out, the Fitbit will assume you're fast asleep. Well, I guess that's the thing about technology: There's always room for improvement.

Sources

• How FitBit Works--HowStuffWorks.com
• How does Fitbit estimate how many calories I've burned?--Help.Fitbit.com
• Fitbit image--Fitbit.com

The Boolean Pythagorean Triples Problem & How It Was Solved

It should come as a surprise to no one: Computers are good at math.

So good, in fact, that they can solve a problem that mathematicians have been trying to crack for thirty years in just two days.

Such is the case of a problem solved only a few months ago called the Boolean Pythagorean triples problem. A mathematician named Ronald Graham first offered a $100 prize to anyone who could solve it in the 1980s, and no one was able until May of this year. The Stampede supercomputer at the University of Texas did the job, which just so happens to be the largest-ever math proof in the history of ever: 200 terabytes.

Stampede, our heroic supercomputer

So, what's the essence of this colossal and seemingly intractable math problem? Well, you probably remember the unsuitably-named Pythagorean theorem from middle school: a^2 + b^2 = c^2. The integers a, b, and c in the Pythagorean theorem are known as Pythagorean triples. The problem asks whether it's possible for each of the positive integers can be colored either red or blue so that a combination of the Pythagorean triple can satisfy the Pythagorean theorem without having any of the integers be the same color.

In May, researchers at the University of Texas had Stampede attempt to solve the problem, and it spent two days working it out. (Now just imagine how long it would take a human, or an army of humans.) Stampede determined that yes, this is attainable. Another program even checked Stampede's resulting 200 terabyte proof and found it to be sound. 

Answering the question posed by the Boolean Pythagorean triples problem only raised a veritable legion of different questions, however. For example, the question remains of just why it's possible for all of the integers to not have the same color. Also, it's apparently only possible to color the integers differently for 7,824. After the 7,825th integer, it just becomes impossible. Why is that?

I think perhaps this just speaks to our own relative lack of understanding of how the problem really works. But, like any science, it'll probably take dozens of minds and years of collaboration before humanity as a whole has really increased and improved our knowledge base. Really makes me wonder how long it would've taken people to figure the problem out without the use of a supercomputer, though.

Sources

• World's Largest Math Proof Solved. And It Takes Up 200 Terabytes. From Futurism.com
• Image Representation of the Problem's Solution. From Nature.
• Photograph of Stampede