Snapchat's Facial Recognition and Image Processing

In my twenty-first century, first-world, millennial view, few things in life are as annoying and yet tragically ubiquitous as the face-altering filters on Snapchat. I think they were cool at first, solely on account of their novelty, and there have even been some neat ones, like the X-Men ones from last summer. In my opinion, however, they've become increasingly overused and ridiculous. 

A filter that makes your head into a tomato and then has a stream of more tomatoes shoot out of your mouth with a revolting sound when you open your mouth? An evil, screaming rabbit? Seriously, why? People think this sort of thing is cute, but it's not. I think My friend even tells me that half of all the girls he swipes right for on Tinder have those iconic dog ears and nose plastered over their faces in their photos, which really annoys me him because how is he even supposed to get a real sense of what someone looks like?

Anyway, partly because I'm curious as to how this bane of my social media consumption works and also because I was in desperate need of a blog topic, I did a little research on what's really behind the augmented reality of Snapchat filters. Here's what I found:

Snapchat acquired the technology from a Ukrainian company called Looksery for $150 million in September of last year. It turns out that Snapchat has been pretty reluctant to share the details of its face filter secrets, but you can find the patent for the technology can be found online, so it kind of doesn't matter.

The first step in the filtering process is called detection. How does the Snapchat app recognize that the image it's getting from your phone's front-facing camera is a face? The tech is somewhat similar to how you can scan VR codes or how primitive robots can see where they're going (see my post from last week). They call it computer vision.

But seeing as human faces are markedly more complex than barcodes or wall corners, you'd think that the method to get a computer to recognize individual ones would be more complicated, no?

Recall that a computer cannot truly recognize color--it only reads binary assignments for specific color values of each individual pixel of the image. Essentially, the computer looks for the contrast of light and dark color values to discern whether or not the image is a face. For example, the bridge of your nose typically appears as a lighter shade than the sides of your nose, so the computer will pick up on that and tell itself, "Yep, that's a face." In this way, the computer uses something called the Viola-Jones algorithm.

Next, the computer needs to figure out your facial structure so it knows where to situate the virtual flower crown on your head or where it should put those god-awful dog ears. It does this with an active shape model. In succinct terms, the computer knows where your eyes, forehead, chin, etc. should be because the programmers manually marked the location of such features on a plethora of models' faces and then ran them through the computer as examples. So since humans have more or less the same facial structure (think broadly), the computer has a pretty good repertoire of faces to go off of.

After plotting the location of your features like a map, the computer creates a mesh--that is, an augmented reality mask that will move as your face does an adjust itself when you, say, open your mouth to let tomatoes spill out.

That's the gist of it. I must say that after reading about how it works I've garnered a bit more respect for the creators of these Snapchat filters. It is pretty intriguing once you see how it works.

This facial recognition software is pretty cool--but there is a dark side. Facebook, for example, has begun living up to its name by amassing a database of millions of faces based off of when people tag their friends in photos. That's just a bit creepy. Even worse, the federal government can do the same thing, which should be more than a little troubling.

Sources

• Kenyan Muslim Antichrist Who Wants to Take Our Guns w/ Dog Filter
• The Incredible Underlying Technology of Snapchat's Selfie Lenses--DigitalTrends.com
• "How Snapchat's filters work" by Vox--YouTube.com

How Robots See

I can see you.

The idea that the field of robotics might one day become so advanced that robots can function virtually the same way as living organisms has long been the subject of a plethora of science fiction films and novels. While robotics has indeed made significant strides, one impediment to its further advancement is the fact that robots are as of yet still unable to truly see the world, at least totally in the sense that humans can. But let's take a look at how most robots are able to process the world with our current technology.

So, just how do humans see? In abridged layman's terms, we use our eyes to collect light that reflects off of the matter around us. The eyes then convert that light into electric signals that travel to the brain via the optic nerves. Obviously, the brain does the heavy lifting here--and some researchers have postulated that up to 50% of our brain mass is involved, one way or another, in the process of seeing. The brain, then, processes those electric signals into valuable information about our surroundings for us.

Therefore, it is no surprise that enabling a robot to gather information about the world in this way, just as animals do, would be largely beneficial to advancing robotics.

Currently, technology allows for robots "see" the way you probably think they might: A video camera is used to collect a constant stream of images, which is then passed to the computer inside the robot. From there, a few different things can happen.

Roboticists use features in the stream of images--say, corners, lines, or unique textures-- to let the robot "see." The features are then stored in a library. They then write code that will recognize the patterns in these features to help the robot comprehend what's around it.

This code forces the robot to evaluate the information it receives from its cameras and compare its features with those it has stored in its library. So if a robot has a feature that looks like the corner of a room in its library, then it ought to be able to interpret another corner for what it is.

It's a somewhat laborious and complicated process, but it is definitely efficient.

Sources

• How do robots 'see' the world?--the conversation.com
• C-3PO--Flickr.com

How Fitbit Works

It's the brand new craze that's sweeping the nation--Fitbit! I'm sure most have, one way or another, come across these personal fitness trackers. Marketed as a way for the average consumer to keep a close watch on their daily activity, a Fitbit is a watch or wearable clip that monitors the steps you take, the calories you burn, and (depending on the version you own) your heart rate. Downloading the Fitbit app for your smartphone takes things a step further: If you sync the app with your Fitbit, you can scan the bar codes of your food and count the calories you consume, log in what you do when you go to the gym, and even challenge friends to walk more steps than you. More recently, Fitbit has come out with a sleep tracking feature.

Now, I know humanity's made some pretty decent technological progress in the few hundreds of thousands of years we've been ambling around this planet. We're in the Digital Age. We've put men on the moon, split the atom, and invented Hot Pockets. But for a long while, I was skeptical about Fitbits. How could a piece of plastic count how many calories I've burned? So I looked it up, and here's what I found.

To track steps, Fitbits use something called a three-dimensional accelerometer to track the user's movement as well as the intensity of that movement. It's quite similar to what's used in Wii remotes. This, simply, is how Fitbit tracks steps. However, this raw accelerometer data is pretty useless on its own. Fitbit relies on special algorithms to interpret it into something useful--the caloric and perambulatory information we so crave.

It seems that the engineers of Fitbit had to resort to plain old trial and error in order to finely tune the algorithm that the devices use. They compared Fitbit's algorithm to that of other, more established test machines in order to see how well it worked. For example, when the engineers were developing the feature assessing how many calories the user burns, they compared Fitbit's results with those of a portable telemetric gas analysis system. (The gas analysis system, which is so great an assessor of calorie use that googling it yields mostly scholarly articles I haven't got the time to peruse, analyzes gas composition as we exhale.) Fitbit then takes into account your basic metabolic rate (BMR), which includes heart rate and brain activity, and adds it to the data collected from the accelerometer to calculate the calories you burn.

Fitbit's sleep tracker is similar to its step tracker; it merely logs whether or not you're moving. So, if you're wearing your Fitbit while you're in bed but you can't fall out, the Fitbit will assume you're fast asleep. Well, I guess that's the thing about technology: There's always room for improvement.

Sources

• How FitBit Works--HowStuffWorks.com
• How does Fitbit estimate how many calories I've burned?--Help.Fitbit.com
• Fitbit image--Fitbit.com

The Boolean Pythagorean Triples Problem & How It Was Solved

It should come as a surprise to no one: Computers are good at math.

So good, in fact, that they can solve a problem that mathematicians have been trying to crack for thirty years in just two days.

Such is the case of a problem solved only a few months ago called the Boolean Pythagorean triples problem. A mathematician named Ronald Graham first offered a $100 prize to anyone who could solve it in the 1980s, and no one was able until May of this year. The Stampede supercomputer at the University of Texas did the job, which just so happens to be the largest-ever math proof in the history of ever: 200 terabytes.

Stampede, our heroic supercomputer

So, what's the essence of this colossal and seemingly intractable math problem? Well, you probably remember the unsuitably-named Pythagorean theorem from middle school: a^2 + b^2 = c^2. The integers a, b, and c in the Pythagorean theorem are known as Pythagorean triples. The problem asks whether it's possible for each of the positive integers can be colored either red or blue so that a combination of the Pythagorean triple can satisfy the Pythagorean theorem without having any of the integers be the same color.

In May, researchers at the University of Texas had Stampede attempt to solve the problem, and it spent two days working it out. (Now just imagine how long it would take a human, or an army of humans.) Stampede determined that yes, this is attainable. Another program even checked Stampede's resulting 200 terabyte proof and found it to be sound. 

Answering the question posed by the Boolean Pythagorean triples problem only raised a veritable legion of different questions, however. For example, the question remains of just why it's possible for all of the integers to not have the same color. Also, it's apparently only possible to color the integers differently for 7,824. After the 7,825th integer, it just becomes impossible. Why is that?

I think perhaps this just speaks to our own relative lack of understanding of how the problem really works. But, like any science, it'll probably take dozens of minds and years of collaboration before humanity as a whole has really increased and improved our knowledge base. Really makes me wonder how long it would've taken people to figure the problem out without the use of a supercomputer, though.

Sources

• World's Largest Math Proof Solved. And It Takes Up 200 Terabytes. From Futurism.com
• Image Representation of the Problem's Solution. From Nature.
• Photograph of Stampede

What're the Little Gold Chips on Our Debit Cards For?

You've probably had the awkward experience of purchasing an item at a store, swiping your debit or credit card, and having the cashier tell you, "Please insert your chip." Or maybe you've inserted your card's chip only to have the cashier say, "Please swipe." 

Actually, you've most likely experienced both instances, assuming you buy things at stores and don't dumpster-dive or shoplift or subsist solely on manna from heaven. If you're like me, you've probably wondered what the little golden chip on my debit card was, how it works, or if people are just full of hot air when they tell you it's "safer."

So I googled what the skinny on these chips were figuring it had something to do with computer science, and (for once) I was right. Here's how it works, and how it apparently protects us consumers against fraud:

Interestingly enough, it turns out that much like marriage equality and affordable healthcare, cards with chips are a global trend that America has been slow to adopt. They've already been used for years by over 80 other countries. They're called EMV cards (the acronym stands for Europay, MasterCard, and Visa) or smart cards. (They might be smart, but are they smart enough to know why kids love the taste of Cinnamon Toast Crunch? I doubt it.)

Cards that use magnetic stripes on the back--that's how we've normally been doing it--store data that stays exactly the same with each new purchase. Therefore, any old counterfeiter with access to that data could use it to uncover enough information about the cardholder to make purchases under their name, which would not be all too favorable.

But EMV chips are different. Every time it's used for a new purchase, the chip creates a new transaction code that can never be used again. So even if a hacker gets his or her hands on that sweet, sweet, data, it won't matter a lick, because the next time the customer buys something, the card will use an entirely new set of data.

"The introduction of dynamic data is what makes EMV cards so effective at bringing down counterfeit card rates in other countries," says Julie Conroy, research director for retail banking at Aite Group, a financial industry research company.

I really hadn't a clue about any of this stuff before reading about it. Now I'm pretty okay with inserting a computer chip on my card, even if it does take a little longer than swiping a mag stripe. I guess this all just goes to show how important--and powerful--data is.

I bet this teen social outcast from the 90s who's brandishing her credit card really wished she could travel to the mid-2010s and use an EMV chip card, because clearly so much of her money's been stolen by counterfeiters that she can't even afford an undershirt. Shame, really.

(I just wanted to use a weird stock photo, okay?)

Sources

• CreditCards.com: 8 FAQs about EMV credit cards
• YouTube video: How do EMV Chip Cards work?
• Visa card example image
• Social outcast teen displaying her credit card

New Computer Program Can Identify Communication Problems in Children

Most people I've talked with have probably notice at some point or other that I've got a speech impediment: a stutter. I won't blame anyone if they haven't noticed, however, because the darn thing likes to come and go as it pleases and manifests itself sporadically.

It has, at times, made some things (ordering at drive-throughs, for example) more difficult for me, but for the most part I'd say I've got it under control. This is due in part to a few years of speech therapy I went through in grade school. Taming it might've been easier to do had I started therapy earlier. Unfortunately, upwards of 60% of kids are like I was--they aren't diagnosed with their speech or language disorders before years after kindergarten, and as a result get less time to benefit from treatment.

That's why I'm intrigued by the work of researchers at the Computer Science and Artificial Intelligence Laboratory at MIT and Massachusetts General Hospital’s Institute of Health Professions.
Earlier this week at Interspeech, a conference on speech processing, the researchers reported on the results of recent experiments with a computer system they designed to not only identify language and speech disorders, but also recommend what courses of actions should be taken.

Pretty cool, I guess, but I wondered: Couldn't you tell if a kid has speaking problems by just, well, listening to him or her? Well, that's exactly what the computer system does. It analyzes audio recordings of children reading a story and looks for irregularities in their speech patterns.

To teach the system how to do this, researchers John Guttag and Jen Gong used a method called machine learning, in which a computer searches huge sets of data "for patterns that correspond to particular classifications."

That data was gathered by Tiffany Hogan and Jordan Green, scientists who helped design the program. They said it mainly looks for pauses in the child's speech when he or she is reading.

The MIT researchers even took age and gender into account when training the computer system using a technique called residual analysis. After identifying correlations between the gender and age of test subjects and the "acoustic features" of the way they spoke, Gong corrected for those correlations before inserting the data into the machine-learning program.

All things considered, this interesting development may prove vital in identifying and correcting people's language and speech problems in the future.

Sources

• Automated screening for childhood communication disorders--MIT News
• Picture

Spit Takes (And It Gives a Little, Too)

One of the breakthroughs of modern science is the mapping of the human genome. We now know a great deal about how our genes--what genes cause what traits, make us more susceptible diseases, and even what parts of the world our recent ancestors came from.

23andMe is the eminent personal genetic service. Since its founding in 2006, the company has grown to become the go-to company if you want to have your genome sequenced. 23andMe is perhaps most well-known for the health information it provides: If a user has a genetic variant that has been proven in scientific studies to be associated with developing, say, Tay-Sachs disease, then 23andMe will notify them. The same goes for other diseases and conditions like sickle cell anemia, lupus, and lactose intolerance. It can be useful for people who want to stay healthy and get the jump on preventing what they me genetically predisposed to. However, much of the health information 23andMe provides has been limited by the FDA, because Uncle Sam won't let anyone have any fun.

Another interesting aspect of 23andMe is its Ancestry Composition, which basically tells you how much of your DNA comes from certain population groups around the world. The company does this by comparing your DNA to that of people in 31 "reference populations"--that is, people whose families have supposedly lived in the same region for the past half millennium or so. It's only an estimate, but it's widely considered accurate. Here are my results, and I think they're pretty accurate considering what I know about my own family history:

(Once I found out 2/3 of my genome comes from African populations and 1/3 from European ones, the oreo jokes lobbed at me in middle school took on a whole new meaning.)

I was interested in how they do this, so I did some digging and found that (of course) a lot of it involves computer science.

So, how does it work? Well, first, 23andMe sends the customer a kit, and then the customer provides a saliva sample and sends it to one of the 23andMe labs in California or North Carolina. At the lab, the scientists extract DNA from the cells in the spit, determining exactly which base pairs (A, C, T, or G) the customer has along each of their 23 chromosomes. This raw data can be downloaded and viewed in the terminal, however as a layman I find it totally useless without it being interpreted.

23andMe interprets genetic code for its Ancestry Composition using a modified version of the (apparently) famed computer program, BEAGLE, which it calls "Finch." First, 23andMe compares the customer's DNA with that of "a set of 10,418 people with known ancestry, from within 23andMe and from public sources." This is called "phasing."

After phasing, 23andMe uses something called a support vector machine (SVM for short) to determine which chunk of your DNA most likely originated in which population group. For example, the SVM clearly recognized that most of my DNA is most common among people living in Sub-Saharan Africa, so it gives an estimate of that amount.

Next, the 23andMe scientists move on to "smoothing," "calibration," and "aggregation," a holy trinity of fancy words that essentially means they're just checking and re-checking their work. And then, voila! The user receives an email that their reports are ready and can then view them online.

23andMe could definitely use some improvement, such as larger African and Asian reference populations. Still, I marvel that something like this is available to us. It's practically magic.

Sources:

• Ancestry Composition: 23andMe's State-of-the-Art Geographic Ancestry Analysis
• 23andMe: Wikipedia
• 23andMe Logo