What color is human skin?

When I grew up, there was only one answer to the question "What color is human skin?" At the time I was born, Binney and Smith [1] had one crayon (called "flesh") for skin tone. Then in 1962, the company renamed the "flesh" crayon "peach" [2]. Then there was the Civil Rights Act of 1964. Finally, in 1992, Binney and Smith introduced their Multicultural Crayon set. This set includes six crayons that represent an actual range of skin tones: apricot, burnt sienna, mahogany, peach, sepia, and tan. White and black are included for mixing. Or maybe they were added because of the "eight crayons in a box" box rule. Either way, no one really has pure white or pure  black skin.

Photo egregiously stolen from Dick Blick website

The Humanae Project

I was excited this week when I found out about the Humanae Project. A lady from Brazil by the name of Angelica Dass has taken on the task of collecting images of a zillion people [3] to catch their skin tones. A small area of the image of each face was averaged and Photoshopped in as the background.  

Six of the hundreds of people who have volunteered for the project

How could I resist doing a little math on these images and blogging about it?

Visual look

At the time I looked, there were 420 images. I pulled the RGB values from the background of each of these images and went at it. The first look is a montage of all the colors. My impression? Each little box looks like a skin tone, but it does seem a bit light on the dark - that is - the darker skin tones seem under-represented.

A collection of 420 real skin tones

Here is another look at this data. In the graph below, each dot stands for one person's skin tone. The horizontal position of the dot gives the red value of the color, and the vertical position gives the green value. One thing that the graph shows is that there is a very definite set of colors that qualify as skin tones [4].

Red and green values of all skin tones in the collection

When I look at these graphs, I also see that, for the most part, the collection of skin tone colors form a nice line. Well, a nice crooked line. And maybe the line is kinda fat. Still, this says to me that you could make a pretty decent approximation of all the skin tones by assigning each of the skin tones to a single spot on that crooked, kinda fat line. To put this a different way, if you were to collect the 420 people in a room and ask them to line up in order from darkest skin to lightest skin, they would make a more or less smooth transition

I have done just that in the image below. Each person has a narrow vertical strip, and all the strips are arranged in order according to the average of the red, green, and blue values.

Another collection of 420 skin tones
"Everyone line up in order from darkest to lightest!"

If you stand back and defocus your eyes a bit, this looks smooth. This shows that skin tones form a line. But, if you look closely, many of the individual strips can be seen, contrasting with their neighbors as being perhaps redder or perhaps greener. This shows that the line is kinda fat. People with the same average lightness of skin vary a little bit on the hue of their skin.

This is consistent with a paper that is a lot more rigorous than this blog post [5]. Their conclusion is that one number is enough to characterize around 95% of the information in the spectra of skin. Well, I didn't need none of their sophisticated "principle component analysis" to show the same thing, did I? Show offs.

Motivation for the rest of this blog

I am about to dive into the deep end of the mathogeek pool. Before I do, let me provide some idea of the various applications--to explain why it might be worthwhile to swim in those slide-rule-infested waters.

Application #1 - Suppose someone wanted to generate skin tones of various ethnicity, maybe for a game or to create random avatars. Having a simple parametric equation that describes a wide range of plausible skin tones would be a great way to do that. The word "parametric" in the previous sentence means that you could randomly select a parameter (call it "k") that would be in some range of values (I dunno, maybe from -3 to +3), and the equation would give you an RGB triplet that would be a plausible skin tone.

Application #2 - Suppose someone wanted to find faces in an image. One of several necessary criteria for a pixel to be a face pixel is that the RGB values must belong to the club of plausible skin tones. So, it would be cool to have some sort of equation that describes the set of plausible skin tones, so that any given pixel can be tested for membership in that club.

Application #3 - Suppose someone wanted to characterize someone's skin tone. This characterization might be used to recommend makeup, or to categorize someone as having "winter" coloring. If there were a magic equation with a parameter it would be possible to find the particular value of that parameter which best characterizes the skin for that person. That parameter would be the characterization of that person's skin tone.

These applications all have to do directly with skin tones. I have a few other applications in mind that would use the technique that I describe.

Application #4 - Suppose some math geek (or more likely, a stats geek) needs a way to describe a set of multi-dimensional data--much like one would use mean and standard deviation to describe single-dimensional data. I describe a method below to determine the ellipse (or hyper-ellipse) that describes multi-dimensional data in a statistical way. Since I am not aware of a word for an ellipse or hyper-ellipse that serves as a proxy for a large amount of data, I will invent the word: proxellipse. The proxellipse is an extension of standard deviation to multi-dimensions.  

Application #5 - Suppose some other geek (maybe a scientist of some kind) wanted to display a scatter plot of two-dimensional data. If the scientist had twenty points in that plot, it would give some appearance of the amount of spread. But, if the scientist had two hundred points, the scatter plot would give the impression that the spread of data is far broader. The analysis below is an alternative. By displaying a scatter plot along with the proxellipse, one won't be misled by the crowd effect. The proxellipse would show the boundary in which a certain percent of the data will likely fall (with all the normal assumptions about whether the data is normally distributed).

Statistical look

The graph below is the same data as the red/green plot above, only this time, I am looking at red versus blue values. This perspective shows a bit more crookedness. It looks like there are two separate populations: those with darker skin, and those with lighter skin. It looks like two different lines are needed to describe the different sets. 

Red and blue values of same data, with a discontinuity highlighted

So, there is a crooked line in RGB space that defines skin tones. For the sake of simplicity, I am going to start by looking at the statistics of just the brighter dots (where R > 180, G > 120, and B > 110). This reduced the set from 420 data points to 404. To be clear, I am quite literally discriminating here on the basis of skin color, excluding the darker skin tones. I apologize, but the darker tones belong to a different statistical distribution. 

Now it is my turn to show off some really golly-whiz bang math. This is a green/blue view of the segregated data points, with a red line showing three axes of an ellipse. This ellipse is the proxellipse (with coverage of 3), which is an ellipse that is basically in the same shape, size, and orientation of the data.

Green and blue values of skin tones, with the axes of an ellipse shown in red

Now I'll say a bit more about the proxellipse. It's a statistical thing. Essentially, I have extended the idea of standard deviation to three dimensions. The size of this proxellipse is three standard deviation units in each of the directions. Now, I assume that I am not the first to discover the technique, but it would appear that this technique is not well known in the color science community [6]. Or maybe it's just a dumb idea?

Getting back to the kinda fatness of the line. The degree of kinda fatness can now be quantified. To demonstrate that the data points here are very close to being a line, the length of the major axis is 28.8 gray values. The other two minor axes are 5.3 and 3.3 gray values. For those who don't recall, 28.8 is much bigger than 5.3 and 3.3.

An algebraic look

Let's just pretend that someone wanted to come up with an equation that could be used to generate a sequence of reasonable skin tones. I called that Application #1. One approach would be to apply linear regression to any of the graphs above. One might, for example, determine a best fit function for G as a function of R, and another regression would determine the best fit of B as a function of R. In this way, the value of R is a parameter that can be used to determine the other two color coordinates.

This may work, but I have some reservations about this technique. First, such a regression treats R as an independent variable and G as a dependent one. Really, the two should be in the same category. It seems like something must go wrong. (OK, that's just a philosophical argument.) 

Second, the choice of expressing G as a function of R, for example, is a bit problematic due to the relatively steep slope. A small change in R will cause a large change in G. And (the important part) a small change in the random data will cause a large change in the slope that is calculated through regression. That's a bummer.

Third, I have a little known fact about statistics. Garden variety linear regression starts with the assumption that there is no noise in the independent variable. That is to say, it is based on the assumption that we know the R values perfectly, and any deviation from a straight line is strictly because of random variation in G. Now here is my little known fact: If you add random noise to both of the variables in a linear regression, the slope of the regression line will move toward zero. The noise in the independent variable adds a bias.

So... how about another approach? Using my proxellipse analysis, I arrived at the following equations for the RGB values of the skin tones:

     R = 224.3 + 9.6 k

     G = 193.1 + 17.0 k

     B = 177.6 + 21.0 k

When k is allowed to go from -3 to 3, this will provide RGB triplets for reasonable lighter skin tones. The following equations will give reasonable RGB triplets for darker skin tones.

     R = 168.8 + 38.5 k

     G = 122.5 + 32.1 k

     B = 96.7 + 26.3 k

I apologize again... I have created separate but equal equations. <sigh>

Caveats

Science?
As a scientific experiment, this is perhaps not controlled enough to be accepted into a peer reviewed journal. Now, I have every reason to believe that Angela has controlled conditions as best as she can. But, she is using a camera, and cameras are not color measurement devices. As far as I know, she has not provided an image with a Munsell color checker card that would serve to calibrate the colors to real units. There isn't a statement about the type of camera or the settings. In particular, there is no statement of the gamma setting of the camera. (Gamma is a setting that will increase the brightness of midtones in order to give the picture a better appearance. Most cameras have this enabled by default, and it gets in the way of accurate color measurement.)

But these are not my big bugaboo. RGB cameras (at least almost every one on the market today) do not see color the same way we do. The red, green, and blue filters in a camera do not give the camera the same spectral response of the three cones in the eye. I investigated this in my paper Why do Color Transforms Work?

Also, the RGB response differs from camera to camera, so as they say, results may vary.

Ideally, the purist in me would like to see spectral data on everyone's skin, but the purist in me is too darn mired in the details to ever get a blog out. And besides, the images on her website do all look like reasonable skin tone on both of the computer monitors that I routinely use.

Skin blemishes and goniophotometry and umbrophotometry

This analysis is rather simplistic in that it associates one single RGB value with the color of a person's skin. That's just plain silly, for two reasons. First, skin is not uniform in color, especially as one gets older. Second, the color of the skin (and just about anything for that matter) depends on the angles that it is illuminated from and the angle from which it is viewed. If the lighting angle, surface orientation, and the camera are all at the right angle, you can see a very white specular reflection on a surface. This effect is called goniophotometry, the measurement of light as a function of angle.

The third effect is that there are generally shadows on a person's face. Even when illuminated diffusely, there still may be (for example) an area under the nose [7] that is darker because of the shadow. Clearly this effect is accentuated in some people. I have just this moment coined the word "umbrophotometry" to characterize the measure of this effect. "Umbra" means shadow, and is the root of the word "umbrella".

--------------------------------------------
[1] There used to be a company named Binney and Smith. In 2007, the company name was changed to Crayola. Clearly if I talk about stuff they have done after 2007, I should refer to them as Crayola, but how do I refer to the company back before it changed its name? Is it Binney and Smith, or Crayola? And when I want to talk about the guy who won the 1985 Grammy for the Album of the Year, do I refer to him as Prince, the Artist Formerly Known as Prince, or the Artists Formerly Known as the Artist Formerly Known as Prince? [8]

[2] Personally, I don't think that "peach" is quite the right name for that crayon. Maybe I'm wrong, but I think that peaches are more of an orange color than the crayon. I don't have a better suggestion, mind you. I just like to kvetch.

[3] I could have said that she is taking pictures of a brazillion people. Give me credit for not going for that pun!

[4] I am being a bit loose here, since I have only shown the view of R and G. The other views are pretty similar, though.

[5] Sun and Fairchild, Statistical Characterization of Face Spectral Reflectances and Its Application to Human Portraiture Spectral Estimation, Society for Imaging Science and Technology, Volume 46, 2002
http://www.cis.rit.edu/fairchild/PDFs/PAP14.pdf

[6] ASTM 2214 (2002), Standard Practice for Specifying and Verifying the Performance of Color-Measuring Instruments (Section 6.1.1 anticipates my technique. They are discussing a way to evaluate the variability of a collection of color measurements and state:

"Since color is a multidimensional property of a material, repeatability should be reported in terms of the multidimensional standard deviations, derived from the square root of the absolute value of the variance–covariance matrix."

Egg-headed stuff, indeed. If only they knew my method when they wrote that!

[7] George Carlin informed a generation of people that this part of the body is called the philtrum.

[8] This was a trick question. The correct answer is "Lionel Richie". Prince's album Purple Rain was nominated in 1985, but Lionel Richie's Can't Slow Down won the Grammy. 

Lousy weather every weekend

It was a Monday after a rainy weekend. Naturally, since I had to go to work, the weather was excellent. I was particularly downcast since the weather had put a damper on the previous weekend as well. I mentioned it at work, and got a surprising response.

Actual photo of me enjoying the lovely weekend weather [1]

I should tell you that this was quite a long time ago, back before Al Gore invented the internet. I was working at the University of Wisconsin Space Science and Engineering Center. You would think that with name like that, we would all be working on something dull like a space telescope, a Mars rover, or a space shuttle porta-potty. But it was far more exciting. We were crunching data from weather satellites.

Getting back to my story about a forlorn math guy complaining about the weather, when I made my comment about two weekends lost to lousy weather, I just happened to be surrounded by meteorologists. I was told that everybody knew that weather was cyclical, and it tends to follow a weekly pattern. If you know the weather on one particular Saturday, then you have a darn good guess at the weather the following Saturday.

I have used this factoid frequently. It always sounds impressive when I tell people what next weekend's weather is going to be like. I should say, it sounds impressive when they believe me. Whether my prediction comes to is actually irrelevant. No one remembers my predictions.

Is the "weekly weather pattern" factoid for real? I put it to the test.

The data  

It isn't hard to find information on the current weather conditions. But yesterday's weather? That's a bit harder to find. And historical data? I found  a place to dig up old weather. [2]

I downloaded four year's worth of weather data from Milwaukee; from July 22, 2009 to July 22, 2013. Why four years? Well... there is a story there. Once upon a time, there was a computer magazine called Byte. It was the magazine for computer geeks. They published an article with a BASIC program for doing a fast Fourier transform. I dug into the code and found it wasn't remotely the FFT (fast Fourier transform) algorithm that Cooley and Tukey made famous. Later, they had an article that looked at sales data that spanned many years of time. They said that a "four year analysis" was used to look for trends. I dropped my subscription. [3]

Mean temperature, Fourier anaylsis

Let's look first at the temperature (see plot below). It is apparent from the sine wave kinda thingie in the plot that I have collected four years' of data, and that the start of the data is kinda mid-summer.

Average daily temperature in Milwaukee over the past four years

That graph is and of itself is pretty cool, but playing with the data? I don't know about other math guys, but the first thing I like to do when I get a new fresh pile of data is to start taking Fourier transforms. It took a bit of massaging of the raw FFT output, but here is a graph showing a section of the frequency data. This graph shows the strength of the periodicity going from a four day period (on the left) to a 10 day period (on the right). There is no discernible peak or bump near the once-per-week mark.

FFT of the mean daily temperature

Correlation

Here is another look at that same data. In this case, instead of doing the whole Fourier bit, I did some correlations. I check to see if the daily temperature on a given day correlated with the daily temperature on the previous day. So... an array of four years of data, 1462 data points. I computed the correlation coefficient between two sub arrays, one going from day 1 to day 1461, and the other going from day 2 to day 1462. Then I computed the same thing, only with a lag of two days, and three days, and so on.  

The graph below shows the results. At the very, very left, the correlation between each day's temperature and that same day's temperature is 1.0. Well, duh. The temperature on a given day looks a heckuva lot like the temperature on that same day. The plot below shows that today's temperature also looks a lot like yesterday's, with a correlation coefficient of close to 0.95. In the middle of the chart we see the correlation of today with one week ago. According to the hoity-toity meteorologists, I should see a big spike there, but no. Using the temperature a week ago to predict today's temperature is good (r = 0.85), but it is no better or worse than using the temperature six days or eight days ago.

Correlation coefficient of temperature as a function of spacing (in days)

So, I would say this myth is pretty well busted. That's the way I like my myths: pretty and well-busted. Temperature does not follow a hebdomadal pattern. That's a pity. It would have explained why a week isn't six days, or (shudder the thought!) eight days.

But wait! What about rain?!?!?

I missed something here, didn't I?  This blog post started out talking about rainy, not temperature. Maybe I should have a look see at the rain data? Luckily, this database contains precipitation data, as well as wind speed and direction, barometric pressure, humidity, ... but all I want is the rain data.

I present below another correlation chart, this one showing the correlation of precipitation. There it is, as plain as the nose on Owen Wilson's face, a spike at lucky seven. There is a correlation (r = 0.1). This is roughly in the range of 99.95% significance range. So. Maybe my meteorologist friends weren't all that dumb after all?

Correlation coefficient of precipitation as a function of spacing (in days)

I should make a comment here about how big the number 0.1 is when it comes to correlation coefficients. Let's just say that Norm decided to make some money on these results. Every night, for four years, he would sit at a bar in Milwaukee, and take bets on whether it would rain the following week. I know, ideal job, right? If Norm follows the advice in this blog, betting that the rain today can predict that of one week from today, I guarantee that he would come out ahead at the end of the four years.

Norm, pondering this money-making scheme

On the other hand, the number 0.1 is very tiny. The predictive strength when r = 0.1 is on the order of 0.005. That's an indication of how much the odds are in Norm's favor. He is pretty sure of making money, but he needs to make a lot of bets to get there.

This may sound like a bad business model, but this is how casinos work. They want to run games where the table is tilted ever so slightly in their favor. Too much of a tilt (too low a chance of the patron winning), and people will walk away. Too little of a tilt, and the casino won't make a buck.

Another comment on the plot above. There is also a peak out at 29 and 31 days. Hmmmm... Maybe this is the effect of the moon?

---------------------------------

[1] If truth be told, although I have sung before, and have been in the rain, but this is actually a picture of Gene Kelly, not me. If more truth be told, Jean Kelly is my sister. Here is my favorite painting of hers.

[2] If truth be told, I didn't find this website on my own. Nate Silver told me about it. If even more truth be told, he didn't actually tell me in person. I read it in one of his books: The Signal and the Noise: Why So Many Predictions Fail — but Some Don't. He and I are good buddies. At least we would be if we ever met. That's my prediction.

[3] I had a house that I was getting ready to sell. The realtor told me that the entry way needed work - first impressions and all that. Updating this would give the whole house a new image. So, I gave it a fresh coat of paint, updated the light fixture, and polished up the handle of the big front door. I guess you could say that I performed image enhancement through a fast foyer transform. Some of you will find this joke incredibly funny.