Scribbling away that scratch on my car

I received an interesting comment/question on a LinkedIn thread about the Scribble pen. Answering this question provides me the opportunity to touch on a number of topics and color science, as well as the opportunity to show off just how incredibly smart I am. Since both of these are high on my list of things to do, I decided to dedicate a blog post to answering his question.

Malcolm

Malcolm Cutting

Paint & Development...looking for challenge to develop your business. Call me.

Just using imagination here and the touch up pen for the automotive smart repair scratches....now that could be quite possible.....or could it? Worth thinking about.

The short answer

No.

Requirements are tough

Malcolm is one of my oldest and dearest friends. I just met him on LinkedIn about 15 minutes ago. Feel free to draw your own conclusions about my track record with friends. From reading his LinkedIn profile, I know that he has been in the automotive coating business for multiple decades. He knows that matching paint on a car is one of the most demanding color applications, if not the most demanding.

The first issue is that on a scratch, you are trying to match two colors that are side by side. The human visual system (which includes those parts of the brain that are not devoted to memorizing every episode of Laverne and Shirley) is also very good at picking up on coherent anomalies, such as a line.

Juxtaposition accentuates color differences

The second issue is one that most of my readers are aware of. If you are successful enough in life to have access to the internet, then it is almost certain that your car is more expensive than mine. Naturally, we want our cars to look expensive to show that we are indeed more successful than some dufus who turns to blogging to feed his massive ego. Most of us express our identity through the cars we drive.

Would Guy Fieri still be cool in a Saturn?  I think not.

In short, people are picky about touching up scratches on their cars.

It has to be the same color

Naturally, when a scratch is retouched, it has to be the same color. In order to make it the same color, we need to measure the color accurately. Back in the days of Moses and Pythagoras it was done with the eyeball of someone who is both discerning and not colorblind. Ideally, this person would have been more discerning than the most demanding of customers. 

Nowadays, critical color matching is sometimes/usually/always done by expensive spectrophotometers, which are sometimes/usually/always more discerning than the most demanding of customers. I suspect if you go round and chat up the guy who does that painting at the local body shop, you'll find that he needed to go back to college to pick up a PhD in astrophysics (or some other branch of astrology) just to be able to run a spectro.

The kid who paints my bumpers 

The Scribble pen? It uses an RGBC sensor. Four channels. That's it. The standard ISO 13655, which provides the definition for a color measuring device in the graphic arts, requires that a spectrophotometer be used, and that the spectrophotometer must have at least 15 channels, and preferably 31. I ain't know arithmetic guy, buy I think 3 or 4 is less than 15.

In my own personal experience, I spent about two years working with a bunch of smart guys trying to teach an RGB camera to accurately measure color. Here is one of many papers I wrote about this sad experience:

Why do color transforms work? And here is another: Color measurement with an RGB camera. I have seen a lot of claims about this, and it burns my butt like a candle on a toilet seat. If you want discerning color measurement, you can't use RGB to measure the color.

The color must match under any lighting

In order to be happy about the matching of the color on a scratch, most car owners (I think) would want the match to be good wherever the car is. The illumination could be direct lighting from the Sun, diffuse bluer lighting from the clear sky, the incandescent lighting from one of the bulbs in my garage, the fluorescent lighting from the other bulb, and the sodium vapor lighting you see in parking structures. 

This may seem like no big deal, but those who have practiced saying the word metamerism will realize that this is not a given. It is possible for two objects to match under one lighting, but not under another. If you don't believe me, try to guess the color of a vehicle in a parking structure at night. If you want to match a color under all lighting, you will need to do spectral matching and you will need a lot of different pigments.

The RHEM indicator

The Scribble pen, as cool as it is, only has four pigments: cyan, magenta, yellow, black, and white. Or maybe that's five? I'm a math guy, not an accountant. Spectral matching is pretty much out of the question.

Hmmm... Metamerism might be a good topic for a future blog. Look for it at your favorite blog sites.

The color must match at all angles

If you were impressed by my use of the word "metamerism", I introduce another important-sounding word: goniophotometry. If you can slip those two words into casual conversation, you can pretty much bluff your way into any of those wild color science parties that you are always hearing about. Goniophotometry is the measurement of light when you change the lighting and viewing to different angles. When you reposition the lighting, or move your head around, there is a subtle or sometimes huge change in the color of an object.

The simplest example of this effect is something that is so ubiquitous that it probably goes unnoticed. In fact, I am going to guess that I might get some arguments about whether this is even a color change. I'm talking about gloss. When you view a glossy object at the gloss angle, the color of the object changes. 

This is a point where there might be some argument. One might argue that the color of the object didn't change just because of the angle of lighting and viewing. I would argue that the composition and intensity of the reflected light changes drastically at the gloss angle. 

For the purpose of our discussion, gloss is important. Would you say that a touch-up paint matches another if there is a difference in gloss? I think not! Once again, as cool as the Scribble pen is, it does not have a mechanism to adjust the gloss of the ink.

Metallic effects are another example of goniochromism (when color depends on angles of lighting and viewing). Cars often have a metallic paint. This is achieved by embedding flakes of metals within the paint, laying the paint down in thin layers to make sure that the flakes land flat, and polishing the heck out of it between layers. (This is my understanding... I am open to hearing an explanation from someone who has actually painted a car before.)

Painting a car is an exacting science

The Scribble pen, as cool as it is, does not include any metallic flakes. Nor does it include a polishing thingie. So, I am gonna guess that metallic effects might be a bit tough with this pen.

The creation of gloss and metallic effects are two areas where this pen, as cool as it is, is gonna fall short of the requirements for fixing my paint job when some mathaphobe keys my car. But I skipped over another missing feature. The Fix-The-Scratch-In-My-Car pen must also be able to measure the goniophotometric properties of the existing paint. Some cars are metallic and glossy. Some are just glossy. Other cars, particularly those left outside in Arizona, have lost some of that gloss. Some cars, such as mine, which reside in areas where they use salt on the streets to melt ice, have a glorious texture where a dull rust color is interspersed with the drab shade of pale lavender greenish orange. 

As I understand it, the big car companies use a goniospectrophotometer for quality control of their paint process. Such an instrument runs something over $100K and measures the spectra of reflected light at a zillion and a half different angles in order to make sure the color is correct from all angles. These are run by more astrophysics PhDs.

Goniospectrophotometer

Now, I could have that wrong. Maybe the big car companies use an abridged goniospectrophotometer, such as X-Rite's MA98, for quality control. Rather than a zillion and a half angles, these instruments make measurements at maybe 20 or 30 different angles. These instruments probably make more sense on the production line, since the measurement time is on the seconds, rather than minutes or hours.

The Scribble pen, as cool as it is, probably does not include even an abridged goniospectrophotometer. If it did, the price would be pretty cool, since the abridged gonios cost around $25K. 

Still more

There are a few more requirements for a Fix-The-Scratch-In-My-Car pen. Someone should probably mention that there is a difference between ink and paint. Ink is transparent. Paint is opaque. You want to paint your car with paint, since the bare metal is not all that nice looking.

Someone should also mention that the paint has to be durable. Durable enough to stay put under the effect or driving rain, beating sun, slush, and sandstorms. My wife also tells me that there are these places called car washes where these sudsy rollers come out to rub the grime off your car. The only place I am aware of like that has rollers who are wearing bikinis.

And then someone might wanna mention that it would be good if this pen could avoid scratching the car? I don't have all the details on the Scribble pen, but one possible implementation of the inking mechanism has an ink jet head spraying the back side of a ball point.

Conclusion

Malcolm, I think you got a cool idea there. I'm thinking the Scribble pen might not quite be up to the task, though. But, if we loosen up some of the constraints that came from making this device a handheld pen...

Scribble Pen, a real Color Picker Pen?

I wrote a blog post a while ago called "Color Picker Pen". This pen had a color sensor and could change the color of the ink to match. There weren't a lot of technical details available on the pen, but what I read was enough to convince me that this pen is fiction. I recently found out about another pen that's under development which just might be real. I could find nothing suspicious in their press release and emails, and it seems like the miniaturization technology might just be available today.

Meet the Scribble. Ok,.. cute name, but is this one real?  The website says that they will be having a Kickstarter campaign. I sent an email to their "info" email address, and I got a human-generated reply. So far, so good. 

The Scribble pen

The website and email provided some technical details. The pen uses an ARM 9 processor, has a rechargeable 325 maH lithium battery, 1 Gb of memory, and communicates through Bluetooth or micro USB. At the very least, this shows that they have actually through through some of the details.

Unlike the Color Picker Pen, they also demonstrate some knowledge of basic color science. The Scribblers had the foresight to include a white LED as part of the sensor. They use cyan, magenta, yellow, and black inks instead of the RGB inks in the Color Picker. And another detail shows that they have (at the very least) thought through some details here is that they also mention the use of white ink.

Why white ink?

Why do they add white ink, you may ask? My desktop printer doesn't use white. The printing press that printed my copy of the Edmund Optics Swimsuit Issue used CMYK inks, but no white. Why would this fabulous pen need white?

There are two reasons why white is used when mixing stuff to get colors. White flood-coat is often used on clear packaging to back up the standard inks. This provides opacity that is not found in the substrate or the inks. Much like writing with a red pen on black paper, the ink wouldn't show up without a base of white. But, I suspect that this isn't the reason that the engineers of the Scribble Pen added the complexity of white ink.

If I want to print a pink highlight with an ink jet printer or on a web offset press, the approach is pretty simple. Don't put much ink down. Just a dribble of magenta ink, or just a sprinkling of tiny halftone dots. But if you are searching for that same color to paint your wall, you need a different solution. You need the same volume of paint, just less color. When mixing up a can of paint, the paint mixologist first reaches for a can of white base. 

You see something similar when you can look at the Pantone fan deck, which includes the recipe for each of the thousand-plus colors of ink. Pantone 217, for example, calls for 1 part of rubine red ink and 31 parts of something called "transparent white". Transparent white is unpigmented ink, and would look transparent (not actually white) if printed by itself.

The Scribblers may actually be using ink that contains white pigment (titanium dioxide or zinc oxide, maybe). But I think it's more likely that it is more likely that they use a transparent white ink. It would be neat to write on black paper with a pen, but I would guess that it is nearly impossible to get any sort of decent opacity with the amount of ink dispensed with a pen. Think about how much "ink" that a WhiteOut dispenser spits out.

Color sensor

I was told a tiny bit about the sensor in my email from Scribble. It is an RGBC sensor. This means they collect light through a red, green, and blue filter, like practically every digital camera in the world. But they add a "C" filter, which mean "clear". That is, the fourth channel measures all the light together: red, green, and blue.

Why RGBC, you might ask? Digital cameras with this fourth channel are a relatively new development. Here is an article about the RGBC imager in Google's Moto X cell phone. The big selling point for the extra channel is that you get better sensitivity when the light is low, since the C channel sees about three times more light. 

This is a trick that the human eye plays. There are actually four light sensors in the eye. We have the equivalent of red, green, and blue sensors (the rods) that we use when the lights are bright. When the lights get low and the mood romantic, we make use of our rods, which are more sensitive to light. It's a cool design, but it does mean that we can't see color very well at night. 

Just like black and white TV, the rods can't distinguish color. The Moody Blues lamented this at the end of Nights in White Satin:

Cold hearted orb that rules the night,

Removes the colors from our sight,

Red is gray and yellow, white,

And who decides which is right,

And which is ... an illusion.

A legendary album about color perception

Which makes me wonder why they chose that sensor rather than a straight RGB sensor. I mean, they want to measure color, right? I have a few thoughts, but obviously I can't speak for the designers.

The whole "better performance at low lighting levels" thing might have been the motivator, but I'm not sure that's such a big deal. If I am taking pictures of my buddies getting drunk at the bar, I want high sensitivity to low levels of light because the lighting in the bar is low. But the Scribble pen has a white LED, so they should theoretically have enough light. If the sensor picks up a low light level, then it's likely to be a dark object. 

Then again, maybe dark objects are where the four channel can really help?

Another possibility... RGB sensors aren't good at measuring color. Yes, really. They're not. I have a whole bunch of technical papers on the topic, including this one Why Do Color Transforms Work?  Having a fourth channel might help a tiny bit, but probably not much.

Here's another thought. Years ago before computers were ubiquitous, color separation was done as an integral part of scanning. The image through a red filter would determine how much cyan was needed, the image through a green filter would determine magenta, and the image through a blue filter would determine the yellow. Black ink was determined by using no filter.

I'm thinking that might not be the reason that the Scribblengineers decided on an RBGC sensor, though. My explanation is pretty arcane, and since they have a computer in this device, they could come up with way much more better algorithms.

So, I don't know why they chose this sensor. I'm not saying it's a bad idea. It might be slightly advantageous. Or maybe it's cost effective. I dunno. 

How about the ink mixing?

One thing that puzzled me was how the ink was mixed. The email I got from Scribble was intentionally mute on how they do this part, due to patent stuff. But, it seems like a lot of mechanics to be stuffed into such a little package. I am not an expert here, but I wondered whether it is even possible.

So I did a little patent searching. I found an interesting patent, that is, if patents can be called "interesting".  I have no information about whether the good folks at Scribbler do anything like this particular patent. I have no idea of whether they know of this patent. It's from a little company called HP and is titled a "Writing instrument with user-controlled ink color". Ink jet heads from at least three inks (CMY) feed into a tiny mixing chamber, where it is transferred to the roller ball. 

Pen from US patent #6,749,355 

This patent was filed back in 2002, so evidently twelve years ago someone in HP felt that such a thing could be built. Funny thing about patents, BTW. To get a patent, it must be "reduced to practice". Contrary to what you would normally think, this isn't a requirement that you actually build and test your invention. You can "reduce an invention to practice" just by describing it. Writing the patent application in enough detail is a way of reducing the invention to practice. Go figger. So, I don't know whether this HP invention was ever built. HP did eventually abandon the patent, which usually means that they decided it wasn't worth paying the maintenance fees.

Another feature of the HP invention is that one embodiment includes a color sensor. I quote from the patent: 

The embodiment of FIG. 4 incorporates a scanner (151). This scanner (151) is preferably a three element Charge Coupled Device (CCD) array. Each of the three elements detects photons of a particular wavelength, i.e., color. 

Thus, when the scanner (151) scans a color, the three elements of the CCD array will output signals indicative of the ratio of each of the three primary colors in the scanned color. Consequently, the pen (100b) can be programmed to duplicate those ratios of the three colored inks in the cartridge (102) to reproduce the color scanned by the scanner (151). 

Consequently, the user can sample a color with the scanner (151) from any object at hand. The pen (100b) will then write in that sampled color.

This appears to describe many of the features of the Scribble pen. Uh-oh. Does that mean that the Scribble pen would infringe? That's a deceptively simple question. First, you need to look at the claims to determine infringement. Disclaimer: I am not a patent attorney, and I haven't spent the time to do a thorough analysis, but I did notice one thing in the first claim. In order to infringe this claim, the pen must have a button for every color of ink. So, it looks like the Scribble pen as described would not infringe on claim 1 of this patent. But as for claim 2, who knows? I will leave that up to someone else to decipher. 

But, to do a complete "freedom to operate", one would have to do a thorough patent search, and then do this sort of analysis for every claim in every relevant patent that turns up. In doing so, one would probably stumble upon US patent #8,292,527, which lists a fellow by the name of Kia Silverbrook as one of the inventors. I have heard this name before. As of March 26, 2014, Silverbrook has 246 times more US patents than I do. I guess that's sorta impressive. He's got more patents than God, Thomas Edison, and Shaquille O'Neal put together; 4,665 and counting.

I did not spend a lot of time digging up prior art. I would do this if I were to be submitting a patent application (just to make sure the claims were reasonable) and also to get a sense of what can be built without infringing. My meager prior art search turned up quite a few other patents that could be interesting, so I am going to take a wild guess that deciding on freedom to operate might get a bit involved.

There are a few conclusions here. First, it would appear that the technology to do the Scribble pen exists today or will exist soon. This bodes well for the Scribble folks. Second, it appears that the patentscape might perhaps be a bit rough. Without further investigation, it's hard to say whether this will be an issue for them.

Output colors

Ok... now it's time for me to vent. And scratch to scratch my head. Their press release states that the "innovative pen ... can reproduce over 16 million unique colors." Come on. Really? Where does this number come from?  

First, I'm gonna say straight out that the number did not come from a color scientist. I wrote a very entertaining blog about how many colors there are, and then another where I gave the definitive answer: 346,005. I'm sorry. There aren't 16 million colors in the universe, so you can't produce that many colors. That is, if you define "color" to mean "a unique visual sensation". The image below shows nine "unique" shades of pink. If you can see a difference between them, then perhaps you are able to sense 16 million colors.

Nine of the 16 million purported colors

Of course, I'm just being persnickety about the definition of the word "color", but still, it's a BS number. It's likely that the sensor has three outputs which are each eight bits. That would give 16 million possible inputs to the software. That's not the question though. The question is how many distinct outputs there are. How many distinct levels are there for each of the six inks?

That's the end of my rant on 16 million. Now for the head scratching part. Assuming that they computed the 16 million based on 256 levels for each of three channels R, G, and B... What happened to the C channel? Remember? The sensor is an RGBC sensor. Four channels of eight bits would give 4.3 billion colors. 

Or maybe they are talking about the total number of distinct ink mixtures that they can make? That's an interesting combinatorial problem that I don't want to get into right here.

Ok, so the 16 million number is probably marketing hype. Certainly not any sort of proof that the whole idea is bogus.

Conclusion

In the Color Picker Pen post, I came to the conclusion that the pen that was portrayed was not anything real. In the case of the Scribbler pen, I didn't find any red flags. As far as I can tell, they know what they are doing. It  isn't a final product - I mean, they are doing a Kickstarter to fund the product development. But from the limited information that has been made available, it all looks legit.

But, all the usual disclaimers hold. I am not a patent attorney, and I am not an investment counselor. For that matter, my wife would probably tell you that I can be an idiot at times. My personal liability in your investment in the Kickstarter campaign is limited to the amount of money that you paid me for the opinion. So far, I think that's zero.

Color Picker Pen

(An update to this blog post can be found at "Scribble Pen".)

As John the Math Guy, people are always asking me stuff. Often the questions are like "When are you gonna get off that darn computer and fix that leaky faucet?" Or "Why are you so ugly?" These, of course, aren't really question at all, but every once in a while I get a real question. Today, the question was "how does this gizmo work?"

The gizmo in question is called the Color Picker Pen, designed by Jinsu Park. Lemme tell you, it's pretty cool.

You start by holding one end of the pen up to an object and clicking the color scan button. A color sensor measures the color of that object. In the picture below, the pen has picked up the green of the leaf, and note that the color display (along the side of the pen) has changed to indicate green.

Grab a color

So far, this is a cute little gadget. But now the real fun begins. This is a pen, right? The really cool thing is that you can now use the pen to write with whatever color you just scanned. Kind of like the Photoshop eyedropper tool, only it works in the real world rather than in the cyber world within Photoshop. (You know, where all models are a size zero, and don't have pimples or cellulite?)

Draw with that color

Wow!

But I was asked to explain how it works. The answer is simple. It doesn't. Don't get me wrong. It's a great concept, a cool idea, but, at least as shown in the diagrams, it seriously won't work. And, it was never advertised as a real product. Actually, you can find a lot of websites by googling "color picker pen", and many of them refer to this as a "concept", an idea that has not been turned into a real product. 

Internal organs of the Color Picker Pen

The sensor

The sensor for this pen is plausible. There is a color sensor available from MAZeT that would fit the bill. This is a true XYZ sensor, which means that it is a bit better than an RGB sensor in terms of color fidelity. (An RGB sensor will see some colors distorted.)  The sensor itself is shown above... certainly small enough for the color picker pen. Below the sensor is a prototype board. At the right side of the board, you will see a USB connector, so that gives you an idea of the size of the board. If you remove the USB related components on the board - which would not  be needed - you might be able to fit the board into the end of a fat pen.

MAZeT sensor and prototyping board

Of course, a few things might be missing. If I were designing this, I would likely try to add a white LED because I wouldn't trust ambient light, which can vary in intensity by many orders of magnitude. And especially since the user would generally want to push this thing right up against the leaf, effectively shielding the leaf from any ambient light. But this LED might make the whole thing too large, since it would be necessary to shield the sensor from the LED.

There is still one thing that is bothering me about the sensor, though. Below I show a closeup of the sensor end of the pen. Anything funny about it? Think now. What color is the end? It's a dark gray?!?!?  Why would an optical designer put sunglasses over a color sensor? It's pretty dark. Just taking a wild guess, I would say that it reduces the light to the sensor by a factor of 10. Not a good thing.

Close up of the sensor end of the Color Picker Pen

So far, it sounds like this could perhaps be built, but that it is unlikely that the pictures represent a real design.

The display

The Color Picker Pen offers a nice feedback feature: the pen will light up to tell you what color ink is measured / loaded. Is this feasible?  I borrowed a pen that belongs to my wife to show that, yes, such an animal exists. Those of you who know my wife, I would appreciate if you didn't mention that I borrowed her pen. I told her I was going to be fixing that leaky faucet.

Pen for writing love letters in the dark

This pen has a battery and a blue LED. Clicking the switch on the top will turn the light on or off. This could easily be done with a tricolor LED so that a wide range of colors can be produced along the side of the pen.

The battery in my wife's pen reminded me of something else that is needed. In my palpable excitement over this pen, I forgot all about the need for a battery! I didn't see a place for one, or a way to replace it, but let's just say that could be done. Since there is no evident way to replace a battery, it is sounding more like this is a concept design, and not a real product or prototype.

The other electronics

Of course, maybe there needs to be a microprocessor as well? Ok... I guess we can make some space for that. Maybe it would fit? I dunno.

The inks

Now we come to the fun part. The ink. Once again, I have taken a close up of the internal organs shot. Quite clearly, it shows a red, a green, and a blue ink cartridge. That's where the color scientist in me immediately says "nope, not a real product".

Close up of the ink

Why do I say that? Well.... Lemme ask you a question. What color inks are used in your home printer? Red, green, and blue? Nope. Sorry. They are cyan, magenta, and yellow. Below you see the four printing inks from my home printer. Cyan, magenta, yellow, and black. No red, no green, and no blue.

The inks for my printer

Perhaps I just have a cheap printer, you say? Well, that's true, but let's consider the other end of the spectrum. I put my latest Boston Proper sales flyer on my scanner, and collected this image. Note, especially in the white of her alluring eye, that there are no red, green, or blue dots. The dots are cyan, magenta, and yellow. The printing press that this was printed on cost a little more than my home printer - I dunno the exact figure, but I am putting it around $10 million. They don't use RGB inks. 

The choice of CMY (with the addition of black) over RGB is a fascinating story. I will get around to telling it sometime soon. For the time being, let me say this. If you mix red and green ink, you get a dark and dirty brown, almost black. If you mix red and blue ink, you get a very dark blue, almost black. If you mix green and blue, you get another really dark color. So, mixing RGB inks, you can make red, green, blue, and three lovely shades of black.

Let's change up the inks. If you mix cyan, magenta, and yellow, on the other hand, you can get a whole bunch more colors. Cyan and magenta ink mix to make a purplish blue. Cyan and yellow make green. Magenta and yellow make an orangeish red. If you mix all three inks, you get a dark brown, almost black.

So, my assessment of the Color Picker Pen is that no one ever built one. If they had, they would have realized that red, green, and blue don't make for a very wide collection of colors. They would have then called John the Math guy to help them figure it out and I would have recommended cyan, magenta, and yellow. Since I never received such a call, I have to assume that the Color Picker Pen was never built.

So, why are color cameras RGB? And why is a computer display and a TV RGB? Questions, questions! I will leave them for another blog!