Followup to the spectrophotometric romance

This is a followup to my blog "A spectrophotometric romance". Be forewarned, this post is rated PG, for Print Geek. And it will be only just a little bit silly.

This post is necessary in order to consolidate comments, questions, and answers that have appeared in many different places. To date (Monday, Feb 18) the post has been viewed 753 times. Since there are only 587 certifiable print geeks in the world, this blog post is the Justin Beiber YouTube video of the print geek crowd.

There have been 76 comments in four LinkedIn discussion groups and on the blog. I have had email and phone conversations about the post with nine people, none of whom are in my immediate family. The interested folks include manufacturers of spectrophotometers and reference materials, printers, physicists, color scientists, people in standards groups, and designers.

Netflix called me the other day to complain that my email traffic was starting to bog down streaming video speed in Los Angeles, the greater New York metropolitan area, and one neighborhood in Scranton, PA. So please forgive me if I left out one of your astute comments. 

Can I read a full copy of the TAGA paper?

Sure. Here is a link to the full paper: Evaluation of Reference Materials for Calibration ofSpectrophotometers. The paper was presented at the 2013 TAGA conference in Portland, OR.

Now for the obligatory plug. TAGA is great. I try not to miss the TAGA conference. It is one of the only conferences where you can learn about new technologies in all parts of the printing industry. You will hear from academics and professionals alike, without the sales hype of most conferences. The next TAGA conference will be held in Fort Worth in March 2014. I hope to see you there. If you do, then buy me a beer.

Do you want a second opinion about TAGA?  Here is what Sandy Hubbard had to say about this year’s TAGA conference: TAGA, You’re It!

What is going on in the industry to try to fix the problem of inter-instrument agreement?

Idealliance (in the US) and Fogra (in Europe) have assembled groups of experts from all the graphic arts spectrophotometer manufacturers to work on the issue. I was cautiously optimistic at the start, and am cautiously a bit more optimistic after attending the first meeting. I attended the US meeting. I was impressed that everyone in the room was sincere in their desire to find agreement on how to find agreement. I didn’t hear any finger-pointing or blustering about why one company provided the one true measurement system and everyone else is wrong.

Of course, we spent the first six hours complaining about how spectrophotometer owners use their delicate instruments to hammer in nails and drop ketchup and fired onions into the optics. But we got past that soon enough, and started complaining about cops who give out speeding tickets and about politicians.

The ISO standards group that covers color measurement for the graphics arts (TC130) has also made some recent changes that will eventually improve inter-instrument agreement. See the heading “What is being courageously changed?” below.

Why do the instruments disagree?

Speaking of finger-pointing, why can’t the various spectrophotometer designers just build their instruments the correct way? Then they all would agree, right?

That all makes sense, but the difficulty is that there are many different ways to build a spectrophotometer “correctly”. The different correct ways may not give the same results. You may think that I am just copping out, since I represent an instrument manufacturer, but listen up. I have a parable for you.

Tevye (from Fiddler on the Roof) is in the marketplace talking politics. One person speaks of the great new world that the tsar is bringing. Tevye looks at this man and says, “You’re right”. The second man disagrees, and says that the tsar is bringing ruin to the country. Tevye looks at this man and says, “You’re right”. The third man is beside himself. “Tevye, how can they both be right!?” Tevye ponders a moment, and then says, “You’re also right!” If two instruments disagree, it is totally possible for both of them to be right.

Here is the short list of things that can go cause disagreement.

Fluorescence – Recently, optical brighteners have become ubiquitous. They are added to paper to make the paper appear brighter. The trouble is, the amount of whitening that happens depends on the amount of UV content in the light in the spectro. If two spectros emit different amounts of UV, they will excite the optical brighteners by different amounts, and the spectros will disagree.

I think we should point the finger at the darn paper companies who put the optical brighteners in the paper. Or maybe at the designers who like the paper that is whiter.

White level – Believe it or not, the various standards labs can’t quite agree on what “white” is. If you send a white tile to be measured by several of them, you may see a 1 ΔE of difference in L*. This is a clear case of “you’re right”. I think we should point the finger at the standards labs.

Lateral diffusion – A spectro illuminates a small area of the sample, and collects the light reflected by another small area. One would think that these should be the same size, this would cause a problem. If the sample has some translucence (it’s cloudy), then light will scatter outside the aperture. The smaller the aperture, the bigger the relative effect. Now get this… a few of the BCRA II tiles have bunches of lateral diffusion.

I think we should point the finger at the print buyers who force us to measure colorbar patches that are 1.5 mm tall. Or the paper companies. You would think they could make each sheet on a paper roll just a millimeter or two longer to make room for decent sized color patches.

By the way, someone wrote a paper about lateral diffusion a long time ago. The guy’s probably in a nursing home in Milwaukee singing Those Were the Days.

Linearity – One would expect that a decent engineer would be able to design a spectro that is linear. Of course, there is a little problem that detectors are nonlinear when they come close to saturation. By the way, that’s where the detector has its best noise immunity.

And there is the little problem that transistors and all that stuff like to be nonlinear down where the signal levels are smallest. Oh, and the issue that it is hard to devise a set of samples to calibrate nonlinearity with. One is left with assembling a series of gray samples with different reflectance values, and having a national standards lab measure them.

Did I mention that the national standards labs are ghastly expensive when you ask them to measure something?  I think I’ll open my own national standards lab. Or just blame the stupid laws of physics.

Black level – Theoretically two spectrophotometers could disagree on the reflectance of a perfectly black object. I think they probably do disagree just a tiny bit, but I don’t have enough data to verify this.

Wavelength shift – Just like two spectros might not agree on what 50% reflectance is, they might disagree on where 500 nm is. Thankfully, we have physical standards, such as mercury-argon lamps, that we can use that will emit at certain precise wavelengths. But… it would be nice if there were a few more emissions lines. Stupid laws of physics, anyway.

Spectral bandpass – Don’t even get me started on this one!

Goniophotometry – This one is a favorite of mine, simply because of all the cool words. Besides “goniophotometry”, there is “indicatrix”, and the acronym “BDRF”.

The basic idea is that the amount of light reflected from a surface depends on the angle that the light hits and upon the angle that it is viewed. In practical terms, a 0/45 spectrophotometer that collects all the light emitted between 40 and 50 degrees might not agree with one that collects all the light emitted between 43 and 47 degrees.

When you see me at TAGA 2014, buy me a beer and we can talk about the birds and the goniophotometers.

By the way, someone once wrote a TAGA paper about the goniophotometry of printing ink. The writing in the paper shows many of the early warning signs for dementia.

So many possible issues! In the words of Rabbi Seymour Goldstein, “Oy vey, mein kapella! This is so meshuga, it gives me such tsuris!”

Also in the words of Rabbi Seymour,

“Grant me the courage to change those things that I can change,

The serenity to accept those that cannot be changed,

The wisdom to know the difference,

And a little duct tape and shoe polish to fill in the gaps.”

What is being courageously changed?

There are a number of things that have changed or are changing to try to improve inter-instrument agreement.

One concrete decision that came out of the Idealliance meeting (in the US) was that we all agreed to trace our “white” back to the same national standards lab. We decided to go with the US-based standards lab NIST. I don’t know if this was discussed at the European meeting, but hopefully we can all decide on a single national lab.

The ISO standards group that works in the graphic arts color measurement arena (TC 130) has recently made some important changes that have already, or will, improve inter-instrument agreement.

Most notable is the “M0 / M1 change”, which addresses the optical brightener issue. Spectrophotometers with the M1 designation will agree in UV content so that optically brightened papers will be measured the same. For more information, have a look at some other wonderfully written articles: “The problem with optical brighteners”, and “New lighting conditions”.

Another change (one that is interesting only to the print uber-geeks) was made to ISO 13655, having to do with wavelength bandwidth. Interested in more information? Buy me a couple of beers and we’ll talk. Buy me a few more beers after that, and we can go sing Don’t Stop Believin’ at a karaoke bar.

These changes to the standards are essential, but they take a while to filter down. The really odd thing is that a change to the standards rarely changes existing equipment, you know? For some reason, I need to send my old spectro in to get it updated, or maybe buy a new one, for gosh sakes.

What serenity is required?

Now it is the print buyer’s turn to get the pointy finger.

There is a rule of thumb in the statistical processing world. The accuracy of an instrument is only allowed to take up 30% of a tolerance window. Thus, if the tolerance window for a certain printed color is 2.0 ΔE, then my instrument must be accurate to within 0.6 ΔE. If you go beyond that, it just makes things worse.

The reality is unfortunately that two instruments of different make and model will likely not agree to within 0.6 ΔE. Setting tolerances closer than that will inevitably just make people want to drink more beer.

Anyone have a spare cup of wisdom?

Wisdom… I wish I was smart enough to know what to say about this.

Hopefully my TAGA paper has contributed to the collective wisdom. My message was to know when a courageous attempt to make two instruments agree should have been tempered with a bit of serenity.

Duct tape and shoe polish

Another discussion item in the US inter-instrument agreement consortium was the development of a test strip with a selection of colored patches that could be measured by different instruments. After measurement, something could be done with the readings.

We discussed several options about what could be done:

1) The measurements taken with a single instrument could be compared over time to verify that the instrument has not changed,

2) One instrument could be compared with another to see if they measure substantially the same,

3) An instrument could be compared with measurements from a national standards lab (or perhaps a secondary lab) to assess the accuracy of the spectrophotometer, or

4) The set of samples could be used to standardize one instrument to another -- that is to provide a way of converting one instrument so as to match another.

The four potential goals are successively more complicated and expensive. In particular, obtaining measurements from a national standards lab would cost thousands of dollars, so it is not cost effective for most users. The fourth option may require a color scientist/math guy, so might only be an option for the spectrophotometer companies that can afford to hire color scientist/math guys.

Colligation

I am not done playing with duct tape. I am in the process of cogitating on colligation. The ancient Greek, Ptolemy developed a set of equations that could be used to predict the positions of the planets at any time. The equations were based on a lot of wrong assumptions like the Earth is in the center, and all the rest of the celestial bodies move in circles that revolved around circles. The model worked, at least to an extent.

A millennium or so later, Copernicus decided that the Sun belonged at the center. (I hope everyone celebrated his birthday yesterday?) Then Kepler came along and decided that ellipses made the whole thing simpler, and then Newton provided the big colligation. The inverse square law of gravity was the grand unifying theory that explained the whole enchilada.

My goal is to colligate. I have gone through the Ptolemaic thing in my TAGA paper, and shown that things go subtly awry when you have the wrong starting assumptions. I have done the Keplerian thing and did regression with (what might be) the right assumptions. It’s time to roll up the sleeves and do the Newtonian thing.

I plan to collect a bunch of spectros, enough to represent the range of physical differences that we see out in the field. I plan to collect a bunch of sample sets. I have the ones that I used for the TAGA paper, but I will gather more. Some new samples are already on a plane, headed to my office.

Then I plan to systematically go through specifically designed sample sets to try to ascertain what differences exist between the real world instruments. For example, I will measure a set of white and gray samples with varying levels of gloss to see if there is a goniophotometric difference – a difference in gloss rejection. I will measure a series of white to gray samples to ascertain whether the typical instruments in the field differ in their view of linearity.

And so on.

I am not sure where the colligation will lead. It could be that it will show that courage will be required, and there will be another round of “gosh darn it, we need to tighten up specs on something or other”. Or it could be that more duct tape is in order, and a simple standardization, carefully applied, can make an appreciable improvement.

I am still cogitating on this inter-instrument colligation. I welcome any suggestions or offers of help. At least, up to the point where I spend 23 hours and 56 minutes a day responding to emails.