Tolerance.

[MrToM]

We've all used it, and we all know what it does but when setting the Tolerance for certain tools what exactly is that a Tolerance of?

Being a single integer number from 0-255 is it the average of something?
This I doubt as the 'average' could be obtained in more than one way.

Anything I think of always draws a blank....its the 'single' number thing that gets me.....what in the image is just a single number?

I've also drawn a blank on numerous '< pick a search engine >' searches.....most of it is random guesswork at best so does anybody know what the Tolerance number relates to?

Regards.
MrTom.

 

[ALB68]

From Adobe relative to the Magic Wand tool -"Determines the color range of selected pixels. Enter a value in pixels, ranging from 0 to 255. A low value selects the few colors very similar to the pixel you click. A higher value selects a broader range of colors.

 

[dv8_fx]

What do you think of us? We're DESIGNERS not mathematicians......


I would think of tolerance as range within a spectrum of values. If you were to create a gradient of color from 0 to 255 and hover your pointer tool over a spot, the values shown in the Info Tab represent the values selected within that range.

For the wand tool, if you set the tolerance at 50, it will select (or should select) at most 25 shades lighter or darker within the range(25+25=50). This is the tolerance limit of the tool selection. It won't select any higher but will include everything below the range.

Some say this is inaccurate..... I dunno.

 

[MrToM]

Thanks guys but it still doesn't explain what the Tolerance 'number' relates to.

Let me put it like this...

You select the Magic Wand Tool...(As Larry has already mentioned it)
You Set the Tolerance to say 50...(As dv8 has already mentioned that as a value)
You click on the image...
And you get a selection...

Now...when you click, PS must get some 'value' or 'property' from the pixel you click on.
That 'value' is then compared with the Tolerance value.
The selection is then made based on the same 'property' value within that range...

...but what is that property that PS gets and compares with the Tolerance you set?

Like I said, it can't be an average of RGB as the same average could be obtained in many different ways.....eg, [255 0 0] would be the same average as [0 255 0] and [0 0 255], but those are most certainly not all the same colour.

Even Luminance values could be the same for different colours.

It must some unique property that is denoted by just one single integer number....but what?

I appreciate your thoughts though, as always.

Regards.
MrTom.

 

[Hoogle]

I always saw it as based on the 256 color value but I may be wrong I guess I never really thought any more of it. Comes from back in the day if your graphics card could cope with 256 true colours on your monitor set up. But like I said that was an assumption and never thought more of it til now

 

[ALB68]

Thanks guys but it still doesn't explain what the Tolerance 'number' relates to.

Let me put it like this...

You select the Magic Wand Tool...(As Larry has already mentioned it)
You Set the Tolerance to say 50...(As dv8 has already mentioned that as a value)
You click on the image...
And you get a selection...

Now...when you click, PS must get some 'value' or 'property' from the pixel you click on.
That 'value' is then compared with the Tolerance value.
The selection is then made based on the same 'property' value within that range...

...but what is that property that PS gets and compares with the Tolerance you set?

Like I said, it can't be an average of RGB as the same average could be obtained in many different ways.....eg, [255 0 0] would be the same average as [0 255 0] and [0 0 255], but those are most certainly not all the same colour.

Even Luminance values could be the same for different colours.

It must some unique property that is denoted by just one single integer number....but what?

I appreciate your thoughts though, as always.

Regards.
MrTom.

Sorry MrTom I'm afraid that's above my pay grade. Call Adobe and ask for a code writer LOL

 

[MrToM]

I always saw it as based on the 256 color value ... never thought more of it til now

Thanks Hoogle, thats exactly what I did...assumed it was the colour value.....it could still be I guess.....but how?

Sorry MrTom I'm afraid that's above my pay grade...

LOL....thanks Larry...its above mine too which is why I threw it open to the masses.

Does the question make sense though ?
(And I don't mean to anyone personally I mean in general...or is it just a stoooooopid question?)

Curiosity can be a real pain sometimes.

Thanks again guys, and Larry, I think I'll give the Adobe coders a miss.....they don't even know the difference between dpi and ppi! OH NO!!!! Not that old chestnut again!

Regards.
MrTom.

 

[dv8_fx]

Thanks guys but it still doesn't explain what the Tolerance 'number' relates to.

Let me put it like this...

You select the Magic Wand Tool...(As Larry has already mentioned it)
You Set the Tolerance to say 50...(As dv8 has already mentioned that as a value)
You click on the image...
And you get a selection...

Now...when you click, PS must get some 'value' or 'property' from the pixel you click on.
That 'value' is then compared with the Tolerance value.
The selection is then made based on the same 'property' value within that range...

...but what is that property that PS gets and compares with the Tolerance you set?

Like I said, it can't be an average of RGB as the same average could be obtained in many different ways.....eg, [255 0 0] would be the same average as [0 255 0] and [0 0 255], but those are most certainly not all the same colour.

Even Luminance values could be the same for different colours.

It must some unique property that is denoted by just one single integer number....but what?

I appreciate your thoughts though, as always.

Regards.
MrTom.

Here's something tho the discussion never carried through to an understanding of the issue. But maybe you can make heads or tails use of what the OP in the second post outlined...... http://forum.teamphotoshop.com/26/23334/what-does-tolerance-mean

 

[MrToM]

Interesting...thank-you dv8, I hadn't seen that one before.

I'm still not convinced its that simple though.....what if only 2 of the channels fell within the tolerance?
Would it select the pixel or not?

Regards.
MrTom.

 

[Tom Mann]

I'm between students at the moment, so this has to be very short.


You are very close to the correct answer. The way it is typically done in image processing is to take the differences between the values in the red channel the green channel and the blue channel, and let the difference metric be the largest of the three differences. One can also do an RMS metric which equals the square root of the sum of the squares of the individual channel differences.


More later,


Tom.

 

[MrToM]

@Mr Mann...

I don't want to sound pushy but just wondered if there really was any more you were going to add to this?

No worries if not, I just got the impression there was more to come.

Many thanks.

Regards.
MrTom.

 

[ibclare]

Oy Vey. Yes, do come back and expound upon that a wee bit more. For those of us with mathematical chaallenges, this is quite the mind boggler.But I am sure you can put it into lay terms that the muddled masses can comprehend. Yes?

I'm between students at the moment, so this has to be very short.


You are very close to the correct answer. The way it is typically done in image processing is to take the differences between the values in the red channel the green channel and the blue channel, and let the difference metric be the largest of the three differences. One can also do an RMS metric which equals the square root of the sum of the squares of the individual channel differences.


More later,


Tom.

 

[MikeMc]

He said ...I will make it simple...."RMS metric which equals the square root of the sum of the squares of the individual channel differences."

Well Tom, I do not work on rockets, so for me....PLEASE PUT THAT IN ENGLISH ! ............. thanks for the reply. :banghead:

 

[ALB68]

He said ...I will make it simple...."RMS metric which equals the square root of the sum of the squares of the individual channel differences."

Well Tom, I do not work on rockets, so for me....PLEASE PUT THAT IN ENGLISH ! ............. thanks for the reply. :banghead:

You don't understand that? I'm gonna pull your guru's license!

 

[MikeMc]

You don't understand that? I'm gonna pull your guru's license!

I never did metrics......

 

[MrToM]

I've also tried the application of several Barge Landy's but it just theems to sake mings bice as twad.

Regrads.
mrmtmmmm.

 

[Tom Mann]

Hey, folks - Sorry ... I completely forgot about this thread. I'll fill in the details later today.

Best regards,

Tom M

 

[MrToM]

Thanks Mr Mann, much appreciated.

Regards.
MrTom.

 

[Tom Mann]

Here's a way to visualize exactly how the tolerance adjustment affects the magic eraser, one of the tools that has a tolerance adjustment. The same sort of approach can be taken for other colors, other tolerance settings, as well as other tools that have a tolerance adjustment.

To start, set up a PS canvas with a black background layer, plus three gradients all on a single, second layer. The first gradient is in hue, the second is a gradient in saturation, and the third is a gradient in tone (brightness). Then, place the magic eraser tool over one of these gradients, and click. This will remove the parts of each gradient that are within the tolerance setting of that tool, letting the black background show through. Here are some examples:

1. Removing fully saturated green (by clicking near the center of the top, fully saturated hue bar), tolerance = 128



2. Removing fully saturated green, tolerance = 178



3. Removing fully saturated green, tolerance = 255



4. Removing medium saturated reds (by clicking near the center of the lowest bar, the gradient in saturation), tolerance = 16



5. Removing medium gray tones (by clicking near the center of the middle bar, the gradient in brightness), tolerance = 16



As you can see, increasing the tolerance (ie, #1, 2, and 3) when removing fully saturated greens widens the black gap in the green bar, as expected, but also, when you set it to 255, also starts introducing gaps in the other bars. Similarly, removing medium saturated reds with a tight tolerance (ie, 16) makes a gap in the lowest bar, but doesn't affect the other bars (ie, which have full saturation, or zero saturation). An analogous effect is seen when removing some unsaturated middle tones -- it doesn't affect either the fully or partially saturated colors.

My suggestion is that you set this up for yourself and experiment with it to see exactly what the effects are for tolerance settings and colors that you are interested in. Another very interesting variant of this experiment is to also include gradients in R, G, and B on the same layer, and then see which sections of those curves are removed when you click on say, a mid-tone, mid-saturated, non-primary color. I think you should see something very interesting!

HTH,

Tom M

 

[Tom Mann]

The next step in the progression to understand how the tolerance feature works in these tools is to see what a tool that has this feature does to 2D color gradients. Once again, I "punched holes" using the magic eraser tool, but this time, in a two dimensional gradient.

The two images attached below are 2D color gradients formed by taking a red-to-black horizontal gradient, and then stacking over that a green-to-black gradient in the vertical direction, but with the layer blend mode set to "linear dodge". This is the same as adding the color intensities, point by point throughout the image. The first one has 4 holes punched in it using the magic eraser tool with the tolerance set to 14. The second is just one hole, but with the tolerance set to 90.

The interesting thing about all these holes is that the sides are almost perfect straight lines -- two sides parallel to the x-axis, two sides parallel to the y-axis, and the remaining two sides would be at exactly a 45 degree angle if I had scaled made the spectrum exactly square. This immediately tells us something about how the tolerance algorithm works. For example, if the "hole" had been a circle, it would have told us that the algorithm used the square root of the sum of the squares formula that I mentioned in my recent post. OTOH, if the hole only was four sided, it would have told us that the algorithm used the absolute value of the maximum of the differences in R, and in G, one of the other possibilities I mentioned in my earlier post.

However, one does not need to know the math that lies behind the visual effect to get an idea of how the tolerance slider works: you can check it for yourself by doing familiarization experiments like these. For example, as the next experiment, you could do a 2D gradient in RGB with hue on the horizontal axis and saturation on the vertical axis, and punch some test holes in this. If you do, you'll see that for small values of the tolerance, the holes are almost perfectly circular. This immediately tells us that the appropriate formula for this situation WAS Euclid's classic formula for a circle: The square root of the sum of the squares of the two coordinates is a constant.

Since I gave away the answer to the case of an HSB gradient in RGB, if you are interested in this pursuing this for yourself, make up a 2D spectrum in Lab space. Use a constant, medium value for L, and let the "a" be on the horizontal axis, and the "b" be on the vertical axis. Punch various sized holes in that and show us what you get. I think you will be surprised, as it looks quite familiar, LOL. Once again, what you see tells us a lot about the actual mathematical formula Adobe used to quantitatively select colors that fall within a specified tolerance of a starting value in that space.

HTH,

Tom M

PS - Also, don't forget to try some with brightness / luminosity / value on one of the axes. ;-)