In my experience, this is a surprisingly difficult and annoying problem.
The best results that I have ever seen from a semi-automated method to do what you asked were obtained by using the
frequency separation method, but even these results were disappointing.
The frequency separation method is usually applied to skin / portrait retouching. It works by dividing the image up into two spatial frequency bands. The high spatial frequencies correspond to very small feature sizes, for example, individual pores, or, in your application, the individual threads and weave pattern of the cloth. The low spatial frequencies correspond to larger feature sizes, ie, features that occupy a significant fraction of the image dimensions.
In the case of head shots, the lower spatial frequencies contain the major features of the face, and the shadows cast by them.
In the case of camouflage patterns on cloth, the high spatial frequencies contain (a) the threads, stitching, and weave patterns of the cloth, whereas the low spatial frequencies contain (b1) the overall lighting distribution and shadows, and (b2) the camouflage pattern.
The problem is that to do what you asked, you need to remove (b2),
while not touching (b1) or (a), and you need to do this in a convincingly realistic manner. This is beyond the capabilities of this method. If you can't separate (b1) from (b2) because they are in the same spatial frequency range, and you simply remove or reduce both of them at the same time, then, as you discovered, you've just removed the lighting distribution and shadows along with the camo pattern, and this can look really odd, or, to use your word, "flat".
At the other extreme, if you don't remove all traces of the camo pattern, and, for example, ghosts of the edges of each of the differently colored areas might remain, making it obvious that the image has been manipulated, ie, fake.
My feeling is that if it is essential to your customer's business, their best option is probably to bite the bullet, and make up 2 or 3 versions of the design using a plain, non-camo, but textured fabric, say, one in a light gray, and one in a dark gray, and then use photoshop to turn these into any color they need. Depending on exactly what will be seen, perhaps a seamstress can just sew the new gray fabric right over an existing camo version instead of making up a new one from scratch.
Anyway, I wish you the best of luck. If you find a method that works, I would love to hear about it, so please let us know, and, if at all possible, please post a before-after example.
Cheers,
Tom M
