when painting in the layer mask put your brush sofness right down and paint it in slowly each time increasing the opacity maybe a slight colour match between the layers as well
not sure about your layer setup, but try using a second set of mutant textures layers in folders. one set of mutant textures set normally, the second set to blending mode luminosity. extend the mask of the set using blending mode luminosity further down the finger. could give you a better transition between mutant and finger.
When painting the layer mask to brush sofness on your back and paint it every time in the gradually increasing transparency may have slight color, and between layers.