I agree with Sam, and will hopefully reinforce the point I think he is making.
As you are "organically" (ie, manually) constructing any mosaic, you will come upon areas in the "target image" with specific RGB values. You then need to be able to go to your library of "elemental images" and find a suitable one, ie, one whose average R, G and B values are close to the R, G, and B values of the target image at that point.
So, let's roughly estimate the number of different individual elemental images (ie, the small ones) one would need to have in your library in order to adequately represent a reasonable range of different reds, greens, and blues in the "target" image that the mosaic is trying to reproduce.
As you know, 8 bit-per-channel JPGs do a very reasonable job of representing color images. An 8 bpc JPG can use 256 different values of R, 256 different values of G and 256 different values of B. To match this degree of color and tonal fidelity, your library of elemental images would need to contain at least 256 x 256 x 256 different pictures.
Obviously, this is a lot more than one would need to construct a recognizable overall image. My guess is that the absolute minimum bit depth one could get away with in such a mosaic and not have it look horrible is probably around 4 bits per channel, ie, 16 different values in each of the three channels. This implies that your library of elemental images needs to contain 16 x 16 x 16 (ie, 4096 ) appropriate elemental images. This is quite a large number, but, if one had enough time, energy, and good organizational skills, this wouldn't be out of the question.
The real problem arises because you said that you don't want to artificially change the color or brightness of any of the elemental images -- you want to use all "real" images. This means that you would need to find lots of images for the library whose average R, G and B values are extremely unlikely, eg, a "real" image whose average red value is between 0 and 15 (on a 256 scale), and has a nearly 0 average for the green and blue channels. Then, you would need to find another "real" image whose average red value is between 16 and 31, also with nearly 0 values in G and B, etc. etc. One would be very hard pressed to find such odd images.
Finding such unusual elemental images is precisely why most of the mosaic generation programs instead modify the brightness and colors of one or more base images in order to generate the needed range of colors and brightness in the base images.
I hope the above discussion helps you understand what is feasible when attempting something like this and why successful mosaic software works the way it does.
Cheers,
Tom