Hey, it sounds like you are having problems with the VanDerLee filter due to limited memory. I just fired up my copy of the filter on a 2 layer, 5000x5000 pixel grayscale, 8bpc image of a couple of blobs of light (see below - note the ruler shows that the size of this starting image is indeed 5kpx square) ...
[ATTACH]53690[/ATTACH]
VanDerLee's Halftone filter (at around 20 lpi) turned it into dots in about 10 or 11 seconds with no problems, not even a hickup, LOL. At 200 lpi, it took about a minute - no problems, just slower. Below is a crop of a small area on the edge of one of the blobs of light to show the smooth transition from light to dark.
[ATTACH]53691[/ATTACH]
FYI, I did this test on a PC running Win 8.1. It has 64 Gig of fast RAM, a moderately overclocked i7 processor, and a bunch of fast SSDs. As we talked about earlier, because this filter seems to be limited to running under 32 bit Photoshop, that's what I had to use.
With respect to the 200 lpi limitation, I wouldn't worry too much about it. If you have enough memory, you can do tricks like work on a 2x scaled-up version of your image, process it in "Halftone" at 200 lpi, and then down-rez by 2x to get an effective 400 lpi linescreen.
HTH,
Tom
PS - Glad you figured out that lightening the midtones using "variations" helped with your dot overlap problem. Basically, this is doing almost exactly the same thing that I suggested earlier, except that I like adjusting the tonality of the starting image using either a "levels" or "curves" adjustment layer. These give much finer control than "variations" does.
