Without having the action to try its difficult to say for sure but my guess would be that the action was written on a later version of PS than CS5.
Regardless of what type of document is open, its probably fine, but if the action, for example, is trying to apply one of the newer CC blur filters that don't exist in CS5 then its going to throw an error.
'Not the right kind of document' could mean that the file needs to exist, ie, NOT applied to a 'New' document.....which until you save it doesn't exist except within PS.
Actions rarely work 100% on all versions unless especially simple.
Check the action was written for CS5.....and remember CS5 came in 4 flavours....CS5, CS5 Extended, CS5.1 and CS5.1 extended....make sure it really IS an action written for CS5, NOT CS5.1 and NOT an extended version.
Regards.
MrToM.