Hi - I've had a similar issue.
I have a theory (not fully tested) that it's due to targeting tags that are manually added versus Flickr robot added.

Certainly after manually adding a tag to a number of images in a Flickr account, those images were then returned using a tag and account combination where they had not before (when targeting a machine-added tag).