What causes villagers to refuse hiring now but maybe in the future?

[PALU]

It's rather frustrating to have everyone I ask refuse when I'm trying to round up a posse to take down a robber band. It doesn't help that I have no idea why they refuse (including the members of the village that wants the robbers gone), as I've done a fair number of quests for them, and have done nothing to anger them.
The closest village has refused every time I've tried to hire someone, while I'm fairly sure I've now been rejected by at least two people I've hired to take down previous robber bands (based on their equipment), and as far as I can tell we've parted on good terms.
3 villages of refusals so far...

[nekot]

Has any of your companions died? Or maybe didn't return on his own?

[PALU]

One companion died in the first robber elimination (largely through friendly fire, grr), but my experience is that they usually refuse with the message that they won't join until <X> returns. I did later get a flat refusal from one adventurer, and it might be in the village the dead character came from.

I did eventually get 10 people to take on 5 robbers, so it worked out in the end (and I then proceeded to remove another known robber band of 4 when I had a posse at hand, plus a bonus Njerp). However, I think at least 4 former companions didn't want to join "then, but possibly in the future". Did I pay them too much so they didn't want more food?