It might be the villagers. Try it out in the wilderness, in the same fashion, see if it happens there as well. If not, then it's probably the villagers. However, this probably should be reasoned so that villagers wouldn't bother about feeding fires outside the village area.