"My" vagabonds are usually vagabond woodsmen, hunters, or adventurers, not villagers. In most cases they're milling around my homestead, as that's the most likely spot for my character to see them, and my homesteads are located just outside if a cultures territory. It ought to be possible for a culture associated adventurer in the wild to do it too, but I don't think I've encountered many of those. I've never seen traders cutting trees, though (which should not be interpreted to mean I doubt that has been observed).