In a new game, if you happen to do that again, pretty much stay away from their village for a few days then periodically visit to see what they say. They'll either say get away, or they'll ask "Where have you been?"
Oddly, there's no inbetween commentary that I've seen of "You can come in, but we're watching you"