The first thing happening is that tracks start to disappear prematurely, since they are among the least important things. I believe I've seen that.
Later you're supposed to get a warning, but I've never seen that one. Eventually things start to disappear, starting, I think, with the "least important" ones.
I think the stack limit for item disappearance is 3000, but I'm not certain.
Based on how homesteads seem to interact with villages and villages with each other, I suspect one "area" in the world tile you're in and a ring of probably one, but possibly two tiles outside of that. This is guesswork, though.
Avoiding it is easy, as it's hard to get it in the first place:
- Place your farm plots a little bit away from your homestead (900 tiles is at least 900 items when crops grow).
- Stack things that are stackable (within reason: there is no reason to be obsessed with it, as a few dozen "extra" stacks aren't much). If killing a lot of Njerps, recycle their clothing into bandages (or other things you're probably using regularly). Clothing rarely stack because they're in varying state of wear, while bandages do. Weapons likewise don't stack, again due to wear. If using a metal crafting mod I'd recommend recycling weapons into iron, otherwise selling those (if villagers have anything to offer) is a good idea. If nothing else, weapons can be sold for fur that can be sold to traders for trinkets.
If you're an obsessive hoarder, consider setting up multiple stashes. You should, e.g. be able to dump stuff into one stash until you get a warning, at which time you'd start to fill up the next one.
- Excessive construction can presumably get you into trouble as each wall is one or two stacks of logs, floor/ceiling tiles contribute, fence tiles are made up of one or a few stacks, etc. I haven't had any problem with surrounding my homestead completely with bear trap arrays, though.