It's literally silver, and yes, it can be a pain to get hold of it. Traders do carry it occasionally, and it can occasionally be found in villages. I think I tried 4 or so traders and (half?) a dozen villages before finding any silver (in a village). Generally I'd say traders is the most reliable source (and you've experienced how reliable that is..). I've since found silver providing traders, by the way, and have since built up a respectable mound of trinkets.
My character is Remi, so I found my silver in a Remi village. Driik are the best for anything not tied specifically to a given culture, so I'd expect that to go for trinkets as well. Trinkets in villages is a new thing (traders have carried them for some time), so I can't say how common it is in other cultures, but I'd expect the northern tribes to be the worst for trinkets.
And, by the way, pick up some seeds while down in the south if you intend to do some farming, as northerners don't farm, and thus don't carry seeds (assuming your character is a Seal tribe one).