I don't know, this is a theory with virtually no backup behind it. Brandon likes to foreshadow things, so that when you actually learn the reason for something, you can go back and say "Woah, all those things were leading up to this, it was so obvious!"... except it wasn't, because you were missing the central piece to the puzzle.
We don't know anything about the Parshmen and the Parshendi. There has been no interaction between Alethkar and the Parshendi before Gavilar found them, and when he did, they were just as developed as they are in the book. I think that would rule out your theory, unless you go into some secret kidnapping action, or hidden interaction with other kingdoms... but that would be you trying to save face, I think.
If there is any hint about what and who the Parsh-people are, it would be in Jasnah's chapters and the epigraphs of Part Three - her research and theory that the Parsh are the Voidbringers. I have my own thoughts on her supposition, but I'll have to post them in a different topic, as they are kind of an off-topic here.