Actually, it wouldn't be that random of a plot twist. We know there is something about him that is special, RJ has confirmed as such.
Point 1) His appearance is constantly, constantly brought up in the story; noticeably more often than any other character except maybe Loial, which is probably why everyone thought he was Gaidal Cain reborn.
Point 2) He is absolutely obsessed with the Snakes and Foxes game, almost as if he wants Mat and Co. to be reminded that the Finns can't be beaten (either that, or he is a Moiraine sympathizer, and is trying to show Mat how to win, in his own weird way).
Point 3) No matter what has happened, Olver has managed to stay with Mat. He is never far from him for long. (this last point is a little weak, as several plot devices could explain this). If you have time, take a second to re-read the first Mat chapter in TGS again, then re-read the chapter when Mat meets the Eelfinn. I'm not saying I am right, but if I am, it's not a random plot twist, it's a diabolically clever one.