I'm actually dealing with these problems right now in two series I'm reading. The first is the Pendragon series by DJ McHale. This series has interdimentional travelers going to lots of different worlds (pretty much introducing one new world in each book). With the made up names of the worlds and people and things in the worlds, I'm having a hard time keeping them straight. In the latest book, a MAJOR plot point depends on me remembering details of a world from a book he published a year or two ago, and at first, I thought he was talking about a completely different world. There wasn't a summary of the various worlds in the book itself, and I didn't want to wait the two weeks it would take to order the book in to my library (I've bought the first five, but not the later ones -- there's just not room on my shelves for everything I'd like to own), so I went to Wikiedia, and refreshed my memory, but got hit with some pretty serious spoilers.
With Mistborn 2, I appreciated the refresher course on which allomantic metals did what, but still found myself wishing for a chart of what the allomantic and ferruchemical powers were. I found these (with some minor spoilers), where I expected them, in the BACK of the book (though I was surprised that the summary of book 1 was in the back of the book without any mention of it in the front -- say in a table of contents of some sort). I listen to a lot of books on tape, and Peter and I are reading this one aloud to each other, so it's almost the same experience. In audio format, you couldn't get to the tables and summaries in the back until you'd already finished the rest of the book, at which point they're moot, and many times they're left out entirely.
While reading Mistborn 2, I also found that my memory was very fuzzy about what had happened in the climax of #1. Brandon's climaxes are so action packed, and have so many twists and revelations, that I tend to rush through them and come out a little dazed at the other end. When he talks in book 2 about what Vin remembers about killing the Lord Ruler, I believed him, but I didn't remember several of those details myself. I eventually ended up re-reading the end of book 1 so that I could have some hope of piecing mysteries together at at least the same rate as the characters in the book (if not one step ahead).
So essentially, my opinion on this matter is that you should have reminders of the important details of the previous book in the current book. You should, as a writer of a fantasy series, be able to expect that your reader has read the previous book, but not that they'll remember everything important. I don't like infodumpy chapters with talking heads -- I tend to skim or skip those -- but characters should be reintroduced with a quick (one sentence or less) reminder of what their role/powers are, and "hard" magic systems should have their rules reviewed so that readers can keep pace with the characters.