Actually, that association bothers me. It's one of the things I've been thinking of investigating a little more in depth. My research ended up turning away from that direction, but I did a little reading on it at the beginning of the semester, and it really did grow out of an attitude spearheaded by Henry James at the tail end of the Victorian era. He wanted to write about "adult" issues in his novels and didn't want to have to write to a family audience. Prior to his group's movement, most books were considered family reading, not "children's" or "adult." Fairy tales, especially, were considered tales to be told around the fire to everyone, not just nursery tales. During the two huge movements of literary fairy tales, the salon movements, the tales were mostly written by women, and shared with an adult audience during salon parties, kind of like the storytelling contest that Mary Shelley created Frankenstein for.
So, James wanted to write about sex and adultery and a lot of things considered inappropriate for women and children. That's when the gendering and aging of literature really began. Women's literature began appearing, as did books specifically aimed at children. (In comparison, in the 1880s, 6 of the 10 top ten books at any one time were what are now considered children's books.)
Along with that, fantasy began to be relegated to the children's realm, because if it wasn't realistic, James didn't want anything to do with it. Fantasy wasn't serious literature in his colleague's eyes. So it was shoved off to the kiddies--and therefore not valid.
I need to do more research on this, because I really don't understand it the way I want to, but that's the general impression I got after discussions and some readings in my Victorian lit class. The growth of the fantasy genre among adults has helped to dispel that idea, but our culture still generally sees things as fantasy/magic=kids.