I don't think you can. It's like comic books. Surely people would enjoy them, but I don't know I believe they ever will, because they're stubborn. My parents once role played with my brother and I, made they're own characters and everything. Once. Not ever again. They just didn't want to.
As for putting them in mainstream bookstores... they're already in Barnes & Noble and Borders, where else do you want them?
I do think a prime time ad campaign might help; featuring of course good looking kids, who might look like they play sports or cheerlead or whatever instead of looking like chess club members.
Also diversify the subject. Name an RPG (still in print) that doesn't focus on non-fantasy/science fiction/horror subjects. With a few WWII exceptions, I can't think of any.
BESM is probably best positioned for that spot, since the core book already references sport fantasy settings, but really, how many jocks are going to want to play an Anime setting. You'd have to repackage it with photos of Steve Young, Ty Detmer, and Tiger Woods instead of girls with cat ears.
Basically, you're looking at a decades long project, one that I think has already started. Note the lack of nudity in the D&D 3E as opposed to 1E. Look at the general production quality of RPGs over the last few decades. Nerds will deal with stuff that looks cheezy if it's still interesting (I point at Fell's feelings about Palladium). But the mainstream wants a nice package, so RPGs are starting to deliver that. However, before we have 100 milllion role players, and before I can go to work and just ask some employees if they want to play like the other guys do with B-Ball, we're looking at a LOT of change.