As always, the answer lies in 80s Teen movies. I refer to a) Pretty in Pink, and b) Some Kind of Wonderful. Romantic comedy writers, do yourself a favor and watch these films if you haven't already.
a) Pretty in Pink, Molly Ringwald's (Andie) best friend is Duckie (John Cryer, woo woo!) who is completely in love with her. She knows this, he knows this, he asks her out once, totally oblivious to the fact that she's dealing with another boy at the time, though it should be obvious. Duckie never does win her to his side, but he sticks as a friend and helps her out with her love interest both when it is against his romantic interests, and with someone who belongs to a group he specifically dislikes. I find it a realistic treatment, despite the fact that it's John Hughes. Once it's known that the guy digs his friend who is a girl, it either becomes a romance then, or it never happens.
This jives oddly with my own personal experience, wherein I was very obviously in love with a girl who was my friend, and she knew. It never really blossomed into a romance, but I did get some good make out sessions.
That's the odd thing about friendships to romance. For the party who has a friend who digs on them, it can be tempting for the physical release (Let's make out!) without trying for commitment, which a) strains the friendship, and b) makes the friend who is digging think he's got a shot when he really hasn't.
b) John Hughes gives us the other approach. Mary Stuart Masterson (hubba hubba) plays Watts, who is in love with Keith (Eric Stoltz) who is in love with Amanda (Lea Thompson WOO WOO!). Keith actually wins Amanda over after nearly being beaten by her former boyfriend's goons (saved only by the PUnk Duncan -- played by Elias Koteas, later to play the ever lovin' Casey Jones WOO WOO!). HOWEVER! he's been ignoring some very obvious signs from Watts, his tomboy cum drummer best bud (who helps him practice kissing, how dumb *is* this Keith guy anyway?). Amanda, being a girl and more sensitive to these things, sees how Keith really feels about Watts just as he's becoming conscious of it (but can you ditch a date to go after the one you love? or does that show a complete lack of class). Fortunately, Amanda solves the dilema by sending Keith after Watts, having learned from him already what she needed to know.
So, the friendship can become a romance. The guy more or less has to be hit over the head with a sack of bricks to realize he's been pursuing the wrong one all this time. That's the thing about guys, they tend to go after the wrong one. However, they don't go after jerks, like the stereotypical girl does, they go after the impossible dream (it is my steadfast testimony that deep down inside, every guy is really Don Quijote, and will quite happily scale imaginary castle walls and tilt at windmills for the object of their desires,... so long as they don't have to actually talk to her).
so, my thought? I could write this like cake if it were a guy who was misled. With the girl, it's a bit more odd to me. My perception, along with a lot of guys I know, is that girls love to have guys love them. That's why it appears that they only go after jerks. They already know the nice guys like them, the jerks need more maintenance to ensure their loyalty.
I guess how I'd do this (and yeah, I'm not really answering your specific questions, but I'm throwing out ideas), is that your main char. girl has to realize that nice guy best friend's loyalty is not guaranteed, as he and flighty chick start forming something she realizes that nice guy best friend is the one she really wants to dig on. We must sabatoge this relationship! (kind of a sick My Best Friend's Wedding, I guess). To make her ethical, skip the sabatoge action (just plan it, but be unable to go through with it) and be there for him when flighty chick turns out to really be flighty. Maybe you think that this confirms the
Chimp-Gorilla Hypothesis (warning, pages on this subject probably contain some language, knowing the guys who did that, but it's been a while since I read that particular article) when you don't really believe in it. I, however, think it bucks it as a constant, while retaining the realism that some girls *are* like that, just like some guys are. In the mean time, of course, nice guy best friend needs to not realize he's interested in your main char until he's hit with a sack of bricks.
That would satisfy me.