This is certainly something I've thought about in my own writing.
For me, the line between "hard" or "speculative" magic and fantastic science (ie. science that is inconsistent with how our world works) is whether the phenomenon we're talking about can be defined in terms of our existing physical universe.
If it's only definable or measurable by other magic, it's fantasy. If you can quantify it or qualify it physically in some potential alternative universe, then it's fantastic science. It's obviously a given that there also needs to be an internally consistent set of rules for fantastic science, but on its own I would contend that an internally-consistent system of rules is not enough to make something scientific. Real-world religions have internally-consistent rules, but we don't regard them as rigorous scientific theories. They also need to be falsifiable- that is, measurable and definable on a physical level.
Interestingly, this certainly defines psychic powers as magical, even though they're a long-standing part of science fiction tradition. (Also, it makes the Force into something of a fantastic science, assuming you don't try to fan-retcon out the midichlorians)