Author Topic: Multiple scores  (Read 15485 times)

The Holy Saint, Grand High Poobah, Master of Monkeys, Ehlers

  • Administrator
  • Level 96
  • *****
  • Posts: 19211
  • Fell Points: 17
  • monkeys? yes.
    • View Profile
    • herb's world
Re: Multiple scores
« Reply #45 on: December 19, 2005, 11:59:32 AM »
Quote
SE, while your explanation for using 5 clocks is better then Skar's it still doesn't do much to change the problem.  Why would 5 be easier to understand then 6?  And what makes you think saying 5 clock will be extremely rare will keep people from giving them out all the time like 6s are now?  And Fell will never agree to allow for an editor to change any review score without talking to the reviewer first due to the Killowat incident.  And lastly if we move to a different scoring system who is going to go change the 600+ reviews already in the database?  I sure as heck ain't.

Yes, it does. Because the 5 clock system would be SCALED, instead of being like D&D levels.
See, the way we have it right now, it becomes progressively more difficult to move to the next half-clock the more clocks you already have. The difference between 5.5 and 6 clocks is much less than the difference between 3.5 and 4 clocks. That's what people don't understand. 6 clocks is just an odd number that serves no real purpose.

We'll work out who changes what when we decide what to change to. If nothing else, I can change 2 or 3 a day. no biggie.

And how about we let FELL decide what Fell will agree to. Changing under my system is very different than what happened with Kid. You know that. If you don't realize that changing his score was just hte last straw in a long line of things he was uncomfortable with, you didn't see anything he had to say.



Quote
Again, I see changing our scoring system as a band-aid fix it will only fix things in the short term due to several reasons already stated by myself and others here.

1) We review a lot of what we like so things tend to get higher scores.

2) Scores are only a quick summary of what the review text states.

3) People have the conception, write or wrong, that if they like something it must be a 6 score.

4) Or editors have never been very strict in enforcing the review criteria, even when reviews that don't support the score are brought to their attention.

5) That no scoring system is perfect since everything is all opinion anyway and there's only so many ways you can convert a 800 word review into a few numbers or letters, none of which are as effective as the initial review.

1) I don't think this is something that needs a remedy nor do I think my system tries to do anything about it. Not a problem.
2) I don't see why this affects my argument. It's a feature of all rating systems. If anything, my system proposes a much better reflection of the summary than the existing one.
3) That's why we let editors change scores and make reviewers read an follow the guidelines. Again, I don't see why that affects a decision to change the scale.
4) We haven't been strict, but we have enforced it and made people make changes. Usually this happens via email BEFORE it runs. Thus you don't see it happening. Again, allowing editors to change scores remedies this nicely.
5) none are perfect, but some are more intuitively understood than others. The language is not hard to understand in our description, but the fact that you *need to read* the description is the flaw. The system should be something easily understood without any special knowledge. Our current system fails at that. Maybe we go with 6 clocks instead of 5, but it doesn't matter. THe current system isn't progressive and gives a false impression about the overall feeling about the product. THat is something we *can* remedy, even if the remedy doesn't make it "perfect." No one's looking for perfect here.

Quote
Most of these issues must be addressed individually, #2 and #5 are things that is out of our control as long as we want to assign a score, or no matter how many times we change around our numbers we'll eventually end up in the same place.

I think I've given a very reasonable argument above to why this is not at all out of our control.

Skar

  • Moderator
  • Level 54
  • *****
  • Posts: 3979
  • Fell Points: 7
    • View Profile
Re: Multiple scores
« Reply #46 on: December 19, 2005, 12:43:50 PM »
Quote
SE, while your explanation for using 5 clocks is better then Skar's...


Thank heavens SOMEONE speaks Spriggan around here. ;D

If e is suggesting that we go to a 5 out of 5 system with the ability to add a sixth clock when a "Citizen Caine" comes along, I'm in total agreement.  
"Skar is the kind of bird who, when you try to kill him with a stone, uses it, and the other bird, to take vengeance on you in a swirling melee of death."

-Fellfrosch

Spriggan

  • Administrator
  • Level 78
  • *****
  • Posts: 10582
  • Fell Points: 31
  • Yes, I am this awesome
    • View Profile
    • Legacies Lost
Re: Multiple scores
« Reply #47 on: December 19, 2005, 12:47:23 PM »
Quote

Yes, it does. Because the 5 clock system would be SCALED, instead of being like D&D levels.
See, the way we have it right now, it becomes progressively more difficult to move to the next half-clock the more clocks you already have. The difference between 5.5 and 6 clocks is much less than the difference between 3.5 and 4 clocks. That's what people don't understand. 6 clocks is just an odd number that serves no real purpose.

I still don't see how its harder to grasp a 6 point system over a 5 point system.  The differences between a 5.5-6 and 3.5-4 SHOULDN'T be significant.  They're not, you just are thinking that way,  each .5 is the same percentage.

Quote

We'll work out who changes what when we decide what to change to. If nothing else, I can change 2 or 3 a day. no biggie.

Not to be rude, but I doubt you or anyone else will manage to be consistent enough to actually get this done, let alone having to change the site code to reflect a max of 5.  Also, since we'd have to change our review criteria, we are obligated to contact all the original reviewers and ask them what they would change the score too.

Quote

And how about we let FELL decide what Fell will agree to. Changing under my system is very different than what happened with Kid. You know that. If you don't realize that changing his score was just the last straw in a long line of things he was uncomfortable with, you didn't see anything he had to say.

The not liking criticism thing didn't come out till a few weeks after the initial change, and while it was the final push over the cliff it was a very, very big push.  Let me put it this way, if you changed any of my scores without talking with me first I'd have the same reaction he would.  Fell knows this and has stated several times he will never change a score again without first talking to the submitter.  I do agree the final say on everything is his, but this is something just about any of the long term members could call.

Quote

1) I don't think this is something that needs a remedy nor do I think my system tries to do anything about it. Not a problem.

I disagree, we need to give a hard look at whether or not the item is getting a slightly better score then it deserves based off of any bias.  This involves discussing things with the reviewer more if you feel that's a problem.
Quote

2) I don't see why this affects my argument. It's a feature of all rating systems. If anything, my system proposes a much better reflection of the summary than the existing one.

This affects all our arguments since this whole thread is about letting the score's speak more then the actual content of the review.  If we didn't think this way this discussion wouldn't be going on.

Quote

3) That's why we let editors change scores and make reviewers read an follow the guidelines. Again, I don't see why that affects a decision to change the scale.

No, what we should do is work to actually educate our reviewers on what the scores are.  When I contact people about VG reviews I look at the score and then what they wrote if I don't think they match I e-mail the reviewer and explain that score to them and suggest things to add to make it better reflect.  This has to do with the next number as well.  Too many people are worried about offending people over this when they shouldn't, if someone is offended by advice from an editor about better writing reviews/articles then they shouldn't be writing them in the first place.  We need to flat out say "this isn't good enough please rewrite it".

Quote

4) We haven't been strict, but we have enforced it and made people make changes. Usually this happens via email BEFORE it runs. Thus you don't see it happening. Again, allowing editors to change scores remedies this nicely.

I've done this several times with reviews this year, and past years, that were in the waiting area and not once did you or Fell ever look at any or acknowledge you did such.  If you're going to adopt this policy then don't ask reviewers to submit scores since I doubt any of our originals will ever be used.

« Last Edit: December 19, 2005, 12:49:47 PM by Spriggan »
Screw it, I'm buying crayons and paper. I can imagineer my own adventures! Wheeee!

Chuck Norris is the reason Waldo is hiding.


Spriggan

  • Administrator
  • Level 78
  • *****
  • Posts: 10582
  • Fell Points: 31
  • Yes, I am this awesome
    • View Profile
    • Legacies Lost
Re: Multiple scores
« Reply #48 on: December 19, 2005, 12:47:43 PM »
Part 2 since my post was too long.

Quote

5) none are perfect, but some are more intuitively understood than others. The language is not hard to understand in our description, but the fact that you *need to read* the description is the flaw. The system should be something easily understood without any special knowledge. Our current system fails at that. Maybe we go with 6 clocks instead of 5, but it doesn't matter. The current system isn't progressive and gives a false impression about the overall feeling about the product. That is something we *can* remedy, even if the remedy doesn't make it "perfect." No one's looking for perfect here.


Numbers are intuitive because people can see what it's out of and do the math themselves.  There's no problem with the 6 clock system that the 5, 10 or an 8 would have except that it's unique.  But anyone worth a grain of salt can tell that 5/6 is great but not perfect.  I've never heard of a person reading a review not understanding our system, the only complaint I've heard is a reviewer upset that we don't use a system they prefer.  It's all a matter of personal taste, that's all.

Quote

I think I've given a very reasonable argument above to why this is not at all out of our control.

And that is you want all the control and want to give the reviewers none.
Screw it, I'm buying crayons and paper. I can imagineer my own adventures! Wheeee!

Chuck Norris is the reason Waldo is hiding.


Spriggan

  • Administrator
  • Level 78
  • *****
  • Posts: 10582
  • Fell Points: 31
  • Yes, I am this awesome
    • View Profile
    • Legacies Lost
Re: Multiple scores
« Reply #49 on: December 19, 2005, 12:48:48 PM »
Quote


Thank heavens SOMEONE speaks Spriggan around here. ;D

If e is suggesting that we go to a 5 out of 5 system with the ability to add a sixth clock when a "Citizen Caine" comes along, I'm in total agreement.  


no his suggestion was dumping the 6th all together since it's supposedly too hard for people to grasp.
Screw it, I'm buying crayons and paper. I can imagineer my own adventures! Wheeee!

Chuck Norris is the reason Waldo is hiding.


The Holy Saint, Grand High Poobah, Master of Monkeys, Ehlers

  • Administrator
  • Level 96
  • *****
  • Posts: 19211
  • Fell Points: 17
  • monkeys? yes.
    • View Profile
    • herb's world
Re: Multiple scores
« Reply #50 on: December 19, 2005, 01:00:31 PM »
*sigh*
I won't go into an extended lecture about why 5 is a more comfortable number for nearly everyone than 6 is. It is though. It's easier to intuitively see the differences for the size they are

As for the current system, I'm not talking about math or anything. I'm talking about how the scale, as it is currently written up, is NOT evenly progressive. AS IT IS WRITTEN, the difference between 5.5 and 6.0 is smaller than the difference between 4 and 4.5. That's a problem with our system.

What everyone who's seriously argued with me has missed thus far is this: Neither side is completely right. I'm arguing for system change, but you've conveniently ignored, or tried to wipe away any suggestion at correcting reviewer misuse of the system. Try giving AND taking a little, Sprig. The system is NOT intuitive. I've had publisher NOT link to us and NOT send any more material because they were given 4s and 4.5s. Those were GOOD REVIEWS, but it didn't look like it. The whole fact that we're having this argument shows that there are many confusing things about our system. I've had at least a dozen cases where people couldn't understand it, and a dozen more where people didn't understand it at first. Even if you say the wording is clear, any system that requires you to reference a completely separate document to understand it is non-intuitive. That drives people away.

Clocks/stars/whatever give a quick and dirty summary of the review. You can't get away from that. Either we drop the clocks, or we accept that and make it as easy as possible for that first perception to be as accurate as possible.

Any other course of action is foolishness.
« Last Edit: December 19, 2005, 01:01:23 PM by SaintEhlers »

Spriggan

  • Administrator
  • Level 78
  • *****
  • Posts: 10582
  • Fell Points: 31
  • Yes, I am this awesome
    • View Profile
    • Legacies Lost
Re: Multiple scores
« Reply #51 on: December 19, 2005, 01:14:44 PM »
Quote

As for the current system, I'm not talking about math or anything. I'm talking about how the scale, as it is currently written up, is NOT evenly progressive. AS IT IS WRITTEN, the difference between 5.5 and 6.0 is smaller than the difference between 4 and 4.5. That's a problem with our system.


Ahh, my bad I was reading that completely wrong.  I have no argument with a clarification/rewrite of out guidelines.

Quote

What everyone who's seriously argued with me has missed thus far is this: Neither side is completely right. I'm arguing for system change, but you've conveniently ignored, or tried to wipe away any suggestion at correcting reviewer misuse of the system. Try giving AND taking a little, Sprig. The system is NOT intuitive.

I'm not trying to take that system away, when did I ever state that?  I'm encouraging proactive steps to avoid abuse.  How is what you're suggesting not misuse itself?  You're not partial, you don't think or have the same likes or dislikes as the reviewer.  To me your system is nothing but abusive, all scores will then reflect your and Fell's personal tastes and not anyone else's.

Quote

I've had publisher NOT link to us and NOT send any more material because they were given 4s and 4.5s. Those were GOOD REVIEWS, but it didn't look like it. The whole fact that we're having this argument shows that there are many confusing things about our system. I've had at least a dozen cases where people couldn't understand it, and a dozen more where people didn't understand it at first. Even if you say the wording is clear, any system that requires you to reference a completely separate document to understand it is non-intuitive. That drives people away.

It's probably because they thought it was out of 10, seriously the difference stat wise between a 5 and 6 is so small if they had a problem with it they probably would have had an issue with 4/5.

Quote

Clocks/stars/whatever give a quick and dirty summary of the review. You can't get away from that. Either we drop the clocks, or we accept that and make it as easy as possible for that first perception to be as accurate as possible.

Any other course of action is foolishness.


If the score is causing that big of a problem then lets drop it, there is no score that will not have an initial confusion by someone somewhere, and there will always be publishers that don't like what score they get.
Screw it, I'm buying crayons and paper. I can imagineer my own adventures! Wheeee!

Chuck Norris is the reason Waldo is hiding.


42

  • RPG Editors
  • Level 56
  • *
  • Posts: 4350
  • Fell Points: 8
  • Unofficial World Saver
    • View Profile
Re: Multiple scores
« Reply #52 on: December 19, 2005, 01:20:46 PM »
I don't think there should be a change to 5 from 6.

I don't see how 5 is more intuitive than 6.

I wouldn't mind getting rid of the half-clocks.

I think some reviewers have been writing reviews long enough that it is time for them to step-up the quality of their work.

I really despise this site being used as a way to market products for companies. If a company won't send us product because we don't give them glowing reviews--too bad. Reading a review shouldn't sound like a sales pitch--which is how many of our reviews read.
The Folly of youth is to think that intelligence is a subsitute for experience. The folly of age is to think that experience is a subsitute for intelligence.

Skar

  • Moderator
  • Level 54
  • *****
  • Posts: 3979
  • Fell Points: 7
    • View Profile
Re: Multiple scores
« Reply #53 on: December 19, 2005, 01:30:58 PM »
Quote
I still don't see how its harder to grasp a 6 point system over a 5 point system.  The differences between a 5.5-6 and 3.5-4 SHOULDN'T be significant.  They're not, you just are thinking that way,  each .5 is the same percentage.


The difference between a 5.5 and a 6 is far more significant than the others because it's defined in the submission guidelines as something more than perfect.  The reviewers are told to consider a 5.5 a perfect score and 6 something beyond that.  So you give something a "perfect score" and, visually, it doesn't look perfect.  That's a problem.

Another very real problem is what e is talking about when it comes to intuitively being able to interpret our score.  Readers should not have to reference a seperate document in order to correctly interpret our scores.  If you take a new reader and show him 5.5 out of 6 clocks he's not going to think 5.5 is perfect, as our guidelines state.  He's going to think it fell short of perfection. Problem.

If we lose the 6th clock altogether, we lose the ability to signify truly extraordinary works, the "beyond perfect" that we were trying to be able to show with a 6 clock score, as defined in our guidelines.  So, I still think my solution, 5 out of 5 with a 6th clock getting tagged onto the end in the rare cases, is the way to go.  As for all the old scores, not a problem.  All we have to do is put a note in the FAQ about the switch and when it happened so we don't have to change any old scores at all. That seems to address all the concerns except one.

The one it doesn't address is Spriggan's lack of desire to recode the site.  Perhaps we should take a page out of Spriggan's manual for dealing with reviewers and just tell him that if he doesn't feel like doing the work to maintain the site we'll go to someone else.

I'm not seriously suggesting that we boot Spriggan, anymore than I hope he was serious about telling reviewers to go jump.  
"Skar is the kind of bird who, when you try to kill him with a stone, uses it, and the other bird, to take vengeance on you in a swirling melee of death."

-Fellfrosch

Fellfrosch

  • Administrator
  • Level 68
  • *
  • Posts: 7033
  • Fell Points: 42
  • Walkin' with a dead man over my shoulder.
    • View Profile
    • Fearful Symmetry
Re: Multiple scores
« Reply #54 on: December 19, 2005, 01:32:38 PM »
I am of the opinion that any wholesale change to our rating system would ideally involve dropping numbers altogether and substituting adjectives: Poor, Fair, Good, Excellent, Perfect, etc. (though of course we'd have more than that). That would make it obvious what we mean, without the need to quantify artistic expression.
"Tragedy is when I cut my finger. Comedy is when you walk into an open sewer and die." --Mel Brooks

My author website: http://www.fearfulsymmetry.net

Spriggan

  • Administrator
  • Level 78
  • *****
  • Posts: 10582
  • Fell Points: 31
  • Yes, I am this awesome
    • View Profile
    • Legacies Lost
Re: Multiple scores
« Reply #55 on: December 19, 2005, 01:37:03 PM »
I could see that in addition to the 6 clocks, I really don't think it's something that should be dropped unless we get rid of Scores all together.
Screw it, I'm buying crayons and paper. I can imagineer my own adventures! Wheeee!

Chuck Norris is the reason Waldo is hiding.


Spriggan

  • Administrator
  • Level 78
  • *****
  • Posts: 10582
  • Fell Points: 31
  • Yes, I am this awesome
    • View Profile
    • Legacies Lost
Re: Multiple scores
« Reply #56 on: December 19, 2005, 01:38:59 PM »
Quote
The one it doesn't address is Spriggan's lack of desire to recode the site.  Perhaps we should take a page out of Spriggan's manual for dealing with reviewers and just tell him that if he doesn't feel like doing the work to maintain the site we'll go to someone else.


Actually you're more the welcome to tell me that, but I have nothing to do with this current site.  You'd have to convince Tage to do so since I'm only involved with twg 4.0.  I'm willing to code any changes into that, but we know that it wont be done soon enough for what these changes would warrant.

Quote
I'm not seriously suggesting that we boot Spriggan, anymore than I hope he was serious about telling reviewers to go jump.


I am serious, you work with a reviewer and if they do a substandard job even after multiple reviews and sessions with the editors we should move on to someone that is better.  What substandard isn't for me to decide but I can only think of one person that fit the bill for this and he doesn't even write for us anymore.  If a reviewer can make the grade or insist on ignoring our site's system then they should be asked to stop writing reviews.  I don't see what's so mean about that.
« Last Edit: December 19, 2005, 01:46:54 PM by Spriggan »
Screw it, I'm buying crayons and paper. I can imagineer my own adventures! Wheeee!

Chuck Norris is the reason Waldo is hiding.


stacer

  • Level 58
  • *
  • Posts: 4641
  • Fell Points: 0
    • View Profile
    • Stacy Whitman's Grimoire
Re: Multiple scores
« Reply #57 on: December 19, 2005, 02:30:27 PM »
Except that free reviewers are hard to come by, sometimes.
Help start a small press dedicated to publishing multicultural fantasy and science fiction for children and young adults. http://preview.tinyurl.com/pzojaf.

Follow our blog at http://www.tupublishing.com
We're on Twitter, too! http://www.twitter.com/tupublishing

Spriggan

  • Administrator
  • Level 78
  • *****
  • Posts: 10582
  • Fell Points: 31
  • Yes, I am this awesome
    • View Profile
    • Legacies Lost
Re: Multiple scores
« Reply #58 on: December 19, 2005, 02:38:44 PM »
True, but those that really want to do this are willing to improve and work with the people at the site.  What I don't think we need are those that come in and say our methods are stupid and we should use theirs or not care enough to put in any effort.

There have been several people that have submitted VG reviews to us but weren't up to par, they would have like 200 words for a 5-6 clock review and I'd contact them and explain what I liked about the review and what they should add to make them better.  Basically following Fell's guidelines and then giving us a review that was average length.  None of them write for us anymore probably because they realized they had to put some effort into this and we are serious about having good reviews.

No one expects someone's first few reviews to be perfect or even good, be we should expect a reviewer to challenge themselves to do the best job they can.
« Last Edit: December 19, 2005, 02:39:28 PM by Spriggan »
Screw it, I'm buying crayons and paper. I can imagineer my own adventures! Wheeee!

Chuck Norris is the reason Waldo is hiding.


The Holy Saint, Grand High Poobah, Master of Monkeys, Ehlers

  • Administrator
  • Level 96
  • *****
  • Posts: 19211
  • Fell Points: 17
  • monkeys? yes.
    • View Profile
    • herb's world
Re: Multiple scores
« Reply #59 on: December 19, 2005, 03:29:29 PM »
Quote
I'm not trying to take that system away, when did I ever state that?  I'm encouraging proactive steps to avoid abuse.  How is what you're suggesting not misuse itself?  You're not partial, you don't think or have the same likes or dislikes as the reviewer.  To me your system is nothing but abusive, all scores will then reflect your and Fell's personal tastes and not anyone else's.

This actually ignores what I've already said. The editor changes the score to match what the text says. The choice is based on the text, not whether or not I liked the product. For example, the case of movies. I've been to the theater exactly 4 times this year. I saw Sky High, Fantastic Four, and Star Wars twice. Thus I *can't* have an opinion on movies.

So no, it's not remotely abuse. It's a correction from the people who have the most experience with the system and deal with it constantly for those people who aren't using that system properly.

Otherwise, I have to ask, why the hell am I an editor? To correct spelling? If that's the case, I resign.