Author Topic: Multiple scores  (Read 15305 times)

Mad Dr Jeffe

  • Level 74
  • *
  • Posts: 9162
  • Fell Points: 7
  • Devils Advocate General
    • View Profile
Re: Multiple scores
« Reply #30 on: December 18, 2005, 12:59:02 AM »
According to the FAQ 6 is and has always been the perfect score.

If you as the reviewer feel like the movie was perfect then you should give it a six and back it up... If you cant or wont, then it wasnt perfect and you shouldnt be complaining about not being able to give it a perfect score.

The best way to think of a six is a Congressional Medal of Honor, you have to do a lot to get it. Like wipe out a company of Nazi's with nothing but a service .45 and and pack of chewing gum while you drag your platoon out of harms way.

a 5.5 is a Silver Star, equally heroic, but just short of MOH status. But hey you both get to be buried in Arlington National Cemetary.

5 isnt considered perfect... its exceptional... but not perfect theres that certain special something missing...
The exact quote is....
Quote

5 clocks: Top notch. Fun and innovative, with extremely high production values. You might have one complaint about the work, but it's minor.


You cant make it any clearer than that really. And yes that minor thing is enough to withold the final star, a star that should be there to show that we have higher standards than everyone else.

Both 5.5 and 6 also have clear concise critera...

Quote
6 clocks: Perfect. More than you could have hoped for. There is absolutely nothing wrong with this game/movie. If you give anything this rating, I will ask you to justify it before it gets posted.


This is it the holy grail of movies... Citizen Caine, The Sea Hawk, Charade... its a perfect representation of what cinima should be.

Quote
5.5 clocks: Nearly perfect. Consider this the "perfect" score for most works that do not distinguish themselves above and beyond the call of duty.


Yeah, if one 60 second scene had been cut it would have been perfect.

Both have their place. And both need to be on the scale to give our audience an idea of what we shoot for.

« Last Edit: December 18, 2005, 01:07:13 AM by ElJeffe »
Its an automated robot. Based on Science!

42

  • RPG Editors
  • Level 56
  • *
  • Posts: 4350
  • Fell Points: 8
  • Unofficial World Saver
    • View Profile
Re: Multiple scores
« Reply #31 on: December 18, 2005, 04:24:32 AM »
I can see why not having the sixth clock show would visually make it easier to understand that a 5 clock movie is a very good movie.

Yet, I don't agree that we should remove the sixth clock. We're not trying to sell these movies, just review them. We are not obligated in anyway to present movies as flawless like they were some showroom model. If someone won't go see a movie because we didn't give it a perfect score--that is their own loss. Some people will refuse to see movies for more trivial things than that. We only review films, we do not market them.

Generally, I think the movie going public is intelligent enough (or I hope they are intelligent enough) to make informed decisions about the shows they watch. They can read a review and tell wether or not they agree or disagree with the reviewer. They can tell if they share the same values or same definitions as the reviewer. Then based on their analysis of the review along with their analysis of the promotional material, decide if they think it is worth seeing the movie for themselves.

It's a chance for the reviewer to express his or her opinions and analysis of the film sharing what insight they can bring in an organized manner. The rating assigned is just a way to sum up those opinions. As I see it, any rating system will do, but being consitsant is important.

I think the 6 clock system works just fine. I don't see the 4 star system as being the industry standard. Not one of the film critics I typically read uses the 4 star system, but they do have some sort of rating system. I don't see how the 6 clock rating system is worse than the 4 star, 5 star, grades, thumbs, percentage, or point systems. To avoid confusion, we should stay with the 6 clock system.
« Last Edit: December 18, 2005, 04:35:33 AM by 42 »
The Folly of youth is to think that intelligence is a subsitute for experience. The folly of age is to think that experience is a subsitute for intelligence.

Entsuropi

  • Level 60
  • *
  • Posts: 5033
  • Fell Points: 0
  • =^_^= Captain of the highschool Daydreaming team
    • View Profile
Re: Multiple scores
« Reply #32 on: December 18, 2005, 08:58:25 AM »
And again, if someone is paying that much attention to the clocks that he doesn't go because you gave it 5, not 6, then it's unlikely he was affected much by the review text. Let us not forget the only point of the score is to give a quick-reference to the review, not to be the be-all and end-all of that review.
If you're ever in an argument and Entropy winds up looking staid and temperate in comparison, it might be time to cut your losses and start a new thread about something else :)

Fellfrosch

Spriggan

  • Administrator
  • Level 78
  • *****
  • Posts: 10582
  • Fell Points: 31
  • Yes, I am this awesome
    • View Profile
    • Legacies Lost
Re: Multiple scores
« Reply #33 on: December 18, 2005, 06:22:19 PM »
Quote
Let us not forget the only point of the score is to give a quick-reference to the review, not to be the be-all and end-all of that review.


Exactly.  Couldn't have said it better myself.
Screw it, I'm buying crayons and paper. I can imagineer my own adventures! Wheeee!

Chuck Norris is the reason Waldo is hiding.


Mad Dr Jeffe

  • Level 74
  • *
  • Posts: 9162
  • Fell Points: 7
  • Devils Advocate General
    • View Profile
Re: Multiple scores
« Reply #34 on: December 18, 2005, 07:07:13 PM »
what can one say but Saliari...
Its an automated robot. Based on Science!

The Jade Knight

  • Moderator
  • Level 39
  • *****
  • Posts: 2507
  • Fell Points: 1
  • Lord of the Absent-Minded
    • View Profile
    • Don't go here
Re: Multiple scores
« Reply #35 on: December 18, 2005, 07:58:33 PM »
Maybe we should just use a quota system?  A reviewer cannot give more than 10% of the items he reviews a rating higher than 5 clocks?  Just to help people cool things down?
"Never argue with a fool; they'll bring you down to their level, and then beat you with experience."

Mad Dr Jeffe

  • Level 74
  • *
  • Posts: 9162
  • Fell Points: 7
  • Devils Advocate General
    • View Profile
Re: Multiple scores
« Reply #36 on: December 18, 2005, 08:01:12 PM »
I dont see how they arent calm...

is anyone mad?

I dont think a quota would work... what if a reviewer does 10 films in a year... they can only give one a 5? Even if its a year with lots of other strong contenders?
« Last Edit: December 18, 2005, 08:02:58 PM by ElJeffe »
Its an automated robot. Based on Science!

The Jade Knight

  • Moderator
  • Level 39
  • *****
  • Posts: 2507
  • Fell Points: 1
  • Lord of the Absent-Minded
    • View Profile
    • Don't go here
Re: Multiple scores
« Reply #37 on: December 18, 2005, 08:05:28 PM »
They can give as many 5s as they like.  Only one can have a 5.5 or 6.  At any rate, it's just a suggestion; I agree with many others here that too many 5.5s and 6s are being given out.

I did give out a 5.5, but that was for Mere Christianity, which I feel was near perfect and certainly revolutionary in many ways.
"Never argue with a fool; they'll bring you down to their level, and then beat you with experience."

Mad Dr Jeffe

  • Level 74
  • *
  • Posts: 9162
  • Fell Points: 7
  • Devils Advocate General
    • View Profile
Re: Multiple scores
« Reply #38 on: December 18, 2005, 08:46:52 PM »
if anything the propensity of 5.5's is the fault of the reviewer and not the system... changing the system or putting a limit on the scores probably wont fix that. The big change is that scores generally go down the longer you do reviews...

Its an automated robot. Based on Science!

MsFish

  • Level 44
  • *
  • Posts: 2947
  • Fell Points: 7
  • Geek Girl, Undercover
    • View Profile
Re: Multiple scores
« Reply #39 on: December 19, 2005, 12:19:33 AM »
Quote


Frankly I think Spriggans right...


Those words from Jeffe's mouth have got to be a sign of the apocalypse.
Hold fast to dreams, for when dreams die, life is a broken winged bird that cannot fly.  Hold fast to dreams, for when dreams go, life is a barren field frozen with snow.  -Langston Hughes

Mad Dr Jeffe

  • Level 74
  • *
  • Posts: 9162
  • Fell Points: 7
  • Devils Advocate General
    • View Profile
Re: Multiple scores
« Reply #40 on: December 19, 2005, 12:21:28 AM »
I think so to so why did that have to happen around Christmas...
Its an automated robot. Based on Science!

stacer

  • Level 58
  • *
  • Posts: 4641
  • Fell Points: 0
    • View Profile
    • Stacy Whitman's Grimoire
Re: Multiple scores
« Reply #41 on: December 19, 2005, 12:37:39 AM »
I think Fell's experiment is a good one. Scoring systems differ across genres--books, for example give *one* star if the book is exceptional, and otherwise don't rate the book on a numerical scale at all. I think Skar's suggestion that the sixth clock only show up when a sixth clock is warrented is a good compromise, if it will keep a good reviewer happy.

The worry that we give too many good scores is a fault of a volunteer system, I think. When I want to do a review, it's often because I've read or seen something that struck me--something I've paid to spend my time on. I don't usually care to pay for movies or books that don't merit my attention, so I'm less likely to review them. Back when e was passing random books on to me to review (before I started working full time), I had a more varied range of things to review.

There are a number of review outlets that only review things that have already gone through an initial screening process. Children's books are a great example--The Horn Book magazine only reviews books it feels merit a closer look. Their audience (teachers, booksellers, and librarians) doesn't have the time to read all the books out there, so the magazine works as a way of sifting through the chaff, recommending things they should definitely buy for the library or pay attention to in the bookstore.

So I personally don't worry about high scores. We're simply saying we concentrate on things already worth wasting time on. If we occasionally waste time on something and wish we had that time back, then give it a bad score, sure. But don't worry that scores kind ofew high.
Help start a small press dedicated to publishing multicultural fantasy and science fiction for children and young adults. http://preview.tinyurl.com/pzojaf.

Follow our blog at http://www.tupublishing.com
We're on Twitter, too! http://www.twitter.com/tupublishing

The Holy Saint, Grand High Poobah, Master of Monkeys, Ehlers

  • Administrator
  • Level 96
  • *****
  • Posts: 19211
  • Fell Points: 17
  • monkeys? yes.
    • View Profile
    • herb's world
Re: Multiple scores
« Reply #42 on: December 19, 2005, 09:11:16 AM »
ok, my 2-bits as associate editor. For whatever that counts (nothing?)

Yes, books and movies and music have different "formulas" for rating. But there really isn't a standardized reviewing formula for games. So we had to make one.

and yeah, they have different methods, but they also have different AUDIENCES. People who usually write and read movie reviews don't go to a gaming site for their material. And, like it or not, as much as we like movies and books, TWG's primarily identity is that of a site for gamers and games.

It seems to me that simplifying anything is for the better. I dislike the idea of experimenting with 2 different systems because that adds to the complexity. It means there's more you have to become familiar with to make any judgement. I think it is not selfish or mean to ask our reviewers, especially frequent contributors, to try to understand the system we already use rather than try to impose their own. That seems absurd. It would be more user and editor friendly for writers to at least try to understand our system rather than go with their own.

And, for the record, this article is linked to from the submission page. At the top. It's not like it's hidden from reviewers. Nor is the language complex, imo.

On the other hand, the 6 clock system is not exactly a continuum as described. It is not one that is easily grasped by the average reader. I feel compelled to quote sections of that rating description to publishers whenever I tell them we've reviewed their stuff. I don't think it's remotely fair to expect readers or publishers to have to read our rating description in depth to understand our system. Four clocks is a darn good game. But in terms of percentage, it's only 66%. Where I went to high school, that was nearly a failing grade. And, whatever we've said it really means, that's the impression I believe most people get.

What's the answer then?

I don't think we should do two systems, as I said, but the current system isn't working for everyone.
I think we should change the system. I think a five clock system, where 5 entire clocks are extremely rare, is the best answer. Readers and publishers will understand this system intuitively. Reviewers may need to get used to it, but that's the perils of writing.

I think that editors should have carte blanche to change a reviews score based on the impression they get from the text. If they get the wrong impression, I'm thinking the reviewer should have been more clear. This can be argued about later and changed, but esp. with reviewers who haven't tried to understand the nuances of our scoring system against other systems, it's easier for an editor to change the clocks than to have to wait for a lot of responses to get it cleared up.

I am not for conforming to some other existing system. the 4 stars of movies and the 1 star of books and the 1-10 scale for music doesn't work for everything. We don't review restaurants or hotels. We need our own system, and unless someone comes up with something better, I believe my previous paragraph is the closest anyone has come.

Another thought I had was to at least put some sort of label on the clocks. Vertical text or something that gives a word to go with it. 4 being "pretty darn good" or soemthing, 5 being "nearly perfect," 5.5 being "perfect" and 6 being "beyond perfect"
« Last Edit: December 19, 2005, 09:14:24 AM by SaintEhlers »

42

  • RPG Editors
  • Level 56
  • *
  • Posts: 4350
  • Fell Points: 8
  • Unofficial World Saver
    • View Profile
Re: Multiple scores
« Reply #43 on: December 19, 2005, 09:18:50 AM »
I like the idea of adding text to the clock scores to help clarify what each rating means.
The Folly of youth is to think that intelligence is a subsitute for experience. The folly of age is to think that experience is a subsitute for intelligence.

Spriggan

  • Administrator
  • Level 78
  • *****
  • Posts: 10582
  • Fell Points: 31
  • Yes, I am this awesome
    • View Profile
    • Legacies Lost
Re: Multiple scores
« Reply #44 on: December 19, 2005, 10:37:05 AM »
SE, while your explanation for using 5 clocks is better then Skar's it still doesn't do much to change the problem.  Why would 5 be easier to understand then 6?  And what makes you think saying 5 clock will be extremely rare will keep people from giving them out all the time like 6s are now?  And Fell will never agree to allow for an editor to change any review score without talking to the reviewer first due to the Killowat incident.  And lastly if we move to a different scoring system who is going to go change the 600+ reviews already in the database?  I sure as heck ain't.

Again, I see changing our scoring system as a band-aid fix it will only fix things in the short term due to several reasons already stated by myself and others here.

1) We review a lot of what we like so things tend to get higher scores.

2) Scores are only a quick summary of what the review text states.

3) People have the conception, write or wrong, that if they like something it must be a 6 score.

4) Or editors have never been very strict in enforcing the review criteria, even when reviews that don't support the score are brought to their attention.

5) That no scoring system is perfect since everything is all opinion anyway and there's only so many ways you can convert a 800 word review into a few numbers or letters, none of which are as effective as the initial review.

Most of these issues must be addressed individually, #2 and #5 are things that is out of our control as long as we want to assign a score, or no matter how many times we change around our numbers we'll eventually end up in the same place.
« Last Edit: December 19, 2005, 10:42:20 AM by Spriggan »
Screw it, I'm buying crayons and paper. I can imagineer my own adventures! Wheeee!

Chuck Norris is the reason Waldo is hiding.