Timewaster's Guide Archive

General => Suggestions Box => Topic started by: Spriggan on December 17, 2005, 02:10:19 AM

Title: Multiple scores
Post by: Spriggan on December 17, 2005, 02:10:19 AM
Ok, since everyone has been posting about it elsewhere I'm going to ask people to please post here about it so we have one thread about it instead of lots.

I'll start off with my gripe about it, which is what's the point of having a scoring system for the site if everyone is going to want to use a different one.  Seriously, I can understand if our movie reviewers don't like the 6 score system but I didn't when I first joined, I've always preferred the 10 system.  So now that they're doing it does it mean I can create my own as well?  What about SE, what if he wants to start his own?   This site has always used 6 clocks and I see no reason to change it because one person doesn't like it, I also see no reason to allow them to add their own on top of ours if it doesn't add to the clock system.
Title: Re: Multiple scores
Post by: The Jade Knight on December 17, 2005, 02:55:36 AM
I agree with Sprigganbaum.
Title: Re: Multiple scores
Post by: Mad Dr Jeffe on December 17, 2005, 12:47:24 PM
Also I fail to see why according to the submission guidelines that the clock system doesnt work...

I saw the mention of originality by Mr. Gibbs in the other post, but it isnt the sole criteria we ask for even with the clock system. Yes we ask if the movie has a fresh take,... and we should, this year there is a rehash of the Producers for instance... if it were just a scene by scene remake, why watch it over the original? Do they do anything fresh? Do the actors try different takes on the same characters? These are not only valid questions but important ones.

But lets take the example of Aeon Flux for a moment... sure it had a different idea and it was more original than most.... but did it do it well, and allow for suspension of disbelief? Those questions are also in the critera. It doesnt matter how original a movie is if it sucks... in fact its originality can become a liability and make the movie that much worse... If the actors dont understand the concept how can they carry it off and so on. So as a reviewer you start with a set number of stars...  

The way I see it.
The issue comes in when people think that every movie has to be a 5 or a 6 to be good.
after all 4.5 or 5 clocks is defined as Great....  while 5.5 to 6 is incredibly exceptional... There might be one film every 3-5 years like this. The original Producers for example.

In many ways its better to see our six clock system as a kind of Spinal Tap Gradeing system... yeah it goes up to 6 (or 11) but really its a 4 star system with 2 extra levels.... (3 if you count zero)

Most people seem to get that 4 is a pretty good flick though....

Title: Re: Multiple scores
Post by: Entsuropi on December 17, 2005, 02:03:57 PM
Quote
The issue comes in when people think that every movie has to be a 5 or a 6 to be good.
after all 4.5 or 5 clocks is defined as Great....  while 5.5 to 6 is incredibly exceptional... There might be one film every 3-5 years like this. The original Producers for example.


Notably, the brothers like to give out 5.5 and 6 scores quite often, and maybe they want to be able to give out 'top marks' without treading on the current scoring...

But even so, I simply feel that if there is two scores then that is two numbers I'm not paying much attention to. The number should sum up the text, not stand in it's place, and from that perspective changing to a 4 star system is fairly pointless.
Title: Re: Multiple scores
Post by: Fellfrosch on December 17, 2005, 02:33:44 PM
The primary reason for this two-score experiment is that virtually every movie reviewer we've ever had has hated and misused the clock system. The way we think of it, and the way Jeffe described it, is great for games and theoretically great for movies, yet none of our movie reviewers have ever liked it or felt comfortable with it. Rather than lose another one (we go through movie reviewers like candy), I've decided to try this two-score system to see what happens.
Title: Re: Multiple scores
Post by: Spriggan on December 17, 2005, 02:55:24 PM
then why bother with the 6 clock system if it's to hard to use?  Honestly having two different scoring systems on one article is incredibly stupid that's why no one uses it.  It's confusing, contradictory, and defeats the purpose of having scores.

Title: Re: Multiple scores
Post by: Mad Dr Jeffe on December 17, 2005, 03:30:21 PM
Ive never seen it as hard to use... ever...
I just see it as someone wanting it to be done their way rather than working with our system... No offense meant to anyone who doesnt like the 6 clocks system but whats so difficult about it? And whats to hate? The only thing I can infer is that you didnt read all the guidelines. I mean our six clock system is just like any 4 star system with two more levels... (or 3 if you count the never ever been used Zero.
It just moves the mid level run of the mill just ok film to 3 or 3.5 rather than putting it at 2 or 2.5. And gives the reviewer a better middle spread to boot. Sort of an It was pretty good, good enough to go see in the theater but not good enough to go opening week kind of review.


Having said that...

I could see applying the 6 clock system to more variables...

Like 3 clocks for dialog 3 for costuming 6 for cgi and 2 for weak bladder penalty (the 3 hour + factor)

Title: Re: Multiple scores
Post by: Spriggan on December 17, 2005, 03:33:00 PM
Quote

I could see applying the 6 clock system to more variables...

Ie 3 clocks for dialog 3 for costuming 6 for cgi and 2 for weak bladder penalty (the 3 hour + factor)



I use to do this with video games and some anime I reviewed way back when, we also use it for webcomics and books sometimes.  I think that's a great way to explain your score more if you want to use it.
Title: Re: Multiple scores
Post by: Mad Dr Jeffe on December 17, 2005, 03:38:14 PM
I could deal with that...
I just dont get how decreasing the range makes it better.

If someone really has a good reason I wouldnt mind hearing it...

I guess what Im saying to the people who want to change the system is convince us that your way is better...
Title: Re: Multiple scores
Post by: Entsuropi on December 17, 2005, 03:40:14 PM
Hrm. We lost kid kilowatt because he didn't like negative feedback to his reviews. We lost michael somethingorother because, uh, he left? Who else did we lose, because I wasn't aware we lost any of them due to the clock system :(
Title: Re: Multiple scores
Post by: Mad Dr Jeffe on December 17, 2005, 03:41:08 PM
Skar doesnt seem to mind the clock system.

Do ya Skar?
Title: Re: Multiple scores
Post by: Spriggan on December 17, 2005, 03:43:41 PM
Kid left more because of Fell disagreed with his score and changed it (made it better) of the second matrix movie.
Title: Re: Multiple scores
Post by: Mad Dr Jeffe on December 17, 2005, 03:48:04 PM
well and because people like me antagonized him over his RPG reviews...

But anyone who gets mad over what Fell does is a big baby....

:D
Title: Re: Multiple scores
Post by: Spriggan on December 17, 2005, 04:05:38 PM
Well, in that case he had a right to get upset since it was his review and no one had the right to change his score without talking to him.  But that's a discussion for another place and time.
Title: Re: Multiple scores
Post by: Skar on December 17, 2005, 05:05:20 PM
Quote
Skar doesnt seem to mind the clock system.

Do ya Skar?



The problem as I see it is this:  Real 6 clock movies (11s in spinal tap world), as someone pointed out, come along a few every 3-5 years.  They are so rare as to be almost outside the regular movie continuum.  By that I mean most four star movies in the regular reviewer's universe are not a 6 clocker in ours.  

The sixth clock should be a rare and "better than top marks" score.  The answer would seem simple.  Give 5 clocks to the movies you'd regularly give 4 stars and adjust the scale accordingly.  The problem arises with that final empty and fishwhite sixth clock stuck there on the end in every case.  You give a movie top marks, 5 clocks-4 stars, but it still looks to the casual observer, because of that empty clock, like it fell short of the mark.

My suggestion is to make 5 clocks all that show up.  So a five clock score looks full and maximized.  Then when the three to five year wonder comes along, throw on the extra clock.  This would probably require some coding to take place and thus will probably have to wait until the new site occurs.

Did that make any sense?
Title: Re: Multiple scores
Post by: The Jade Knight on December 17, 2005, 05:13:02 PM
Makes sense to me.  Would there ever be 5.5 clocks?
Title: Re: Multiple scores
Post by: Peter Ahlstrom on December 17, 2005, 05:16:20 PM
I agree with the merits of the "6th clock appears only if something gets 6 clocks" idea.
Title: Re: Multiple scores
Post by: Mad Dr Jeffe on December 17, 2005, 05:29:33 PM
I dont think people think its short of the mark though...

take restaurants which typically go up to 5 stars, people dont say, "oh it just got 4 stars, it must just be good  and not great." Somewhere above three it becomes great or very good and the 5 becomes sublime... People understand that 4 stars is really good... but that there is a higher level of quality an establishment can obtain...
However if you just want to
Perhaps if we just showed the number of Clocks the movie (or book or whatever) that would be enough to mollify folks.

So a 2.5 just gets 2.5 clocks and readers assume it gets more...
Title: Re: Multiple scores
Post by: Shrain on December 17, 2005, 05:29:46 PM
Quote
I agree with the merits of the "6th clock appears only if something gets 6 clocks" idea.

Totally.
I'd hate to see the clock system go caput. I like the extra spread it allows. To me, it makes the ratings more accurate.
Always having an empty clock might give the wrong impression. So I think Froskar's idea makes god sense.
Title: Re: Multiple scores
Post by: Spriggan on December 17, 2005, 06:02:18 PM
I don't think that will help, since then users have no idea how many points its out of?  2.5 out of 4? out of 5? out of 10? out of 100?  Considering that we have an unique number system, out of 6, we should keep it the way it is.  
Title: Re: Multiple scores
Post by: Mad Dr Jeffe on December 17, 2005, 06:21:11 PM
I agree actually the more I think about it. So what it we end up having 5 out of six a lot. Its not like it will hurt the review. People will just think we're being honest.  When we do have six out of six it will mean something and everyone will know.

Plus there is a place for 5.5 clocks...
Title: Re: Multiple scores
Post by: Skar on December 17, 2005, 07:01:20 PM
Under my suggestion, 5 clocks would always appear.  A score of 2.5 would include 2.5 empty clocks.  That way a 6th clock really means something.

It does give the wrong impression if there's always a 6th empty clock.  The industry standard in movie reviewing and pretty much anything else, (perhaps restaurants are an exception) seems to be that an excellent film gets full marks.  We simply reserve the right to add an extra clock if a film, game, whatever, goes beyond that.

As for there being 5.5 clocks.  Sure.  the half full clock only appears if there's a score of 5.5.
Title: Re: Multiple scores
Post by: Skar on December 17, 2005, 07:04:46 PM
Quote
Considering that we have an unique number system, out of 6, we should keep it the way it is.  


But that's part of the problem.  It's not really out of 6.  It's actually out of 5, with the possibility of getting 6 if the piece is over-the-top-amazing-good.

Movie reviewers and others hand out 4 stars for far more than we think is reasonable to hand out 6 clocks.  4 in an outside system equates more to 5 in our system than a 6.
Title: Re: Multiple scores
Post by: Spriggan on December 17, 2005, 07:10:31 PM
Well then you're suggesting changing our actually scoring system which is something I don't agree with.  It's 6 clocks for a reason (it has to do with hours) not because 6 was a fun number to use.
Title: Re: Multiple scores
Post by: Mad Dr Jeffe on December 17, 2005, 07:13:09 PM
Why should a movie get full marks when it doesnt deserve it. Why pretend a 5 is a six? If you think it deserves a 6 give it one, and explain why. Fell will run it if you back it up. Ive had a six on a review I did for hero, because I backed it up. (which appears to be gone now... but we have several 5.5's) (although I would probably change it If I did it again) If the movie doesnt deserve a six, then maybe it does deserve a 5 (excellent but not perfect like the rating says). Why change that when it explains our policy perfectly.

And it is out of 6 ... or 7 actually...
I mean it could also get a zero, which we've never given out.
Should we get rid of zero too?




Title: Re: Multiple scores
Post by: Spriggan on December 17, 2005, 07:26:46 PM
We've had one or two 0s before but I believe Fell made them change it (one of our old movie reviwers gave Panic Room a 0).  And if you want to get techincal our score is out of 14 not 7 (you have to doubble the number since there are .5s).

The reason I'm against not having the 6th clock appear unless selected is because the problem that lead to this discussion isn't a problem with the review system its a problem with people always giving out 5.5 or 6 reviews (most noticeably our movie reviewers).  So saw we implement Skar's system and now all the 6 clock movies become 5 clock, well now 5 is considered the perfect score because people don't see the 6th clock so we get the same comments like Mustard made saying they give things too high of scores.  So we come back here and have another discussion this time with the solution being "lets make it out of 4 clocks then" and we just start a cycle.

This is just a band-aid fix and all it does is temporally calm people down instead of actually fixing the problem which is just because you like something doesn't mean its a 6 clock movie, book, game, etc.  Fell is the type of guy that doesn't like to step on toes and while that's great most of the time in this case it's not.  Some one needs to flat out say, if you can't adjust to our 6 clock system then don't write reviews for us.  We've been using it for almost 5 years now and so far only one person has really complained about it so we're not going to change just because ONE person doesn't like it when everyone else does.
Title: Re: Multiple scores
Post by: Mad Dr Jeffe on December 17, 2005, 07:37:54 PM
I will say that lately Ive had to rethink my review numbers because I feel like I was being too nice to the company or the product. Some of that is because I've had to resort to reviewing products I love because TWG east hasnt been able to get a steady stream of product. The best reviews I think I've done got poor reviews instead of good ones. I had to think a lot more for them.
Still there are several products I would re-review If I had the time and inclinaiton. I might even do so...
Something along the line of XXXXX a year later.

I dont know how it is on the Movie review side but its tempting to give things good reviews on the RPG side because hey we get this stuff for free half the time. But maybe its something we should address with the reviews themselves and not the number of clocks.

Frankly I think Spriggans right... make five the new top dog and everyone will be doing fives.
Title: Re: Multiple scores
Post by: fuzzyoctopus on December 17, 2005, 08:19:28 PM
Quote
I agree with Sprigganbaum.


I also agree with him.  Welcome to the Department of Redundancy Department.
Title: Re: Multiple scores
Post by: Mad Dr Jeffe on December 17, 2005, 08:37:27 PM
thanks I plan to pay the department after I go to the automatic teller machine machine...
Title: Re: Multiple scores
Post by: Skar on December 17, 2005, 11:50:10 PM
Quote

5.5 clocks: Nearly perfect. Consider this the "perfect" score for most works that do not distinguish themselves above and beyond the call of duty.


So, right now 5.5 clocks is supposed to be considered the perfect score with 6 being something beyond that.

I get the impression that 6 was supposed to be the 11 on the volume knob, and extremely rare.  That does not describe most 4 star movies.

The root problem here is that 5.5, with half an empty clock still to go visually, is supposed to stand for perfect.  The half an empty clock doesn't look perfect.  It looks like it fell short of it by half a clock.  When a reviewer wants to give something a "perfect score" they don't want it to LOOK like it was half a clock short of perfect.  That's reasonable in my opinion.

The reason Fell wants an explanation for every 6 clock review (correct me if I'm wrong Fell) is because 6 is supposed to be something truly special, something beyond a normal run-of-the-mill perfect score.  It's outside the normal continuum.

Making 5 clocks appear with the 6th popping up only in the truly amazing instances is true to the spirit of our rating system.  Having an empty clock hanging there on a "perfect" score is misleading and doesn't represent what we're trying to say.  It just takes away from the 5's perfectness.
Title: Re: Multiple scores
Post by: Mad Dr Jeffe on December 18, 2005, 12:59:02 AM
According to the FAQ 6 is and has always been the perfect score.

If you as the reviewer feel like the movie was perfect then you should give it a six and back it up... If you cant or wont, then it wasnt perfect and you shouldnt be complaining about not being able to give it a perfect score.

The best way to think of a six is a Congressional Medal of Honor, you have to do a lot to get it. Like wipe out a company of Nazi's with nothing but a service .45 and and pack of chewing gum while you drag your platoon out of harms way.

a 5.5 is a Silver Star, equally heroic, but just short of MOH status. But hey you both get to be buried in Arlington National Cemetary.

5 isnt considered perfect... its exceptional... but not perfect theres that certain special something missing...
The exact quote is....
Quote

5 clocks: Top notch. Fun and innovative, with extremely high production values. You might have one complaint about the work, but it's minor.


You cant make it any clearer than that really. And yes that minor thing is enough to withold the final star, a star that should be there to show that we have higher standards than everyone else.

Both 5.5 and 6 also have clear concise critera...

Quote
6 clocks: Perfect. More than you could have hoped for. There is absolutely nothing wrong with this game/movie. If you give anything this rating, I will ask you to justify it before it gets posted.


This is it the holy grail of movies... Citizen Caine, The Sea Hawk, Charade... its a perfect representation of what cinima should be.

Quote
5.5 clocks: Nearly perfect. Consider this the "perfect" score for most works that do not distinguish themselves above and beyond the call of duty.


Yeah, if one 60 second scene had been cut it would have been perfect.

Both have their place. And both need to be on the scale to give our audience an idea of what we shoot for.

Title: Re: Multiple scores
Post by: 42 on December 18, 2005, 04:24:32 AM
I can see why not having the sixth clock show would visually make it easier to understand that a 5 clock movie is a very good movie.

Yet, I don't agree that we should remove the sixth clock. We're not trying to sell these movies, just review them. We are not obligated in anyway to present movies as flawless like they were some showroom model. If someone won't go see a movie because we didn't give it a perfect score--that is their own loss. Some people will refuse to see movies for more trivial things than that. We only review films, we do not market them.

Generally, I think the movie going public is intelligent enough (or I hope they are intelligent enough) to make informed decisions about the shows they watch. They can read a review and tell wether or not they agree or disagree with the reviewer. They can tell if they share the same values or same definitions as the reviewer. Then based on their analysis of the review along with their analysis of the promotional material, decide if they think it is worth seeing the movie for themselves.

It's a chance for the reviewer to express his or her opinions and analysis of the film sharing what insight they can bring in an organized manner. The rating assigned is just a way to sum up those opinions. As I see it, any rating system will do, but being consitsant is important.

I think the 6 clock system works just fine. I don't see the 4 star system as being the industry standard. Not one of the film critics I typically read uses the 4 star system, but they do have some sort of rating system. I don't see how the 6 clock rating system is worse than the 4 star, 5 star, grades, thumbs, percentage, or point systems. To avoid confusion, we should stay with the 6 clock system.
Title: Re: Multiple scores
Post by: Entsuropi on December 18, 2005, 08:58:25 AM
And again, if someone is paying that much attention to the clocks that he doesn't go because you gave it 5, not 6, then it's unlikely he was affected much by the review text. Let us not forget the only point of the score is to give a quick-reference to the review, not to be the be-all and end-all of that review.
Title: Re: Multiple scores
Post by: Spriggan on December 18, 2005, 06:22:19 PM
Quote
Let us not forget the only point of the score is to give a quick-reference to the review, not to be the be-all and end-all of that review.


Exactly.  Couldn't have said it better myself.
Title: Re: Multiple scores
Post by: Mad Dr Jeffe on December 18, 2005, 07:07:13 PM
what can one say but Saliari...
Title: Re: Multiple scores
Post by: The Jade Knight on December 18, 2005, 07:58:33 PM
Maybe we should just use a quota system?  A reviewer cannot give more than 10% of the items he reviews a rating higher than 5 clocks?  Just to help people cool things down?
Title: Re: Multiple scores
Post by: Mad Dr Jeffe on December 18, 2005, 08:01:12 PM
I dont see how they arent calm...

is anyone mad?

I dont think a quota would work... what if a reviewer does 10 films in a year... they can only give one a 5? Even if its a year with lots of other strong contenders?
Title: Re: Multiple scores
Post by: The Jade Knight on December 18, 2005, 08:05:28 PM
They can give as many 5s as they like.  Only one can have a 5.5 or 6.  At any rate, it's just a suggestion; I agree with many others here that too many 5.5s and 6s are being given out.

I did give out a 5.5, but that was for Mere Christianity, which I feel was near perfect and certainly revolutionary in many ways.
Title: Re: Multiple scores
Post by: Mad Dr Jeffe on December 18, 2005, 08:46:52 PM
if anything the propensity of 5.5's is the fault of the reviewer and not the system... changing the system or putting a limit on the scores probably wont fix that. The big change is that scores generally go down the longer you do reviews...

Title: Re: Multiple scores
Post by: MsFish on December 19, 2005, 12:19:33 AM
Quote


Frankly I think Spriggans right...


Those words from Jeffe's mouth have got to be a sign of the apocalypse.
Title: Re: Multiple scores
Post by: Mad Dr Jeffe on December 19, 2005, 12:21:28 AM
I think so to so why did that have to happen around Christmas...
Title: Re: Multiple scores
Post by: stacer on December 19, 2005, 12:37:39 AM
I think Fell's experiment is a good one. Scoring systems differ across genres--books, for example give *one* star if the book is exceptional, and otherwise don't rate the book on a numerical scale at all. I think Skar's suggestion that the sixth clock only show up when a sixth clock is warrented is a good compromise, if it will keep a good reviewer happy.

The worry that we give too many good scores is a fault of a volunteer system, I think. When I want to do a review, it's often because I've read or seen something that struck me--something I've paid to spend my time on. I don't usually care to pay for movies or books that don't merit my attention, so I'm less likely to review them. Back when e was passing random books on to me to review (before I started working full time), I had a more varied range of things to review.

There are a number of review outlets that only review things that have already gone through an initial screening process. Children's books are a great example--The Horn Book magazine only reviews books it feels merit a closer look. Their audience (teachers, booksellers, and librarians) doesn't have the time to read all the books out there, so the magazine works as a way of sifting through the chaff, recommending things they should definitely buy for the library or pay attention to in the bookstore.

So I personally don't worry about high scores. We're simply saying we concentrate on things already worth wasting time on. If we occasionally waste time on something and wish we had that time back, then give it a bad score, sure. But don't worry that scores kind ofew high.
Title: Re: Multiple scores
Post by: The Holy Saint, Grand High Poobah, Master of Monkeys, Ehlers on December 19, 2005, 09:11:16 AM
ok, my 2-bits as associate editor. For whatever that counts (nothing?)

Yes, books and movies and music have different "formulas" for rating. But there really isn't a standardized reviewing formula for games. So we had to make one.

and yeah, they have different methods, but they also have different AUDIENCES. People who usually write and read movie reviews don't go to a gaming site for their material. And, like it or not, as much as we like movies and books, TWG's primarily identity is that of a site for gamers and games.

It seems to me that simplifying anything is for the better. I dislike the idea of experimenting with 2 different systems because that adds to the complexity. It means there's more you have to become familiar with to make any judgement. I think it is not selfish or mean to ask our reviewers, especially frequent contributors, to try to understand the system we already use rather than try to impose their own. That seems absurd. It would be more user and editor friendly for writers to at least try to understand our system rather than go with their own.

And, for the record, this article (http://www.timewastersguide.com/view.php?id=530) is linked to from the submission page. At the top. It's not like it's hidden from reviewers. Nor is the language complex, imo.

On the other hand, the 6 clock system is not exactly a continuum as described. It is not one that is easily grasped by the average reader. I feel compelled to quote sections of that rating description to publishers whenever I tell them we've reviewed their stuff. I don't think it's remotely fair to expect readers or publishers to have to read our rating description in depth to understand our system. Four clocks is a darn good game. But in terms of percentage, it's only 66%. Where I went to high school, that was nearly a failing grade. And, whatever we've said it really means, that's the impression I believe most people get.

What's the answer then?

I don't think we should do two systems, as I said, but the current system isn't working for everyone.
I think we should change the system. I think a five clock system, where 5 entire clocks are extremely rare, is the best answer. Readers and publishers will understand this system intuitively. Reviewers may need to get used to it, but that's the perils of writing.

I think that editors should have carte blanche to change a reviews score based on the impression they get from the text. If they get the wrong impression, I'm thinking the reviewer should have been more clear. This can be argued about later and changed, but esp. with reviewers who haven't tried to understand the nuances of our scoring system against other systems, it's easier for an editor to change the clocks than to have to wait for a lot of responses to get it cleared up.

I am not for conforming to some other existing system. the 4 stars of movies and the 1 star of books and the 1-10 scale for music doesn't work for everything. We don't review restaurants or hotels. We need our own system, and unless someone comes up with something better, I believe my previous paragraph is the closest anyone has come.

Another thought I had was to at least put some sort of label on the clocks. Vertical text or something that gives a word to go with it. 4 being "pretty darn good" or soemthing, 5 being "nearly perfect," 5.5 being "perfect" and 6 being "beyond perfect"
Title: Re: Multiple scores
Post by: 42 on December 19, 2005, 09:18:50 AM
I like the idea of adding text to the clock scores to help clarify what each rating means.
Title: Re: Multiple scores
Post by: Spriggan on December 19, 2005, 10:37:05 AM
SE, while your explanation for using 5 clocks is better then Skar's it still doesn't do much to change the problem.  Why would 5 be easier to understand then 6?  And what makes you think saying 5 clock will be extremely rare will keep people from giving them out all the time like 6s are now?  And Fell will never agree to allow for an editor to change any review score without talking to the reviewer first due to the Killowat incident.  And lastly if we move to a different scoring system who is going to go change the 600+ reviews already in the database?  I sure as heck ain't.

Again, I see changing our scoring system as a band-aid fix it will only fix things in the short term due to several reasons already stated by myself and others here.

1) We review a lot of what we like so things tend to get higher scores.

2) Scores are only a quick summary of what the review text states.

3) People have the conception, write or wrong, that if they like something it must be a 6 score.

4) Or editors have never been very strict in enforcing the review criteria, even when reviews that don't support the score are brought to their attention.

5) That no scoring system is perfect since everything is all opinion anyway and there's only so many ways you can convert a 800 word review into a few numbers or letters, none of which are as effective as the initial review.

Most of these issues must be addressed individually, #2 and #5 are things that is out of our control as long as we want to assign a score, or no matter how many times we change around our numbers we'll eventually end up in the same place.
Title: Re: Multiple scores
Post by: The Holy Saint, Grand High Poobah, Master of Monkeys, Ehlers on December 19, 2005, 11:59:32 AM
Quote
SE, while your explanation for using 5 clocks is better then Skar's it still doesn't do much to change the problem.  Why would 5 be easier to understand then 6?  And what makes you think saying 5 clock will be extremely rare will keep people from giving them out all the time like 6s are now?  And Fell will never agree to allow for an editor to change any review score without talking to the reviewer first due to the Killowat incident.  And lastly if we move to a different scoring system who is going to go change the 600+ reviews already in the database?  I sure as heck ain't.

Yes, it does. Because the 5 clock system would be SCALED, instead of being like D&D levels.
See, the way we have it right now, it becomes progressively more difficult to move to the next half-clock the more clocks you already have. The difference between 5.5 and 6 clocks is much less than the difference between 3.5 and 4 clocks. That's what people don't understand. 6 clocks is just an odd number that serves no real purpose.

We'll work out who changes what when we decide what to change to. If nothing else, I can change 2 or 3 a day. no biggie.

And how about we let FELL decide what Fell will agree to. Changing under my system is very different than what happened with Kid. You know that. If you don't realize that changing his score was just hte last straw in a long line of things he was uncomfortable with, you didn't see anything he had to say.



Quote
Again, I see changing our scoring system as a band-aid fix it will only fix things in the short term due to several reasons already stated by myself and others here.

1) We review a lot of what we like so things tend to get higher scores.

2) Scores are only a quick summary of what the review text states.

3) People have the conception, write or wrong, that if they like something it must be a 6 score.

4) Or editors have never been very strict in enforcing the review criteria, even when reviews that don't support the score are brought to their attention.

5) That no scoring system is perfect since everything is all opinion anyway and there's only so many ways you can convert a 800 word review into a few numbers or letters, none of which are as effective as the initial review.

1) I don't think this is something that needs a remedy nor do I think my system tries to do anything about it. Not a problem.
2) I don't see why this affects my argument. It's a feature of all rating systems. If anything, my system proposes a much better reflection of the summary than the existing one.
3) That's why we let editors change scores and make reviewers read an follow the guidelines. Again, I don't see why that affects a decision to change the scale.
4) We haven't been strict, but we have enforced it and made people make changes. Usually this happens via email BEFORE it runs. Thus you don't see it happening. Again, allowing editors to change scores remedies this nicely.
5) none are perfect, but some are more intuitively understood than others. The language is not hard to understand in our description, but the fact that you *need to read* the description is the flaw. The system should be something easily understood without any special knowledge. Our current system fails at that. Maybe we go with 6 clocks instead of 5, but it doesn't matter. THe current system isn't progressive and gives a false impression about the overall feeling about the product. THat is something we *can* remedy, even if the remedy doesn't make it "perfect." No one's looking for perfect here.

Quote
Most of these issues must be addressed individually, #2 and #5 are things that is out of our control as long as we want to assign a score, or no matter how many times we change around our numbers we'll eventually end up in the same place.

I think I've given a very reasonable argument above to why this is not at all out of our control.
Title: Re: Multiple scores
Post by: Skar on December 19, 2005, 12:43:50 PM
Quote
SE, while your explanation for using 5 clocks is better then Skar's...


Thank heavens SOMEONE speaks Spriggan around here. ;D

If e is suggesting that we go to a 5 out of 5 system with the ability to add a sixth clock when a "Citizen Caine" comes along, I'm in total agreement.  
Title: Re: Multiple scores
Post by: Spriggan on December 19, 2005, 12:47:23 PM
Quote

Yes, it does. Because the 5 clock system would be SCALED, instead of being like D&D levels.
See, the way we have it right now, it becomes progressively more difficult to move to the next half-clock the more clocks you already have. The difference between 5.5 and 6 clocks is much less than the difference between 3.5 and 4 clocks. That's what people don't understand. 6 clocks is just an odd number that serves no real purpose.

I still don't see how its harder to grasp a 6 point system over a 5 point system.  The differences between a 5.5-6 and 3.5-4 SHOULDN'T be significant.  They're not, you just are thinking that way,  each .5 is the same percentage.

Quote

We'll work out who changes what when we decide what to change to. If nothing else, I can change 2 or 3 a day. no biggie.

Not to be rude, but I doubt you or anyone else will manage to be consistent enough to actually get this done, let alone having to change the site code to reflect a max of 5.  Also, since we'd have to change our review criteria, we are obligated to contact all the original reviewers and ask them what they would change the score too.

Quote

And how about we let FELL decide what Fell will agree to. Changing under my system is very different than what happened with Kid. You know that. If you don't realize that changing his score was just the last straw in a long line of things he was uncomfortable with, you didn't see anything he had to say.

The not liking criticism thing didn't come out till a few weeks after the initial change, and while it was the final push over the cliff it was a very, very big push.  Let me put it this way, if you changed any of my scores without talking with me first I'd have the same reaction he would.  Fell knows this and has stated several times he will never change a score again without first talking to the submitter.  I do agree the final say on everything is his, but this is something just about any of the long term members could call.

Quote

1) I don't think this is something that needs a remedy nor do I think my system tries to do anything about it. Not a problem.

I disagree, we need to give a hard look at whether or not the item is getting a slightly better score then it deserves based off of any bias.  This involves discussing things with the reviewer more if you feel that's a problem.
Quote

2) I don't see why this affects my argument. It's a feature of all rating systems. If anything, my system proposes a much better reflection of the summary than the existing one.

This affects all our arguments since this whole thread is about letting the score's speak more then the actual content of the review.  If we didn't think this way this discussion wouldn't be going on.

Quote

3) That's why we let editors change scores and make reviewers read an follow the guidelines. Again, I don't see why that affects a decision to change the scale.

No, what we should do is work to actually educate our reviewers on what the scores are.  When I contact people about VG reviews I look at the score and then what they wrote if I don't think they match I e-mail the reviewer and explain that score to them and suggest things to add to make it better reflect.  This has to do with the next number as well.  Too many people are worried about offending people over this when they shouldn't, if someone is offended by advice from an editor about better writing reviews/articles then they shouldn't be writing them in the first place.  We need to flat out say "this isn't good enough please rewrite it".

Quote

4) We haven't been strict, but we have enforced it and made people make changes. Usually this happens via email BEFORE it runs. Thus you don't see it happening. Again, allowing editors to change scores remedies this nicely.

I've done this several times with reviews this year, and past years, that were in the waiting area and not once did you or Fell ever look at any or acknowledge you did such.  If you're going to adopt this policy then don't ask reviewers to submit scores since I doubt any of our originals will ever be used.

Title: Re: Multiple scores
Post by: Spriggan on December 19, 2005, 12:47:43 PM
Part 2 since my post was too long.

Quote

5) none are perfect, but some are more intuitively understood than others. The language is not hard to understand in our description, but the fact that you *need to read* the description is the flaw. The system should be something easily understood without any special knowledge. Our current system fails at that. Maybe we go with 6 clocks instead of 5, but it doesn't matter. The current system isn't progressive and gives a false impression about the overall feeling about the product. That is something we *can* remedy, even if the remedy doesn't make it "perfect." No one's looking for perfect here.


Numbers are intuitive because people can see what it's out of and do the math themselves.  There's no problem with the 6 clock system that the 5, 10 or an 8 would have except that it's unique.  But anyone worth a grain of salt can tell that 5/6 is great but not perfect.  I've never heard of a person reading a review not understanding our system, the only complaint I've heard is a reviewer upset that we don't use a system they prefer.  It's all a matter of personal taste, that's all.

Quote

I think I've given a very reasonable argument above to why this is not at all out of our control.

And that is you want all the control and want to give the reviewers none.
Title: Re: Multiple scores
Post by: Spriggan on December 19, 2005, 12:48:48 PM
Quote


Thank heavens SOMEONE speaks Spriggan around here. ;D

If e is suggesting that we go to a 5 out of 5 system with the ability to add a sixth clock when a "Citizen Caine" comes along, I'm in total agreement.  


no his suggestion was dumping the 6th all together since it's supposedly too hard for people to grasp.
Title: Re: Multiple scores
Post by: The Holy Saint, Grand High Poobah, Master of Monkeys, Ehlers on December 19, 2005, 01:00:31 PM
*sigh*
I won't go into an extended lecture about why 5 is a more comfortable number for nearly everyone than 6 is. It is though. It's easier to intuitively see the differences for the size they are

As for the current system, I'm not talking about math or anything. I'm talking about how the scale, as it is currently written up, is NOT evenly progressive. AS IT IS WRITTEN, the difference between 5.5 and 6.0 is smaller than the difference between 4 and 4.5. That's a problem with our system.

What everyone who's seriously argued with me has missed thus far is this: Neither side is completely right. I'm arguing for system change, but you've conveniently ignored, or tried to wipe away any suggestion at correcting reviewer misuse of the system. Try giving AND taking a little, Sprig. The system is NOT intuitive. I've had publisher NOT link to us and NOT send any more material because they were given 4s and 4.5s. Those were GOOD REVIEWS, but it didn't look like it. The whole fact that we're having this argument shows that there are many confusing things about our system. I've had at least a dozen cases where people couldn't understand it, and a dozen more where people didn't understand it at first. Even if you say the wording is clear, any system that requires you to reference a completely separate document to understand it is non-intuitive. That drives people away.

Clocks/stars/whatever give a quick and dirty summary of the review. You can't get away from that. Either we drop the clocks, or we accept that and make it as easy as possible for that first perception to be as accurate as possible.

Any other course of action is foolishness.
Title: Re: Multiple scores
Post by: Spriggan on December 19, 2005, 01:14:44 PM
Quote

As for the current system, I'm not talking about math or anything. I'm talking about how the scale, as it is currently written up, is NOT evenly progressive. AS IT IS WRITTEN, the difference between 5.5 and 6.0 is smaller than the difference between 4 and 4.5. That's a problem with our system.


Ahh, my bad I was reading that completely wrong.  I have no argument with a clarification/rewrite of out guidelines.

Quote

What everyone who's seriously argued with me has missed thus far is this: Neither side is completely right. I'm arguing for system change, but you've conveniently ignored, or tried to wipe away any suggestion at correcting reviewer misuse of the system. Try giving AND taking a little, Sprig. The system is NOT intuitive.

I'm not trying to take that system away, when did I ever state that?  I'm encouraging proactive steps to avoid abuse.  How is what you're suggesting not misuse itself?  You're not partial, you don't think or have the same likes or dislikes as the reviewer.  To me your system is nothing but abusive, all scores will then reflect your and Fell's personal tastes and not anyone else's.

Quote

I've had publisher NOT link to us and NOT send any more material because they were given 4s and 4.5s. Those were GOOD REVIEWS, but it didn't look like it. The whole fact that we're having this argument shows that there are many confusing things about our system. I've had at least a dozen cases where people couldn't understand it, and a dozen more where people didn't understand it at first. Even if you say the wording is clear, any system that requires you to reference a completely separate document to understand it is non-intuitive. That drives people away.

It's probably because they thought it was out of 10, seriously the difference stat wise between a 5 and 6 is so small if they had a problem with it they probably would have had an issue with 4/5.

Quote

Clocks/stars/whatever give a quick and dirty summary of the review. You can't get away from that. Either we drop the clocks, or we accept that and make it as easy as possible for that first perception to be as accurate as possible.

Any other course of action is foolishness.


If the score is causing that big of a problem then lets drop it, there is no score that will not have an initial confusion by someone somewhere, and there will always be publishers that don't like what score they get.
Title: Re: Multiple scores
Post by: 42 on December 19, 2005, 01:20:46 PM
I don't think there should be a change to 5 from 6.

I don't see how 5 is more intuitive than 6.

I wouldn't mind getting rid of the half-clocks.

I think some reviewers have been writing reviews long enough that it is time for them to step-up the quality of their work.

I really despise this site being used as a way to market products for companies. If a company won't send us product because we don't give them glowing reviews--too bad. Reading a review shouldn't sound like a sales pitch--which is how many of our reviews read.
Title: Re: Multiple scores
Post by: Skar on December 19, 2005, 01:30:58 PM
Quote
I still don't see how its harder to grasp a 6 point system over a 5 point system.  The differences between a 5.5-6 and 3.5-4 SHOULDN'T be significant.  They're not, you just are thinking that way,  each .5 is the same percentage.


The difference between a 5.5 and a 6 is far more significant than the others because it's defined in the submission guidelines as something more than perfect.  The reviewers are told to consider a 5.5 a perfect score and 6 something beyond that.  So you give something a "perfect score" and, visually, it doesn't look perfect.  That's a problem.

Another very real problem is what e is talking about when it comes to intuitively being able to interpret our score.  Readers should not have to reference a seperate document in order to correctly interpret our scores.  If you take a new reader and show him 5.5 out of 6 clocks he's not going to think 5.5 is perfect, as our guidelines state.  He's going to think it fell short of perfection. Problem.

If we lose the 6th clock altogether, we lose the ability to signify truly extraordinary works, the "beyond perfect" that we were trying to be able to show with a 6 clock score, as defined in our guidelines.  So, I still think my solution, 5 out of 5 with a 6th clock getting tagged onto the end in the rare cases, is the way to go.  As for all the old scores, not a problem.  All we have to do is put a note in the FAQ about the switch and when it happened so we don't have to change any old scores at all. That seems to address all the concerns except one.

The one it doesn't address is Spriggan's lack of desire to recode the site.  Perhaps we should take a page out of Spriggan's manual for dealing with reviewers and just tell him that if he doesn't feel like doing the work to maintain the site we'll go to someone else.

I'm not seriously suggesting that we boot Spriggan, anymore than I hope he was serious about telling reviewers to go jump.  
Title: Re: Multiple scores
Post by: Fellfrosch on December 19, 2005, 01:32:38 PM
I am of the opinion that any wholesale change to our rating system would ideally involve dropping numbers altogether and substituting adjectives: Poor, Fair, Good, Excellent, Perfect, etc. (though of course we'd have more than that). That would make it obvious what we mean, without the need to quantify artistic expression.
Title: Re: Multiple scores
Post by: Spriggan on December 19, 2005, 01:37:03 PM
I could see that in addition to the 6 clocks, I really don't think it's something that should be dropped unless we get rid of Scores all together.
Title: Re: Multiple scores
Post by: Spriggan on December 19, 2005, 01:38:59 PM
Quote
The one it doesn't address is Spriggan's lack of desire to recode the site.  Perhaps we should take a page out of Spriggan's manual for dealing with reviewers and just tell him that if he doesn't feel like doing the work to maintain the site we'll go to someone else.


Actually you're more the welcome to tell me that, but I have nothing to do with this current site.  You'd have to convince Tage to do so since I'm only involved with twg 4.0.  I'm willing to code any changes into that, but we know that it wont be done soon enough for what these changes would warrant.

Quote
I'm not seriously suggesting that we boot Spriggan, anymore than I hope he was serious about telling reviewers to go jump.


I am serious, you work with a reviewer and if they do a substandard job even after multiple reviews and sessions with the editors we should move on to someone that is better.  What substandard isn't for me to decide but I can only think of one person that fit the bill for this and he doesn't even write for us anymore.  If a reviewer can make the grade or insist on ignoring our site's system then they should be asked to stop writing reviews.  I don't see what's so mean about that.
Title: Re: Multiple scores
Post by: stacer on December 19, 2005, 02:30:27 PM
Except that free reviewers are hard to come by, sometimes.
Title: Re: Multiple scores
Post by: Spriggan on December 19, 2005, 02:38:44 PM
True, but those that really want to do this are willing to improve and work with the people at the site.  What I don't think we need are those that come in and say our methods are stupid and we should use theirs or not care enough to put in any effort.

There have been several people that have submitted VG reviews to us but weren't up to par, they would have like 200 words for a 5-6 clock review and I'd contact them and explain what I liked about the review and what they should add to make them better.  Basically following Fell's guidelines and then giving us a review that was average length.  None of them write for us anymore probably because they realized they had to put some effort into this and we are serious about having good reviews.

No one expects someone's first few reviews to be perfect or even good, be we should expect a reviewer to challenge themselves to do the best job they can.
Title: Re: Multiple scores
Post by: The Holy Saint, Grand High Poobah, Master of Monkeys, Ehlers on December 19, 2005, 03:29:29 PM
Quote
I'm not trying to take that system away, when did I ever state that?  I'm encouraging proactive steps to avoid abuse.  How is what you're suggesting not misuse itself?  You're not partial, you don't think or have the same likes or dislikes as the reviewer.  To me your system is nothing but abusive, all scores will then reflect your and Fell's personal tastes and not anyone else's.

This actually ignores what I've already said. The editor changes the score to match what the text says. The choice is based on the text, not whether or not I liked the product. For example, the case of movies. I've been to the theater exactly 4 times this year. I saw Sky High, Fantastic Four, and Star Wars twice. Thus I *can't* have an opinion on movies.

So no, it's not remotely abuse. It's a correction from the people who have the most experience with the system and deal with it constantly for those people who aren't using that system properly.

Otherwise, I have to ask, why the hell am I an editor? To correct spelling? If that's the case, I resign.
Title: Re: Multiple scores
Post by: Spriggan on December 19, 2005, 03:59:39 PM
Why the hell can't you just ask people before making such a change?  Seriously how hard would it be to ask  someone "do you think your score is adequately reflected in your review?" and then point them to the review criteria.  If you do that then people will LEARN the system instead of relying on you to do it for them.
Title: Re: Multiple scores
Post by: Skar on December 19, 2005, 05:31:42 PM
Quote
True, but those that really want to do this are willing to improve and work with the people at the site.  What I don't think we need are those that come in and say our methods are stupid and we should use theirs or not care enough to put in any effort.

There have been several people that have submitted VG reviews to us but weren't up to par, they would have like 200 words for a 5-6 clock review and I'd contact them and explain what I liked about the review and what they should add to make them better.  Basically following Fell's guidelines and then giving us a review that was average length.  None of them write for us anymore probably because they realized they had to put some effort into this and we are serious about having good reviews.

No one expects someone's first few reviews to be perfect or even good, be we should expect a reviewer to challenge themselves to do the best job they can.


Dude.  Are you really saying that the Gibbs Brothers, those who pointed out the problem with our review scoring system, don't write good reviews or that they haven't put much effort into it?

I disagree.  I think they write great reviews, have put a lot of effort into not only writing them but getting them to us in a timely fashion, and they have sent a couple of good articles above and beyond as well.  Not only that but they've only been able to take advantage of press screenings once or twice.  It's not like they're getting loads of free stuff from us to offset the work they're doing.

And on top of that I think they've got a point.  And I've made my understanding of that point pretty clear so are you talking about me?
Title: Re: Multiple scores
Post by: The Holy Saint, Grand High Poobah, Master of Monkeys, Ehlers on December 19, 2005, 06:09:19 PM
Quote
Why the hell can't you just ask people before making such a change?  Seriously how hard would it be to ask  someone "do you think your score is adequately reflected in your review?" and then point them to the review criteria.  If you do that then people will LEARN the system instead of relying on you to do it for them.

You've done it. You KNOW how difficult it is to do at times. I've waiting *weeks* for people to get back. People aren't interested in doign a second draft. That's fine, I'll make the second draft if they don't get to it. That's my JOB. I should be allowed to do it. I'm not saying it should be done on every review, but when it comes to the content of hte article, the clock score is a little thing in terms of creative control. You're so anal that you would quit a site over that small change? I seriously cannot see what the problem with it being understood that the editor has the power to make a necessary change in an rticle when he can't get hold of the author in a timely fashion. Seriously. I can talk to them after the fact if I have to make the change. But I seriously think that since this sort of control is technically already in the editor's power, it  is not unreasonable for him to be allowed to exercise it.
Title: Re: Multiple scores
Post by: Mad Dr Jeffe on December 19, 2005, 09:36:43 PM
Quote
I am of the opinion that any wholesale change to our rating system would ideally involve dropping numbers altogether and substituting adjectives: Poor, Fair, Good, Excellent, Perfect, etc. (though of course we'd have more than that). That would make it obvious what we mean, without the need to quantify artistic expression.


Now this is something I can get behind. Frankly I'm skeptical about the merits of 5 being more intuitive than 6 ... no offense meant E but without any real data other than you saying 5 is more intuitive, its a bit of a hard sell. Now if you do indeed have a little data supporting it then I'd love to see it. I mean the whole point that all of us are missing is that we all love the site we've had a hand in creating. Because guess what, as writers and editors we all have an emotional stake in TWG. I think we can agree that the site is special to us all and that we want to see it get better and not worse. We're treating several issues like they are just one issue and that seems to be a big hangup right now. The first issue is that some people misunderstand the six clock system. I have to admit that it seems pretty easy to figure out from my point of view. I mean it is just a stepping stone to the review. However some people are of the opinion that its hard to fathom. Ok, I'll buy it, but changing overnight or displaying two seperate review styles isnt going to make it any easier for our readers to get it. The way I've seen it the 6 clock system has been a site trademark for a while mainly because its unique. There are star systems out there (for 1-4 or 1-5) stoplight systems, thumbs up or down systems and so on. I've always seen our 6 system (with half clocks) as a this site goes up to eleven kind of review. Its a bit rebellious and pretentious at the same time... pretty much like us. To me just abandoning it for another system thats frankly just as arbitrary betrays the spirit of the site. Im trying to decide if thats a good or bad thing though...
It could be very good
or it could be very Comicbook Guy.

Im leaning toward the opinion that changing our system from 6 to 5 clocks just looks arbitrary and lame. Having said that, I would definately fall behind a more descriptive and less arbitrary system ala Fell's adjuctive mania system. It avoids pointless bickering over stars as adjectives are more precise and even friendly.

Our second problem though is more insidious.
For a review site, we rarely give bad reviews. Oh sure we give some, but the bulk of our reviews come off as fannish.
This is because as Stacy says we tend to review stuff we like and only pan stuff when it surprises us with how bad it is. Im not entirely sure how to combat that. Some of it is caused by the Game Company sending us material... which is something that has concerned me a bit. I think we should have to disclose donated or review material status. If Wizards gives us a book, we should say it, if we bought it ourselves we should say it. I think that this might make us think about the product a little more when we review it (and it will definately make the readers a little more critical). I feel like I can talk about it because I feel a little conflicted when I review books I bought myself instead of say a book given to me to review from Atlas. I mean if I love Castle Falkenstein (and I do) how objective can I be? I honestly dont know.
As reviewers we dont seem to take criticism well. This has been seen most recently in Movies but it affects all aspects of our forum system. We need to be a little more relaxed and a little less dogmatic when it comes to discussion... yes even me ;)
Title: Re: Multiple scores
Post by: Mad Dr Jeffe on December 19, 2005, 09:45:23 PM
Our third problem right now is stagnation.

Yes Stagnation.

We used to be one of the best Webcomic review sites out there, untill people stopped writing about them. I know some people on the site read more webcomics than almost everyone out there and I dont know why that happend.

In the RPG front, we get fewer and fewer products to review. Im not sure if thats the fault of the editores responsible for getting them, or if game companies have changed their policies. It needs to be fixed if it can.

I think it needs to be said that movies is our most active section.
Kudos to Skar and the Gibbs brothers for doing that work for us. It must be very taxing to  watch all those movies :D
Seriously though they do a great job and I want them to know that this debate about scores isnt a shot at them. We like their reviews a lot.

I think we could go to a more organized release schedule though if we gave assignments to willing writers. Especially ones who have delivered in the past.

We definately should think about stockpiling some reviews for slow weeks in a kind of dead file. (Especially webcomic reviews). I think our regulars could be trusted to dash out x amount of words on a deadline and not only that I think they might really like the challange.

Also maybe we should put a minimum word length on reviews...

Just a thought.


Title: Re: Multiple scores
Post by: Lightning Eater on December 19, 2005, 10:59:19 PM
I agree with e that a six-clock review system isn't as easy to understand, because lots of people are used to systems on 10 because of fact that the marks in school and most competitions are on 10 and reviews on five can easily be converted.

But I also agree with Mad King WencesJeffe that the six clock system is basically a symbol of the site, and I definately agree that it shouldn't be taken away.

My personal opinion is that the clocks should be kept, except with adjectives like Fellfrosch had suggested

Quote
I am of the opinion that any wholesale change to our rating system would ideally involve dropping numbers altogether and substituting adjectives: Poor, Fair, Good, Excellent, Perfect, etc. (though of course we'd have more than that). That would make it obvious what we mean, without the need to quantify artistic expression.


This way you wouldn't have to spend the time changing the code of the site or editing old reviews, and with descriptions of the score people who find the present system complicated would be able to grasp the six-clock system after only reading a few reviews, from seeing the corresponding adjectives to the number of clocks.

It also solves the problem mentioned by e earlier, of companies no longer helping the site because of misuderstandings about their thing getting a 3 1/2 or 4, since it'll stop people from assuming 4 or so is a bad score.

Also, it accomplishes what the recent double rating system was trying to accomplish without all the recent systems drawbacks.
Title: Re: Multiple scores
Post by: Skar on December 19, 2005, 11:33:54 PM
I think we should keep the clocks.

In the end we have to deal with the fact that we don't exist in a vaccuum.  No matter the subject matter, other review engines hand out full marks, whether it be 5 out of 5, 4 out of 4, or 10 out of 10 to films/games/what-have-you, that we, according to the current definition, would not give 6 clocks.  

Out there in the world full marks does not usually equal world-shattering, genre-forging perfection.  It equals a darn fine piece with minor problems, if any.  Now, don't get me wrong, conformity for conformity's sake is not my cup-of-tea.  If we want to hold to the standard and keep our 6 out of 6 system with a blank clock or more hanging off the end of nearly every review, and force the world to decipher our code, I'll play along forever.  I think it may be a bit of a reach though.  

Perhaps we should bow to the wisdom of an outside perspective (the Gibbs brothers are fairly new to the site but long in the tooth when it comes to the movie review world) and admit that there may be a problem with having an empty clock on every review.

My solution, 5 clocks shown, empty or full, on all reviews with the extra half or full clock tacked on for the truly extraordinary pieces, the 11s as it were, keeps the 6 clock system (nearly intact) doesn't really require a change to old reviews since the standard would essentially be the same just the visual representation would change slightly from here on out, and suddenly, voila, run-of-the-mill perfect scores would look perfect and at the same time be in line with the intuitive standard set by most of the other review engines out there.

I'm not advocating getting rid of the 6th clock altogether, just giving it the visual significance it deserves as a special ultra nifty case.

What doesn't make sense about that?  I really want to know.
Title: Re: Multiple scores
Post by: The Jade Knight on December 20, 2005, 03:19:34 AM
My vote:  We keep the 6 clocks as they stand, but we also add adjectives to describe (sum up) what the clock rating stands for.

I'm pretty sure that that would solve most of our problems, surprisingly enough (except, of course, issues like stagnation or some score inflation).
Title: Re: Multiple scores
Post by: Spriggan on December 20, 2005, 03:51:38 AM
Quote


Dude.  Are you really saying that the Gibbs Brothers, those who pointed out the problem with our review scoring system, don't write good reviews or that they haven't put much effort into it?



Nope, not at all.  Their first review, catwoman, wasn't all that good but it was their first one and they've really improved.

Just to get this out of the way so everyone understands this, there has only been one person that has written for this site that was bad, ie horrible writer, didn't take it serious and other things and he DOSEN'T write for us anymore.  It was quite a while ago, and I have no problems with anyone currently writing for us.

I am annoyed at the Gibbs wanting to add their own system on top of ours, they didn't point out a problem they instead just said we like our method better we're going to use that instead.  They should have started a thread like this if they didn't care for our system and discussed it first with all of us before doing the multiple score system.
Title: Re: Multiple scores
Post by: Spriggan on December 20, 2005, 04:13:44 AM
You bring up some good points, and I wanted to address some of them.
Quote
Our third problem right now is stagnation.

Yes Stagnation.

We used to be one of the best Webcomic review sites out there, until people stopped writing about them. I know some people on the site read more webcomics than almost everyone out there and I don't know why that happened.

They got busy, ie real jobs that took time.  We're a volunteer site, Fell doesn't pay us so things only get done with people have the time.  EUOL, SE and Tage all have better things to do then reading through hundreds of comics just to write one review.  I agree we should have more, 42 has tried to get a few done for us which is great, but they do take a lot of time.

Quote

In the RPG front, we get fewer and fewer products to review. Im not sure if thats the fault of the editors responsible for getting them, or if game companies have changed their policies. It needs to be fixed if it can.

It's not our Editors, it's the companies have changed their policy.  I know Fell tries to get stuff as often as he can but some, like wizards, have changed how they send out review materials and we just aren't big enough to get stuff anymore.  With other companies it's just because they're small, they've got few people working on this and when the higher a new PR person things get lost in the change or misplaced.  That happened several times this year just like it did last year, the convention seasons are notorious for this.
Quote


I think we could go to a more organized release schedule though if we gave assignments to willing writers. Especially ones who have delivered in the past.

We've tried this before, and honestly lots of me and SE's current argument about the score changes would be easier to handle if we had such a schedule and people actually turned things in several days before the actual publish date.  But, as I stated in the first reply of this post, we're volunteer and all our Editors realize that it's a lot to ask people to stick to a schedule when we've all got lives outside this site.  Heck with myself I've got 3 jobs (including TWG) and school I'm dealing with and since my third job dosen't pay me money I have to relegate it to the bottom of the priority list.  It's the same way with lots of people, jobs, school, family, they all take more time and priority over TWG.
Quote

We definitely should think about stockpiling some reviews for slow weeks in a kind of dead file. (Especially webcomic reviews). I think our regulars could be trusted to dash out x amount of words on a deadline and not only that I think they might really like the challenge.

We've tried to do this too, but it always turns out one or two people writing 4+ articles a week and then they get burned out and don't write anything for a few weeks.  Seriously with only about 6 regular writers that's a lot to ask people when many already write 1-2 things a week (any many of us haven't had time this winter to even do that).  It's a great idea, but again it's back to the volunteer problem.
Quote

Also maybe we should put a minimum word length on reviews...

I've always thought this too, but have never pushed for it since I know that people do what they can.  My personal criteria is 800 words.
Title: Re: Multiple scores
Post by: Spriggan on December 20, 2005, 04:17:12 AM
Quote

You've done it. You KNOW how difficult it is to do at times. I've waiting *weeks* for people to get back.


I understand this, and you're right I've done it.  But that doesn't mean we still shouldn't, and you know what if you contact someone and a week later still no reply then I have no problem with you changing the score.  I still think there should be some effort on the part of the editors to inform people about how to write better articles.  And honestly, if you ever ask me, jeffe, or the bro Gibbs I know all three of us would reply very quickly so would any of the regular reviewers.  And those are the ones that it would be best to try and help follow the guidelines.
Title: Re: Multiple scores
Post by: The Holy Saint, Grand High Poobah, Master of Monkeys, Ehlers on December 20, 2005, 09:04:23 AM
I'm more than happy to try first. I just want it known that I reserve the right to change if someone doesn't get off their lazy butt and get back to me.

We think in terms of base-10 numbering, Jeffe. It's a major reason why Arabic numbering is so much easier to use than, say, Roman numbering. Percentages are difficult to figure from a base-6 system. Base 5 makes more sense because of it's relation to 10, and, when you do half-clocks, it's base-10 anyway.

In my opinion, sticking with 6 clocks just because it's different is sort of like wearing that offensive T-shirt simply because your mom doesn't want you to. It is, in Jeffe's words about changing, is arbitrary and lame. I have thought of the clocks as a trademark, but not specifically the 6 clock system. And if it doesn't work, well, that's all the more reason to change your identity.


Look, I'm the associate editor. It's my job to get articles. That means frankly, I couldn't give a whit what your emotional investment is when you aren't contributing. Right now, the Gibbs account for about 30-40% of our content over the last two months or so. That's at least 3 times as much as ANY OTHER CONTRIBUTER. Thus, I'm willing to give their impression that the system doesn't work 3 times as much consideration as anyone else's. I want to make it as easy as possible for new people to jump in and get started with us. If we made a lot of money and weren't frantically searching for content every morning then I'd be a bit more demanding and make them learn us. But since we get so little content right now, I bend to THEIR whims.

If the rest of you were giving me that much, I'd be fine with keeping it even thought I *know* it's an awkward system. As it stands, I can't believe we're even considering not fixing it from the ground up.
Title: Re: Multiple scores
Post by: Spriggan on December 20, 2005, 09:18:10 AM
Well I have to say I'm glad the final decision isn't up to you SE since I find it offensive that you'd put more weight on the opinions of the "flavor of the month" then people that have been with the site as long as you or longer.

And keeping the 6 clock system is nothing like the wearing an offensive tee-shirt just to annoy people, that's insulting too (there's nothing offensive about our system), it's like wearing a stylized tee-shirt to show off your personality.
Title: Re: Multiple scores
Post by: The Holy Saint, Grand High Poobah, Master of Monkeys, Ehlers on December 20, 2005, 09:36:37 AM
Only the Gibbs don't show any sign of slowing down unless *we* hinder them.

Look, I know why people don't contribute as much. It's for the same reasons that *I* don't contribute as much. There's too much other stuff going on: work, school, family, whatever. But if we drive them away, we'll have nothing to run.
Title: Re: Multiple scores
Post by: Spriggan on December 20, 2005, 09:42:49 AM
I know you understand, heck we all know how busy this year has been for you.  I don't see how asking them to keep to our scoring system is hindering.

This will be my last post on the topic, I'm leaving for Christmass break and frankly just don't think there's anything else that can be added to the fray, by me at least, I'll just wait for Fell's decision and keep my trap shut.

Title: Re: Multiple scores
Post by: The Holy Saint, Grand High Poobah, Master of Monkeys, Ehlers on December 20, 2005, 09:50:41 AM
Note: I have never advocated switching to a 4 star movie system. But I do think this is symptomatic. That's why we need to revamp it.
Title: Re: Multiple scores
Post by: Fellfrosch on December 20, 2005, 12:02:48 PM
In the Gibbs' defense, they did in fact present many reasons why they didn't like the 6 clock system, and discussed it at length with Skar and I via email before we decided, collectively, to try the double score system and see how it works. We didn't do it on the forum, but we did do it.
Title: Re: Multiple scores
Post by: Skar on June 30, 2006, 05:11:19 PM
I wish to reiterate that all problems would be solved if we would only show the 6th clock when a movie recieved it. ;D