All right, here's another idea. It seems like the problem with our rating system is that some people are rating too high and some people are rating 0. Mostly the latter.
This has a significant effect because TDC currently calculates it by mean, if I'm not mistaken. Examples of votes if they were calculated by median instead:
Good game: (0,0,0,2,4,4,5,5,5,5)
Bad game: (0,0,0,0,1,2,2,3,3,5)
Great game, with idiots voting against: (0,0,0,3,5,5,5,5,5,5)
Good game: Mean = 3 stars, Median = 4 stars
Bad game: Mean = 1.6 stars, Median = 2 stars
Great game: Mean = 3.3, Median = 5 stars
Sooo.. long story short, a good game will have a good rating. A bad game will still have a bad rating. A great game may have a bloated rating, but could be balanced once more people start voting.
Under the present rating system, the good game will look bad, and the great game will still look bad.
The current downloads rating system does work well because nobody's going to look like an idiot by giving the game a 0 star rating, but it seems to be messing up for articles, because you can't force them to comment to rate, but it lets them rate without shame.
Edited by an Administrator.
Disclaimer: Any sarcasm in my posts will not be mentioned as that would ruin the purpose. It is assumed that the reader is intelligent enough to tell the difference between what is sarcasm and what is not.
Mmm, can be a little unfair at times. I like the 5 star rating system. However it feels a little limited as to the rating you can give. Kinda like you can't fully express your overall feelings towards the game. 10 stars would take up too much room but, if it went up in halves then it would give us more stars to play with.
I don't really think we need more. Reviews let you give more detail if that's what you want. Lol, nobody uses reviews anymore.
Disclaimer: Any sarcasm in my posts will not be mentioned as that would ruin the purpose. It is assumed that the reader is intelligent enough to tell the difference between what is sarcasm and what is not.
The problem with these user ratings is that, there is no specific value to the stars. If you are trying out a game, and it is quite a good game, but not a super game. You want to give it 3 stars, but to avoid misunderstanding, you give it 5. The point is; the game is good, but not super. So if there could be giving more specific values to the stars, f.ex. 5 stars = EPIC (means good), 4 stars = SUPER(means good), 3 stars = GOOD(still good), 2 stars = AVARAGE (the game is ok) and 1 star = NEEDS ALOT OF IMPROVEMENTS. (bad or not quite finished)
Lol, I don't think that's the real problem. People pretty much know that a really good game is 5, an OK one is 3, and anything less than that 0-2 or not deserving of a vote. The problem is when some people start voting 0 to be mean and others start voting 5 to cover up for the 0s.
Disclaimer: Any sarcasm in my posts will not be mentioned as that would ruin the purpose. It is assumed that the reader is intelligent enough to tell the difference between what is sarcasm and what is not.
how about only letting people see the game's overall rating after you've voted. That would go some way to stopping people voting 0 just to bring the average down or 5 just to raise it.
Of course that only works if they're people that see the score and think the average is higher or lower than what they think it should be.
That would kind of mess up the whole idea behind the ratings. If you can't see the rating (or sort for that matter) for a game before downloading it and rating it, many people won't bother downloading many of the uploaded games I believe. It would be hard for visitors to find those old gems if they weren't allowed to see the ratings.
Ok, sorry for my messed up grammar and such, but it's 5:51 am over here and I'm tired.
I don't like the way you presented your examples, Muz. I'll just simplify them (so I can do my maths better).
Good game: (0,0,0,2,4,4,5,5,5,5)
Bad game: (0,0,0,0,1,2,2,3,3,5)
Good game, w/ odd voting: (0,0,0,3,5,5,5,5,5,5)
What if the rating system used variance? Using the program I wrote a few years ago for my calculator, I think it'd be a more accurate system. Something to note is that my program outputs two numbers, and I'm not really sure what they mean, so I'll just report both of them. If anyone with a better sense of math can run the numbers and get the same results, then please tell me what the parenthesed numbers are. Anyways:
The good game would have a rating of 4.61 (or ~2.1471). The bad game would have a rating of ~2.8547 (or ~1.6896). The "oddly voted" game would have a rating of ~5.1790 (or ~2.2757).
Consistent voting patterns are definitely better for variance, it seems.
Median is a bad and often misleading statistic. Mean is fine I think for now, almost every other website in the world with ratings uses mean.
I don't think you can really use outliers - as well as being complicated to do, outliers only really apply to large amounts of data with one or two values which are very far away from the rest. One rating of 1 for a game with an average around 8 wouldn't alter it that much, it's only when large amounts of people vote 0 for a game, in which case those votes cease to be outliers.
I think making halves available would be better, to differentiate between goodness and greatness.
You should not be able to rate something 0, because that is the same as not rating the game at all. The lowest possible score that one should be able to give is a 1. That way when someone gives it a bad rating, perhaps on purpose, the overall rating will not drop as bad.
(0,0,0,2,4,4,5,5,5,5) Average= 3 star rating
(1,1,1,2,4,4,5,5,5,5) Average= 3.3 star rating
Now the difference may not seem like that much here, but on a larger scale it would make a big difference.
The main problem is that we will always have idiots who rate too high or too low just for the heck of it. What we really need here is a "is not an idiot" status that allows you to vote and stuff.
I don't know if it's all that important to make sure people vote properly. If your game gets one 0 rating from someone you can ignore it. If you have 100% 0 ratings then maybe you should start to take notice.
I mean how many people play a game extensively before making their rating? Maybe the problem is people care too much about what score their game gets?
Originally Posted by Hayo The main problem is that we will always have idiots who rate too high or too low just for the heck of it. What we really need here is a "is not an idiot" status that allows you to vote and stuff.
Lol, that I fully agree with. The only problem with that is that some people won't like letting only certain people giving them right to vote. It's actually a good idea, just hard to implement.
Worth1000 automatically allocates idiot and non-idiot status to members (using different terms) and it works great for them. Of course, Worth1000 is almost entirely dependent on ratings, so they worry less about a non-complicated system. I think we did consider something like that before the star rating system was applied to the downloads,
Adam Phant:
Yeah, I'll rewrite the examples. Variance is well.. not really the main point. Ratings especially for a simple site like TDC, are supposed to show how good the game are, not how good the rating system is
Boothman:
Mean is good for most cases, but it's horrible when you have highly biased votes. Which seems to be the big problem with TDC's votes. The main purpose for medians is for highly variant or unreliable data. There's also another one that's somewhere in between, but I forgot the name.
Yami:
It's been said for a long time that the 0 stars shouldn't be there, but many admins don't seem to agree, so that's not really a solution.
Disclaimer: Any sarcasm in my posts will not be mentioned as that would ruin the purpose. It is assumed that the reader is intelligent enough to tell the difference between what is sarcasm and what is not.
What about taking the mean of the mean and median? The good game would get 3.5 stars, the bad game 1.8 stars, the great game with idiot voting 4.15 stars. Sounds simpler than variance!
As weird as the idea sounds, it seems to work! Anyone opposed to it?
Disclaimer: Any sarcasm in my posts will not be mentioned as that would ruin the purpose. It is assumed that the reader is intelligent enough to tell the difference between what is sarcasm and what is not.
There's nothing wrong with mean. In fact there's nothing wrong with the rating system at all. If people are voting down as you say then that's their prerogative - maybe like Andy said people shouldn't take notice of ratings and instead take notice of the comments people put.
Yeah, I say that because most people play a game for a few minutes then delete it and make a small one line comment about it. You probably shouldn't take too much notice of it.
I think the ratings aren't working properly anyhow. For 'FIG' it says on the game's profile that it is "4th Place" but when you sort the downloads by rating it is coming "2nd".
Originally Posted by -Liam/Slink- To be honest, I forgot we could review...
It's because it doesn't list the reviews you've made in your profile.
I think a good idea except just showing the average, is also to show how many have voted different numbers. Like the general poll.
Because, there can only be one. That's silly.
Originally Posted by Don Luciano I think a good idea except just showing the average, is also to show how many have voted different numbers. Like the general poll.
Because, there can only be one. That's silly.
That's not really a bad idea, except that it's confusing to place 5-6 different ratings on the downloads page. But I think on the game page itself, it's not a bad idea.
Disclaimer: Any sarcasm in my posts will not be mentioned as that would ruin the purpose. It is assumed that the reader is intelligent enough to tell the difference between what is sarcasm and what is not.
How about adding a button that says "This game has an unfair rating". Clicking it won't do anything, it's just a way for people to vent their frustration
- Ok, you must admit that was the most creative cussing this site have ever seen -
Originally Posted by Phredreeke How about adding a button that says "This game has an unfair rating". Clicking it won't do anything, it's just a way for people to vent their frustration
Nah, there are too many buttons that do that already.
Disclaimer: Any sarcasm in my posts will not be mentioned as that would ruin the purpose. It is assumed that the reader is intelligent enough to tell the difference between what is sarcasm and what is not.
I see this is a huge problem. I hate it when people see a game getting 10/10 reviews and they think 'HMMM it's a good game but not worth 10. I think it deserves an 8 but I'm going to vote 0 to make my vote count more than everybody elses!!!!1' I think a median system would be good, though there would still be problems. How about a super simple game where some people think 'eww it's really basic 1 star' then others go 'No it's SOOO fun!!!' and give it 5? I can imagine the rating going between 1 and 5 randomly.
If you use the median, surely it would simplify the games ratings completely, eliminating any shades in between?
I know this is a big problem with games. I'm inclined to make the rating system more complex, possibly voting for individual aspects such as audio, technical merit and stuff. At the moment an epic 2 year project can score the same as a 5 minute platformer simply because people can't really be bothered to get into the former. I don't know... just my suggestion, as some of the games on this site are so extreme in quality (both ways) I find it unfair that the same rating system is used.
Don't aim for perfection- you'll miss the deadline
'~Tom~ says (16:41):
well why does the custom controls for the keyboard palyer even affect the menu controls at all whats thep oint jsutm ake it so for the keyboard palyer on the menu screens everything is always up down left right enter regardless of the controls they set'
I think the real problem is actually that the quicker and easier you make it to rate a game the less thought will go behind the vote.
Reviews are 1500 words, so you'll most likely play a bit of the game to be able to actually talk about it. But with ratings you can die after 1 second and in a rage vote 0 stars or thumbs down or 0/10 because 'the game is too hard'.
So yeah, It's not the system thats the problem its the people.
Reposted to a thread that might not actually die over it's own weight in spam.
What if you have 4 stars.
4 = Great
3 = Good
2 = Could be better
1 = Bad
And in order to give a 1, you're required to submit a reason why, and you can only do 1 bad rating a day.
1 can have thumbs up or thumbs down. If the ratio of thumbs down beats the ratio of thumbs up after 3 ratings, then that rating is automatically revoked, because obviously a majority found it inappropriate.
2 can have a similar thumbs up or thumbs down, and if the ratio of thumbs down beats the ratio of thumbs up after 5 or 8 ratings, then the person who posted it can choose to either revoke or change it. If the person doesn't respond in a week, the rating is automatically rounded up to a 3.
Sounds like it may take a tad extra work to implement, but it seems solid for a small community like this. It's just a matter of putting it into action and seeing how it fairs.
muz, please understand this. The only problem with the ratings system is the small sample size. The more popular a game, the more accurate its score. Obviously games with the minimum amount of votes shouldn't outrank those with 3000 downloads as the "Top Rated", and the downvoted games are just those who don't have enough ratings for it to even out.
Hmm.. that's not a bad idea. There could be like a calculation for (standard deviation/squareroot(sample size)) as a "rating inaccuracy" value.
Thus, a game with stddev of 3 and 3 votes would be 1.73 inaccurate, but a game with std dev of 3 and 30 votes would be 0.548 inaccurate. That's brilliant! Central limit theorem FTW!
It also provides a good way to consider the top 10 downloads, rather than just picking the ones with the highest average.
I also like the idea of making reviews count for more. It'll encourage more reviews, which is, at its worst, much better than a person saying "tis game rocks!" and rating 5 stars.
Hmm.. the problem with that method is that it still doesn't get rid of outliers, in the case of overhyped games like Knytt Stories and Eternal Daughter. Games like ED are bound to have a lot of people downvoting or upvoting them. Looks like I'm one of the people who downvoted ED
Disclaimer: Any sarcasm in my posts will not be mentioned as that would ruin the purpose. It is assumed that the reader is intelligent enough to tell the difference between what is sarcasm and what is not.
Outliers could still affect the game score by writing an illusively genuine review, giving it a bad score and laughing along the way.
The only way to get rid of outliers, ney, dedicated outliers is to delete the entire voting system. Then you get game outliers who post chains of crap games, and you fixed that by regulating the submissions. You can't do that for votes. Or can you?
Your voting is voted on. Your voting quality determines the weight you can add to the rating.
Or, depending on your voting authority, you get a different voting weight. If your vote is contrary to popular opinion, your vote weight goes down, but alternatively, popular opinion sends it up. As you spend time inactive on TDC, your voting weight tends toward a normal voter. This system works as long as there are more clickers than outliers.
or
I think I read it earlier in the forum, but give everybody a certain allowance for votes. The more they submit games/articles/previews and talk of the forum, the more voting points they earn. Or, give them 2 a day or something.
Advertisement
Post Reply
Edited by an Administrator.
