rating system mechanics suggestion

Have a good idea? Let us know
User avatar
Yatchen
Posts: 1852
Joined: Sat May 30, 2009 11:43 am
Location: ひみつ

Re: rating system mechanics suggestion

Post by Yatchen » Mon Jun 29, 2009 9:07 am

But when you abstain from the vote it means you a) don't care enought o vote or b) you don't love the image. So it basically is a neutral vote not to vote. I'm personally not about to waste my time voting neutral when my non-voting is neutral enough for me.

What if you could vote something up from a 1-3 (just an example) range but vote it down by one?

I'm not talking about just a random image; I'm just talking about abusing the rating system to have your favorite images at the top of the "top images" section, which I know happens. And there aren't enough people who even rate images on a regular basis to have an accurate judge anyway.
When I talk about an image going from 9 to 2, I'm assuming that the image deserves the 9 in the first place.
私は悪魔です。

User avatar
Merun
Posts: 2897
Joined: Sat Mar 17, 2007 9:44 am
Location: NoWheRe

Re: rating system mechanics suggestion

Post by Merun » Mon Jun 29, 2009 9:16 am

Normally, the more rating there is, the more accurate the rating is. The problem is those with extreme rating which is in fact, not the same scale as most user so it's in the end, not accurate.
I also don't like the idea of an up/down vote on images because opinions on art aren't nearly that cut and dry. Under an up/down, a vote for an image you kinda like counts the same for an image you absolutely love, which is also the same for one you dislike or one you absolutely hate.
This is exactly what is happening with some user right now.

As for those who rate latter, it's possible that they have the bias since there is already a rating on the image. You can also add those with second account to rate. Those who rate only their own picture 10, and the rest 1.

Otherwise I suggest a system with only 6 rating, 3 negatives ( -1, -2, -3 ), 3 positives(+1, +2, +3) ( not neutral as more people would pick neutral so as to go along with the folk ( aka, neutrality bias ) ). Also, they have a limit of point rating ( 10, because I don't like multiple of 3 :D ) they can make in a period of time ( to be determined ) so that their vote will be a more thoughtful one. Though I feel this is somewhat a evil system ^^;
Merun @ Twitter

User avatar
Yatchen
Posts: 1852
Joined: Sat May 30, 2009 11:43 am
Location: ひみつ

Re: rating system mechanics suggestion

Post by Yatchen » Mon Jun 29, 2009 9:57 am

Merun wrote:Those who rate only their own picture 10, and the rest 1.
Those who rate only their own picture +3 and the rest -3.

The same issue is still there.
Unless you'd want to make it so people can't rate their own images, but I don't think that would be very fair or necessary.
私は悪魔です。

User avatar
wrexness
Posts: 1144
Joined: Thu Sep 20, 2007 11:09 pm
Location: Flyover Country

Re: rating system mechanics suggestion

Post by wrexness » Mon Jun 29, 2009 10:34 am

Yatchen wrote:But when you abstain from the vote it means you a) don't care enought o vote or b) you don't love the image. So it basically is a neutral vote not to vote. I'm personally not about to waste my time voting neutral when my non-voting is neutral enough for me.
Well we're getting into degrees here. A vote of 5 out of 10, or 3 out of 5, whatever is the middle point, is different from not voting at all. If you love an image, one you'd normally rate a 10, or if you just like an image, rating it just 6-7, those two would have equal weight of +1 in an up/down system. It's not a matter of not caring, it's a matter of not viewing the image in an extreme light.

If you look at it in terms of straight averages, you have an image that already has two votes: a 1 and a 10. If you can insert a 5 into there, it insulates the rating a little from more extreme high or low votes, whereas a non-vote in a straight up/down system doesn't have the middle ground protection.
Yatchen wrote:What if you could vote something up from a 1-3 (just an example) range but vote it down by one?

I'm not talking about just a random image; I'm just talking about abusing the rating system to have your favorite images at the top of the "top images" section, which I know happens. And there aren't enough people who even rate images on a regular basis to have an accurate judge anyway.
But even that can be 'abused'. There's no such thing as a perfect system - only a system that hasn't been exploited yet.

And not enough people voting is part of the problem, but that's why this discussion was created in the first place. We fairly routinely get about 20 pages of images a day, and at 15 images a page that's an average of 300 new images that a person could vote on in a given day. Even if you take down the reposts and disableds, I'm sure that it'd still be in the 250+ range on a normal day. I can only go by my own feelings on the matter, but I think a significant part of the low voter turnout is the fact that voting on an image is a very cumbersome process.
Yatchen wrote:When I talk about an image going from 9 to 2, I'm assuming that the image deserves the 9 in the first place.
Which, unfortunately, is an assumption that -cannot- be made. Who's to say that you, who gives an image a 9, is right while someone who votes that same image a 2 is wrong? As I think Merun is alluding to, maybe they legitimately think that the images in the top ratings aren't as good as their favorites. Are they wrong? Is it bad of them to vote down an image they don't like as much? I honestly think no, because art cannot be judged objectively.

Maybe that person would vote the image down without any other factors (e.g. the existence of an image they like better), maybe not, but we can't assume that it's a problem because people are subjective creatures and are prone to introduce their own biases into their thinking. Is it abuse? Or is it just using the system as intended?

I don't think a single person voting an image down is a bad thing. Now if this person were to enlist alt accounts and spam votes an image, that's a different story. But one person-one rating, I think is fair, even if it does cause dramatic swings. However, this becomes much less of an issue if you make the system easier to use and make it so that people want - or at least are willing - to vote for a lot of images even if they don't absolutely love or absolutely hate it. Doing that will give you a more accurate representation of the opinions of the people on Shuu and protect against these alleged 'abuses'.
Merun wrote:Normally, the more rating there is, the more accurate the rating is. The problem is those with extreme rating which is in fact, not the same scale as most user so it's in the end, not accurate.
Well, what is the scale for 'most' users? Do most people for 1,5,10 for images? Do most people only vote for images they really like, giving them ratings of 7-10 (myself included)? There is no set standard, especially when you consider the 10-point scale range, which I think is too wide and an issue, especially since it's too easy to compare it to the standard school grading scale in the states with 7 being pretty average, and anything much lower than that is bad, which I'm pretty sure isn't the intent. I think that would be at least be partially alleviated by narrowing the scale.

User avatar
Yatchen
Posts: 1852
Joined: Sat May 30, 2009 11:43 am
Location: ひみつ

Re: rating system mechanics suggestion

Post by Yatchen » Mon Jun 29, 2009 10:47 am

wrexness wrote:If you can insert a 5 into there, it insulates the rating a little from more extreme high or low votes, whereas a non-vote in a straight up/down system doesn't have the middle ground protection.
Which is why I didn't think there necessarily needed to be downvoting in the first place.

I'm not saying that a person can't vote down an image from the top section--I do it sometimes, but I don't vote them down by giving them a rating of 1 just because I want them out of the top. I'd give them the rating I actually think they deserve, even if that's something higher like an 8.

I also think at least narrowing the scale would be a good thing. I only ever vote in the 6-10 range, with 6s only for the images I really don't like (that I've bothered to at least look at, that is). The other ratings seem really harsh to give to an image--just like as you said about the school grading system.

A part of what you're saying about the large amount of images there are to rate assumes that people actually open all the images to rate them in the first place. I open probably ~20 images a day, depending on the artists/sources, and some of those I open only because they seem ugly/weird from the thumbnail. I would generally look at those, then close them without bothering to rate them at all, whereas the other ~15 I look at I would consider rating.
私は悪魔です。

User avatar
wrexness
Posts: 1144
Joined: Thu Sep 20, 2007 11:09 pm
Location: Flyover Country

Re: rating system mechanics suggestion

Post by wrexness » Mon Jun 29, 2009 11:04 am

Yatchen wrote:I'm not saying that a person can't vote down an image from the top section--I do it sometimes, but I don't vote them down by giving them a rating of 1 just because I want them out of the top. I'd give them the rating I actually think they deserve, even if that's something higher like an 8.
That may be the way you do that, but is it any more right than the other person who does down-vote it? There's no set standard, scale or guidelines to follow. In the absence of those, what these 1 point down-voters isn't wrong; it's a difference of opinion. It's all a matter of what you think is right and fair versus the other person's opinion.

But if you get an easier voting system that motivates people to vote more, that one or two people who decide to down-vote becomes less of a problem. It's easier to have an impact on a vote when you're 1 of 4. However, when it's 1 vote out of 50, 100, 200, it's an insignificant outlier.
Yatchen wrote:A part of what you're saying about the large amount of images there are to rate assumes that people actually open all the images to rate them in the first place. I open probably ~20 images a day, depending on the artists/sources, and some of those I open only because they seem ugly/weird from the thumbnail. I would generally look at those, then close them without bothering to rate them at all, whereas the other ~15 I look at I would consider rating.
Actually, no I'm not. :P I do most of my rating by the thumbnails. I'm not a big art person, so details aren't nearly as important to me than the larger, overall image. I'm more of a forest person than a trees person, so to speak. (Especially these 4000x3000 monster images. @_@ ) But even if I rate every single one of those images posted daily without opening the full version of a single one, it'd still take a -long- time to vote for all of them because it's such a long process to make a single vote.

That in and of itself doesn't lend itself to lots of people taking the time to vote on many images. If I go by rough estimates, at 2 seconds to place a vote at 250 new images to vote on a day, it'd take nearly 9 minutes just on voting ALONE (not including viewing the thumbnail and potentially the full-size image, as well as just navigation), which doesn't sound like a long time saying outloud, but keep in mind we're talking about an internet where waiting more than a couple seconds for something to load is unbearably painful.

Count me on board for a single-click voting option with redo/reset capabilities on a small (1-5?) scale. Making a vote quick and easy instead of tedious and cumbersome will help (though not guarantee by any means) more people be willing to vote on a wider variety of images.

User avatar
Merun
Posts: 2897
Joined: Sat Mar 17, 2007 9:44 am
Location: NoWheRe

Re: rating system mechanics suggestion

Post by Merun » Mon Jun 29, 2009 3:12 pm

Reducing the scale would make it better indeed.

Another problem is the number of vote per image. Right now, the best image on the board has a lot less rating than very old favorite. As I said, more vote = more accurate rating, so it's not fair in my opinion for the older image. I think that this could be worked out with some kind of weekly ranking, just to affinate quickly the rating of low voted image. To get into such place, the image need to get it's 4th vote then it will be on the page for one week, before going to an all time ranking.

To make this thing going ( well, whatever the solution we pick), I think that we should engage the mod and tagging team in rating every image they see everyday. Showing the example is the first step into getting people into the vote.
Merun @ Twitter

User avatar
Fuwari
Posts: 538
Joined: Sun Feb 26, 2006 6:20 pm

Re: rating system mechanics suggestion

Post by Fuwari » Mon Jun 29, 2009 7:10 pm

wrexness wrote:Actually, no I'm not. :P I do most of my rating by the thumbnails. I'm not a big art person, so details aren't nearly as important to me than the larger, overall image. I'm more of a forest person than a trees person, so to speak. (Especially these 4000x3000 monster images. @_@ )
If you had two images whose thumbnails were, for all intents and purposes, the same; would that mean they would get the same rating from you? What if one had quite a bit of artifacts that aren't visible on the thumbnail? Admittedly, if I were to vote on all or even half of the images that are posted every day, I wouldn't open them all... but I still find this a bit of a sticking point.

Otherwise, I'll agree with most of what else you mentioned. Single-click, changeable ratings, from 1-5 sound good. Changeable because one's opinions can, in fact, change (and usually do). A neutral value (in this case, 3) is important because no vote != neutral. Especially if we mods and taggers had to vote on every image, like Merun suggests. Keep it as simple as possible.

User avatar
wrexness
Posts: 1144
Joined: Thu Sep 20, 2007 11:09 pm
Location: Flyover Country

Re: rating system mechanics suggestion

Post by wrexness » Mon Jun 29, 2009 9:12 pm

Fuwari wrote:
wrexness wrote:Actually, no I'm not. :P I do most of my rating by the thumbnails. I'm not a big art person, so details aren't nearly as important to me than the larger, overall image. I'm more of a forest person than a trees person, so to speak. (Especially these 4000x3000 monster images. @_@ )
If you had two images whose thumbnails were, for all intents and purposes, the same; would that mean they would get the same rating from you? What if one had quite a bit of artifacts that aren't visible on the thumbnail? Admittedly, if I were to vote on all or even half of the images that are posted every day, I wouldn't open them all... but I still find this a bit of a sticking point.
http://e-shuushuu.net/image/174217/

^I apparently can't see artifacts even when expanding to full size. I opened both images in tabs and literally flipped between the two, both at the exact same position. There was -zero- difference to my eyes. So yes, yes they would get the same rating. XD

crimson
Posts: 4
Joined: Sun Jun 28, 2009 2:14 am

Re: rating system mechanics suggestion

Post by crimson » Tue Jun 30, 2009 3:14 am

current system encourages thumbnail based rating/favoriting: all controls that are needed for that actions are around thumbnails.

i think "favorites" are something very similar to "up/down". just with a "down" value by default and extremely high threshold of expressing an "up" :P

User avatar
Merun
Posts: 2897
Joined: Sat Mar 17, 2007 9:44 am
Location: NoWheRe

Re: rating system mechanics suggestion

Post by Merun » Tue Jun 30, 2009 2:44 pm

Favorite is more of a quick way to find images that one like particularly. The top favoured images is a bit, dead, not much movement.
Merun @ Twitter

User avatar
A.Dantes
Posts: 172
Joined: Wed Nov 14, 2007 11:13 pm

Re: rating system mechanics suggestion

Post by A.Dantes » Wed Jul 01, 2009 2:44 pm

My vote is for a simple Up/Neutral/Down system with the ratings themselves still displayed on our current 10 point scale, after Bayesian weighting has been applied by only taking into account the images with at least one vote (Nearly 2/3 of the images on the board have no ratings at all right now).

* The Bayesian weighting should help address how images with five +1 votes should be displayed relative to one with fifty +1 votes. It doesn't work too well right now primarily because of the extremely low ratings to images ratio.
* The inclusion of a neutral probably won't affect the scoring of the image too much, but it does allow for features like "Search for all images posted in the last four days I haven't voted on" in the future.
* The limited voting choices are sufficiently limited that they can all be displayed at once. Only a single click per image required to vote.

I don't think any measures are needed to try to prevent people from gaming the system, especially if those measures make it less likely for people to actually vote. The people that will try to do this are definitely in the minority. One person when the image only has five votes has a much greater weight than three people when the image has twenty votes.

I'm also in favor of showing most popular images over small time frames (day or month).

User avatar
Valea
Posts: 136
Joined: Mon Mar 19, 2007 2:20 pm
Location: Texas
Contact:

Re: rating system mechanics suggestion

Post by Valea » Sun Jul 12, 2009 1:42 pm

↑ What he said.
Image

Post Reply