Brewing a Better Rating System
(via yongfook)
A very cool rating system. I’m a fan. And it’s a tea related site too. Bonus!
Hey there tea drinkers, we just rolled out some updates to Steepster! For one, you can now specify things like steep times and temperatures in your tasting notes. Another highly requested change is a finer rating system, something we’ve been trying to figure out for a while. We think we’ve come up with a pretty neat solution and wanted to walk through some of the decisions we made.
The Old System
When we first launched Steepster, we weren’t totally sure how people were going to use it. We knew how we wanted to use it, but it’s the little details like…
- “How long would the typical post be?”
- “Would they be more or less review-oriented?
- “Longer and blog-like? Shorter and twitter-like?”
We wanted to wait until we got a better sense of the direction the site before committing to anything too specific. That’s why we erred on the side of quicker, simpler interactions — thumbs up if you liked the tea, thumbs down if you didn’t — thinking that it’d be easier to go narrow as time went on (versus the other way around).
This worked okay for a while, but when we ran the numbers, we saw that ninety-some percent of the teas were getting positive ratings. Now we all know that tea in general is pretty darn good ;) but we still needed to separate the really good from the sorta, kinda good. So we looked at some other options:
5-Star Ratings
They’re easy to understand and they’re everywhere. You get what they are immediately and they’re good for comparing good things to bad things. However, a 5-point scale starts to break down when you’re rating a lot of similar products that are all pretty decent.
It turns out that the average rating for products on sites with 5-star scales is around 4.3. To us, this says that we need to dive deeper — zoom in to a level where it’s clear what the difference is between a really great tea and the best tea you’ve ever had.
100-Point Ratings
I bet there are quite a few out there who can rank every tea they’ve tasted in order of preference the way Rob Gordon knows his record collection in High Fidelity. We noticed that some of you already use such scales on your own websites and blogs. At the same time, 100-point scales can feel daunting to those that are just getting to know tea. Steepster’s about connecting both tea fanatics and new tea drinkers, so we didn’t want to alienate the beginners. We were all newbies at some point :)
Alternative Rating Systems
One idea we had early on was to use smiley faces instead of stars. This way, it was clearer what each rating meant:
We really liked this approach, but realized it was still just a four-point rating system and didn’t quite have the range needed.
Letting it Slide
In the end, we tried to capture the benefits of all the different systems: the granularity of the 100-point scale, the ease of the star rating system, and the built-in meaning of the smilies. Here’s what we came up with:
Who doesn’t like sliders? Behind that slider is a full 100-point scale. Still one problem though: when you write a tasting note for a new tea, you don’t want to have to go back and look up what you rated your other teas. That’s why we added something else:
See those tick marks? We’re taking a few of your most recently rated teas and using them as markers on the slider, giving you a frame of reference for the new tea. They show up when you drag the slider over them or when you hover over the tick marks with your mouse cursor:
We hope this will not only answer the question just how good? but also the question compared to what?
Here’s the full, updated form that pops up when you go to write a new tasting note:
Transitioning Old Ratings
You’ll also see a new ‘Teas’ tab on your tealog page where you can quickly update all the teas you’ve rated to the new scale. Click on a rating and you get a compact version of the slider:
We’ve converted the old thumbs down ratings to 25s and thumbs up ratings to 75s to help you get started, so please go in and update the ratings for teas you’ve logged. It’ll help us keep the overall ratings accurate. Thanks!
As always, there’s still a ton of room for improvement, but we feel like this is pretty big step forward. So what do you think? Thumbs up or down?
