A Star-Studded Rating System – Behind the Scenes at Digital-Storytime.com

| June 7, 2011 | 0 Comments
Digital Storytime Star

Digital-Storytime 5-Star Award Overall

Reviewing apps might seem like an easy job. You like it or you hate it and then you write a few sentences to explain yourself, right? This is the core of the process, certainly, but when we began Digital-Storytime.com our goals were a bit more lofty. I hoped we could create a site where older reviews were still accessible by having a variety of different qualities that we rated from 0-5.

Rating systems vary a lot. The ratings at Digital-Storytime.com represent only my opinions and are not scientific. They do have meaning, however. My approach to evaluating a digital book is guided by each of the rating categories and then any additional information or explanation can be found in the written review. The star system also includes 1/4, 1/2 & 3/4 stars, but nothing else in between. All of our ratings are on a ‘0-5’ Scale with ‘0’ representing the low end of the scale and ‘5’ representing the top end.

If you are familiar with our site, you may have noticed that very few books score below 3 on most of these ratings. This does not mean there are not book apps available that I would rate lower; there are many awful book apps that I wouldn’t even consider worthy of our free deals in fact. The reality is that a single person can’t review the whole market for book apps, so I need to be picky in order to review the books worthy of a download in my eyes. Unless a book is prominent in the charts, I do not review it if I can’t give it at least a 3. And if a developer has requested the review themselves, I give them the option to wait for a future update to re-submit the app rather than have a very negative review published.

To make my rating system a bit more transparent, I have written up brief descriptions of each of our ‘star’ qualities. I have also tried to include a bit of insight into how I decide how many stars to give each book app that I review on our site.

Digital-Storytime Overall Rating:

This is my ‘editorial’ rating, an all-inclusive impression I have of the whole book, based on technological issues, story quality, developmental appropriateness, integration of the story and other elements – if present (like narration, animation, interactivity, extras, etc.). It is NOT directly related to the other categories, so it’s not an ‘added up’ version of my other ‘star’ ratings for a book app. Since I have spent much of my life as a student, teacher or staff member in schools, I often think of it as a grade with a ‘5’ being the equivalent of an “A”.

Often my overall rating may seem completely in sync with the rest of the rating categories, but once in awhile it may seem to tell a different story. This happens for a number of reasons that can be gleaned from the written review. For instance, a lower overall rating for an otherwise highly rated book in the other categories means something else is not quite right about the app, possibly something technological. For apps with high overall ratings but lower ratings in the other categories, there is something special about this book app, possibly something about the story, the developer’s backstory, or the app’s unique ability to fill a need not otherwise filled by other book apps.

Digital-Storytime’s Other ‘Star’ Categories:

Animation – This refers more to the quality than the quantity of the animation, although stories with very little animation will rate slightly lower on this scale to allow them to sort properly. Truly low scores (below 3) usually indicate something is wrong with the animation, like it’s choppy without enough frames. If a book is ‘lightly animated’ but has the perfect amount of animation to compliment the story, like in bedtime books, the app will often get a 3 in this category to indicate that animation is present but not especially remarkable. Only really amazing animation with ‘wow factor’ will rate above a 4 in this area.

Audio – This refers to all the sound associated with the app and the rating is based on its overall quality … is it easy to listen to, is narration (if present) clear and well articulated in English, and does the audio integrate well into the book without interfering with other elements (for example, sound effects or music that interfere with the narration). This is the only quality rating that we no longer use in our sort and at some point it may be phased out of our rating system in favor of having this information only in the written review.

Bedtime – This rating is one of my favorites and a unique feature of our site. It refers to how well the book app rates as a ‘bedtime story’. This rating category tends to move in inverse correlation with the categories of “interactivity” “animation” & “extras” since these elements are distracting or too stimulating at bedtime (and even if the parent could ignore them or turn them off, children tend to beg for these elements and use them as leverage to extend bedtime). Low ratings in this category indicate that the book isn’t suitable for bedtime, not that the book isn’t enjoyable at other times. Books that are scary or violent in any way will also rate low in this category.

Educational – This rating refers to the educational value of a book app. While most books that a child can read to themselves will get decent marks in this area, a book app with high marks has features or content that go beyond reading. The highest scores in this category go only to books with deeper ‘teachable’ messages or features like narration that highlights as it is read, discussion questions following the story or other educational enhancements. Low marks (below 3) usually mean something is seriously wrong with the book that actually works against educational goals, like a story that has grammatical or translation errors, interactivity that overshadows the story, or the story itself has mixed messages or a poorly developed storyline.

Extras – This refers to all the ‘extra’ elements that are often bundled with ebook apps. Sometimes there are games, puzzles, coloring pages or other content added to a storybook. High ratings in this category mean the extras are well made and engaging. Low ratings usually mean the extras are poorly developed, not engaging or developmentally inappropriate for the book’s target age. I don’t really ‘rate’ the quantity of extras, although if a book has something very small added it will usually get no more than a 3, even if that extra is well done. The review should include a lot more details about this category, when present.

Interactivity – This category refers to all the elements of a book app that invite the reader to be involved with the story, usually tapping to elicit sound effects or animation. Higher ratings in this category mean the interactivity is plentiful and well done in a way that enhances the story or is otherwise educational or uniquely enjoyable. Low ratings (below 3) usually mean this element isn’t well integrated into the story or needs technological fixes. This category is similar to animation, so books that have a tiny bit of interactivity (even if it compliments the story well) will usually get no more than a 3 to allow more interactive books to sort higher when that is the quality a reader is seeking in our database.

Originality – This refers to the overall originality of the story, art, concept, design and execution of the app. High marks for this category mean an app really stands out as new or different in some way. A book will often get marked down for using stories or art in the public domain, although if used in a truly unique or original way it won’t get marked down too much. Any book with both original hand-drawn art and a well-written original story will get a boost in this category. This category is often a ‘moving target’ that represents my best evaluation of how original a book is compared to the rest of the digital book app market.

Re-readability – All books need a rest sometimes and can feel fresh a month later, but some simply don’t have enough content or have elements that get stale after just a few reads. This category therefore refers to how much overall value the app offers, either as a book app that can occupy a child for hours or one that will be returned to for many re-reads. Some delightful book apps are very short, getting slightly lower ratings on this scale. Other books with lower ratings are fun for the first few reads, but the magic wears off quickly. The highest rated books in this category have either timeless stories that a child would want to read over and over or a lot of content to keep a child engaged. This is an important category that might help a reader decide how much they want to pay for an app.

We also will be introducing a new system for readers to rate books they own, revamping our user ratings, so stay tuned for this feature by the end of summer …

Since these ‘star’ qualities are the thing that really sets our review site apart, we hope they make finding a good book app a bit easier as our database grows to over 200 reviews. But it is a work in progress and sometimes a bit complicated to explain. If I had one thing I hoped readers, developers and anyone visiting our site would take away about these stars, I would emphasize that low stars in a category (except for ‘overall’) doesn’t necessarily mean a book isn’t great … it just means the book isn’t a great choice if you are looking for a book that is strong in that category.

I hope these rating categories will be useful to readers, and most importantly, that they will allow the books in our growing database to be sorted in meaningful ways. I also welcome your feedback!

Category: All About Apps, Welcome/About

About the Author ()

Carisa Kluver is the the editor of Digital-Storytime.com, an iPad children's book review site. She has a BA in Anthropology from UC Berkeley and an MSW from the University of Washington. Before starting this project, she was a school counselor, health educator and researcher in child & maternal health.