Armchair Reviews: Thoughts on Approach

I'm about to start reviewing some products for the DriveThruRPG and RPGNow sites, and it's made me think about methodology.

How does one give a fair review?

First off, I'd have to have a quick disclaimer that my review is based on a mere reading, rather than an actual playtest of the material. I think this is important, because -- and I think most would agree -- reading a good RPG rulebook or sourcebook is a different experience from actually using it in a game.

This observation is not meant to lower the importance of the initial reading and impression of the material. That first experience informs and inspires the GM and his/her players as to the scope, themes, subtexts, vistas, archetypes, and possibilities of the game system, game setting, or game adventure.

Second, I'd have to review the material not only by component elements (chapters, writing, artwork, layout, fluff, crunch) but also by the gestalt impression left on me. This is because certain works work despite the mad, frankenstein's monster approach to the content and presentation, and certain works lie stillborn on the mad scientist's worktable bereft of life despite their surface beauty.

Third, I'd have to figure out how to condense all that into a five-star rating mechanism.

Hrm.