Frogboy

Join Date 03/2001

+2394

At the mercy of reviewers

When the fetal position won't solve your problems

In the 15 years or so I've been making games I've found that there are three types of reviewers. That's a gross over simplification of course but I'm a grossly oversimplified kind of guy.

Drive-Byers

The first type of reviewer is the drive-by reviewer. If you're an indie developer, you are more apt to run into this. That's the review that makes you feel like, as you read it, that they were miffed having to take a time out from their first person shooter to play some little indie title's turn based strategy game. We don't get these as much as we used to but we still get them.

These guys scare me because I honestly don't feel like we get a really fair shot. And I'm not saying they give only negative reviews. And that's the worst part. Because if you complain about the review when it was a positive rating you will get "What? We gave you an positive rating?" You can almost hear the "You make a great little shareware type game, I'm sure really hard core nerds get a thrill out of your little game."

But the drive by review can be negative too. If they run into any problem, don't expect a follow-up. These guys also live to put in snarky comments in their reviews -- nothing makes ya look cooler than to practice your written venom at the expense of some hapless developer.

But don't take my word for this. Go over to www.Gamerankings.com some time, think of the best indie games of all time, look at the reviews. Then go and think of some big budget drek that's come out and look at who gave them 90s or higher equivalents.

Casual Reviewers

The second type is the "Eh It looks like it has good production values, seems solid, haven't heard of it, looks good but won't give it better than 4.5/5 because if it was truly outstanding, they wouldn't be a small company." I've said this long before our latest game has come out as I followed the reviews of some truly outstanding and innovative games from small developers get an 80% on the same day they'll put up a 96 for some of the worst shovelware from some mega publisher.

The second type though isn't a drive-by review in that it's obvious they played the game decently. Moreover, these reviews are probably closest to what the overall average will be from both reader reviews and all sites combined. Rather, it's more like there's just a cap. I don't really have an issue with these types of reviews because let's face it, it takes a long time to really review a game and we are a smaller developer. As long as we're not patronized I'm a happy camper.

The text to these reads like this "Great great great great great great." with a 4/5 rating.

I don't get worried as much about the second-type. I know we won't get super high reviews but I know we won't get really low reviews either as long as we've put together a solid title.

The Ultras

But it's the third type that scares me the most. The third type is the hard core reviewer. The person who is going to take your game apart. The ULTRA reviewer.

Ultra reviewers dissect a game to make sure it does what it's supposed to do. They know the genre, they know who plays them. It's not enough to make a polished, solid game. Because they know that many gamers could care less about that. To them, the question is, is the game FUN?

These ultra reviewers scare everyone equally though. Whether you're Microsoft or Stardock, they'll crush you like a bug equally. They're freelancers. Publishers have no power over them. Go ahead, threaten to take your ad dollars away. They'll laugh at you.

They may write for big sites or they may write for small sites. It doesn't matter. Their words carry weight because of years of built up integrity. If they give a big name game a 2 star rating, people take notice. And when they give out 5 star ratings it's equally a big deal. It's not that they give a lot of negative scores, it's that they're immune to hype.

Ultras in action

QuarterToThree had a good article that dealt with this. It links to one of Bruce Shelley's (designer of Age of Empires III) blogs.

Bruce wrote:

I am struck by the fact that the two biggest PC gaming magazines in the US gave Age 3 such disparate scores. Computer Gaming World gave it 60% (three stars out of five), while PC Gamer gave it 91%. That seems like it would be very confusing to readers who are considering whether to try the game. Players will establish eventually if the relative score of the game was at one of these extremes or the other, or somewhere in between. Data we’re seeing says that Age 3 has been perhaps the best selling PC game in the world since release in mid-October and that certainly suggests that the 60% score miss-represents the true achievement of the game.

No, what happened is that the CGW review was done by an Ultra and the PC Gamer one was not. I'm not suggesting Age 3 "deserved" any particular score. That's not for me to say, I'm not a reviewer. Just that I, and others who follow this stuff know who the reviewers are and their reputations.

I'm also sure Bruce knows that game sales are not the sole indicator of a game's quality. As a game designer one is almost tempted to put together a crude formula for what determines sales: DISTRIBUTION X MARKETING X GAME QUALITY X DEMOGRAPHIC SIZE.

If my game were massively distributed, I wouldn't use sales as the thrust of my argument. Deer Hunter made the case that we shouldn't rely too much on sales as the sole determiner.

What he's pointing out is that the CGW review of Age of Empires III was not to his liking. The review (and CGM one) points out why they thought what they thought.

If he thinks that's rough, he should walk in our shoes where we could get the same rating for the sin of not being well known. After all, if we had a great game, wouldn't everyone already be hyped about it? That's the real reason games come with fancy opening cut scenes in case you've wondered. They help provide a clue as to a game's production values which is jokingly called "Driveby-begone" in some developer circles.

Ultras and Indies

Ironically, if you're an indie developer, these mega reviewers are your only chance of getting really high scores. And they don't give them out often as Ensemble observed. So talk about being in the shark tank. In fact, they're far far more likely to give out negative reviews than any other type.

Let me be clear lest I give the impression that the mark of an Ultra review is someone who craps on a game. No. What I'm saying is that an Ultra reviewer is simply someone who knows the target audience -- that they are the target audience. And they don't care who you are. They care only about the game.

And anyone reading their review -- including the developers -- will observe and despair at the powerful evidence that justifies their opinion. If your game is good, you'll get a great review and score. But if it's bad, it doesn't matter whether you're EA or Sony or whoever you're dead meat.

When Ultras follow up with developers, they can make the developer realize that they're the worst game designer in the world and that they should have stuck with Electrical Engineering and stayed the hell out of the whole "game thing". And that's even if it ends up a positive review.

Because no game is perfect. The question is purely how flawed the game is and how flawed is it compared to other games in its genre. But as a developer, the case Ultras can make is devastating. Design decisions you thought were sound can suddenly be called into question as they put forth rock solid arguments. Bugs you never heard of are suddenly obvious and easily reproducible.

Not all Ultras follow up. But Ultras tend to be the most thorough and careful. If they run into an issue or have a question, they're the most likely to contact the developers and get clarification.

GalCiv II reviews

Ultras put the fear into me I can tell you that. I'll admit, on GalCiv II, more so than our other games or products in the past, I've followed the posted reviews as well as I can. That's because this game is close to the best I can do.

I can see tons of tweaks here and there but realistically, unlike our other games, I don't look back and wish we could have done a lot of things differently. When we finished GalCiv 1, I wasn't overjoyed. I felt confined by the budget and resources and the MOO3 fans had convinced us we were about to get crushed. And I agreed with them. But not so with GalCiv II. Everything came together.

So I'm particularly sensitive to the reviews. Not so much because I expect a certain score as much as I'm worried deep down that I'll finally have proof that I am not the game designer I'd wanted to be. Reviews so far have been positive. There's been a sampling of all three kinds of reviews. And of course, few reviewers neatly fit into a particular category. Like I said, we're talking a gross oversimplification, sort of like ratings themselves.

I've dealt with the ultra reviewers and they're like the Babylon5 Inquisitor episode. They'll send in questions, point things out and by the time they're done, you not only suspect but know that your game is the worst game ever and that you wasted a lot of people's time having the audacity of releasing such junk. I tend to be fully prepared to agree with criticism. It's in my nature.

One of the ultra reviewers, Jeff Lackey (Gamespot), really liked the game. It was a huge relief. Admittedly, I was terrified before it came out. Like I said, these guys find flaws and design decisions that you might have previously thought were good ideas. But when they're done with you, you start tidying up your resume -- just in case of course...

Ultra reviewers finding flaws doesn't mean your game is doomed to a negative review. But when confronted with a design decision or a bug, it's hard not to start having second thoughts on what you've created. They follow-up. They ask questions. They point things out. And when you're done, you're jelly.

Not all follow-up of course. Sometimes you'll just end up reading a review in which you can tell they understood what we were going for -- for good or bad. The Ultras are the reasons some indie games get a peppering of very high scores (see Darwinia) and why some big named games get a peppering of not so high scores.

I'm aware of a couple of reviews that involve Ultras that are in progress. They aren't likely to see this post and even if they did, it would matter nothing. I could say "Reviewer X, your mother was a hamster and your father smelt of elderberries!" and it would have no effect. The reviews may be very positive or very negative. They may cover minor game flaws or, depending on how many words they're allowed, may choose to focus on the key elements good and bad. It's hard to say.

I plan to link to all reviews that at least seem like they gave us a fair shot. A fair review is all anyone can really ask for.

updated: Edited, cleaned up grammar, terminology.

Locked Post

Advanced Search

Watch this post

Do not email me updates for this post Email me updates for this post

Successfully updated karma reason!

An interesting read, Brad. I admit I'd never really considered before the dynamics between developers, their games, and the reviewers that can make or break them. It would never have occurred to me that GalCiv 2 might get dinged in reviews simply because some consider Stardock to be an "indie" game developer (although in retrospect I admit it makes sense, even if it's not fair).

By the way, which other mega-reviews are left? I thought all the major ones were already out, unless you're including Wargamer. Are you talking about the magazines? If so, I wonder why would they might rate the game noticably lower than almost everyone else. After all, GalCiv 2 so far has an average score of 9/10, and the Big Three (Gamespot, Gamespy, & IGN) all gave the game their Editors Choice Award. If GC2 gets slammed by any of the printed publications now, they'll risk looking like they either didn't spend nearly enough time playing the game, and/or ogres who are just nitpicking because they can't find any major faults with it.

Successfully updated karma reason!

Some interesting points, and I'll make some myself, you see, I'm a (gasp!) GAME reviewer.

I see you are somewhat miffed about small companies not being noticed.

I also see you linked to gamespot and other reviews, yet I recieved a copy of this game, reviewed it, and didn't even get a follow up when I provided my review.

When I do articles for Strategy informer, which has a decent following, I will get to see the review linked, but when I do them for my syndicated sites like Killer betties, not so much as a e mail back.

Basically what I am saying is you do the same thing to the little guys you claim the big review sites do to you.

Just in case some might question my credentials, I have something like 70 published game reviews from three different review sites.

Successfully updated karma reason!

I did a review for a semi-small site (well we've got a print magazine backing, but the site is primarily geared towards a niche audience - wargamers). Brad was way more helpful than he needed to be with me.

Moving on to the topic at hand, those were some interesting points, I'm not sure I entirely agree with the categories but they're generally a good way to look at different styles.

Successfully updated karma reason!

I also see you linked to gamespot and other reviews, yet I recieved a copy of this game, reviewed it, and didn't even get a follow up when I provided my review.

I don't think that's a fair charge. I've tried link to most of the reviews of GalCiv II as they've come out in news items regardless of size of site. Heck, I've even linked to blogs. If I know about the review, I tend to link to it, even if they're not the rating I'd like as long as it seems like they gave the game a fair shot.

Basically what I am saying is you do the same thing to the little guys you claim the big review sites do to you.

Except

A) I don't think your assertion is true here in the general sense (I am really trying to link to everyone I know about

and

I only know about the reviews that are on Gamerankings. Tell you what, you give me the URL of the review and I'll make a news item on it even if it's a negative review.

Successfully updated karma reason!

Well first, most reviewers don't care what other reviewers rate a game. It's their review.

Secondly, what makes the ultra reviewers so effective is that they make their case so well.

I don't know for certain we'll see any really negative reviews. I am not privy to that. I am only saying that the really thorough reviewers see both the good and the bad in great detail and as a developer, seeing lumps in ones own design that they themselves missed is scary.

The ultra reviewers of the world can give out 5 star ratings or they can give out 1 star ratings. But regardless of what they give out, you can be sure there's a good reason for it.

Successfully updated karma reason!

Brad, I made earlier comments about how important it is to make a demo available. What I forgot to mention is to have a demo available at the time of release, becuase reviewers will usually link to it. At the very least reviewers will link to your official site, you will get that first wave of intrested gamers who can download the demo and make up their own minds. Reviewers and Reviews are important basically as advertising, getting the word out; or demonstrating that a game isn't simply not fun, but unstable or otherwise broken. Beyond that I don't listen to game reviews on games I'm intrested in, the old adage about Opinions (Everyone has them....) holds true.

Successfully updated karma reason!

Have you considered taking a vacation?

Successfully updated karma reason!

I usually ignore the stars and read the review. I've played some really terrible games that have gotten good reviews and vice versa and found that much of the review hinges on certain aspects of the game --- I'm sure not have multiplayer hurt your reviews but it's not a factor for me since I only get to play for an hour a night usually so I'd never multiplayer. Personally I like Gamespot for their "vital stats" area which shows their review, the average review and what gamers are saying. I've found some games where there was a full point difference between the reviews and what players thought.

Successfully updated karma reason!

#7 - doesn't EVERYONE working hard consider taking a vacation? hahaha
The thing is, being in advertising and amrketing PR, I know that the interviewers and "white smoke boys" never leave you alone - and if they can't get a quote or idea from the 'top exec pointman' then they'll get it from somewhere else.

I feel for you Brad - you'll probably have a couple of more weeks of 'stimulus' overload, and then you can finally think of taking that team break eh? Hmm, thinking of going to the year 2225 and relaxing on a Class 18 planet?

Successfully updated karma reason!

I would think that Jakob from Firingsquad is one of the better reviewers, as he is the the only reviewer whose opinion I respect. (Although I generally don't read large site reviews because they have so many reviews that ignore problems.)

Successfully updated karma reason!

Jakub is a top reviewer. But one can see that from just reading this review. It's very thorough.

And yes - definitely looking forward to a vacation. Not a day off since Thanksgiving still.

Successfully updated karma reason!

Its not so much a charge as an observation of my personal experience.

I have long been a huge supporter of smaller companies, and will give any game a fair review, and I mean that. I played GC 2 alway through three times before writing anything.

A) Except

I don't think your assertion is true here in the general sense (I am really trying to link to everyone I know about

and

That's really not what the article was about. It's not about whether we get reviewed. It's simply an observation on the different kinds of reviewers out there. The size of the site doesn't determine the quality of the reviewer. In fact, often the opposite is true. Some of the best quality reviews i've seen have come from bloggers.

I only know about the reviews that are on Gamerankings. Tell you what, you give me the URL of the review and I'll make a news item on it even if it's a negative review.

My experience in this business (four years now) is that many reviewers don't even play past a tutorial. I can generally tell if this is the case from the game comments, or lack of them in reviews.

The issue is speed. Many review sites won't pay for a review if the game is out or past two weeks, and those that print free reviews generally feel this way also.

About small publishers, the general feeling is that unless its a mind blowing game a review of a low sales game is a waste of band width, I DO NOT share this opnion, but I don't make editorial decisions, I simply request games and submit the reviews.

Getting back to the topic about snarky reviewers, I feel they are worthless.

The sole purpose of game reviews (IMO) is to tell a potential game buyer if I thought a particular game is worth their limited gaming dollar. I know full well that some reviewers set themselves up to make or break game companies, I have little use for such jack asses, I can honestly say I have never launched any kind of attack through a game review, even if I found the game in question lacking.

On to GC 2, I liked the game a lot, it has plenty going for it.

Is it perfect?

No, no game ever is, but its certainly a huge leap in fun over such messy attempts as MOO3 to do a space empire game.

The comitment to patches and upgrades is also a major selling point, its something Paradox, the European distributor of this game is famous for, and its a great business practice.

You can find my review [url=http://www.killerbetties.com/modules.php?op=modload&name=News&file=article&sid=1542] here. [/url]

Successfully updated karma reason!

Jakub is a top reviewer. But one can see that from just reading this review. It's very thorough.

I can tell you that I bought the original game only because I read his review on it in 2002/2003, or whenever. (I thought it was in 2002, when I was still in school, but I think it may have been 2003.) I stopped at Best Buy that weekend and bought the game. It was fun, but it got monotonous after a while because of the graphics and sound. Even so I was still playing it on and off until I bought the sequel.

Successfully updated karma reason!

I've never really understood how reviewers can give one game a review like "This is a decent game all around, and worth the time to play: 3/5" and then turn around and say "We weren't all that happy with this, there were a lot of problems. Rent it if you like this kind of crap, but don't buy it. 3/5"

You see it on G4 all the time. I've seen them review two games in the same show, hailing one and blasting another, and yet giving them both the same rating. For some reason you tend to see this from the 'ultra' or professional reviewers a lot more than the low-end reviewers, who lean hard toward 10 or 1 on everything they rate.

Successfully updated karma reason!

Stardock and WildGames/WildTangent are my 2 favorite publishers right now, hands down... big or small.

It's really not surprising that the "make or break companies" reviewers exist (and are probably paid pretty well on a comparative scale) -- same thing happens in the music industry.

It's a shame, though. Indie games, much like indie music, have far more to offer than the canned laughter the big guys usually crank out.

It's very much the same as in the art world. Lots of cronyism and nepotism, which inevitably leave the geniuses of the world all but unknown until a couple centuries after their deaths.

Successfully updated karma reason!

Good read Brad. Thanks. As usual, it's more interesting the great majority of texts out there about the gaming industry.

As for the 1 or 10 reviewers, a part of this might have to do with the fact that people don't play every single game and so it really goes "I buy the best, and the rest I never look at". I still think that good reviewers make nuances, but it's a bit like stars which are sooooooooooo great some seem to not be able to find ways to describe them but by hyperboles. Often, only those with some experience and/or formation know a bit more how to deal with hyperbolic material; we see this in academic fields too.

Successfully updated karma reason!

Maybe the english's reviews are mostly dones, but I am still waiting for www.jeuxvideo.com to take out a french review. As I remember, they said high praise about GC1 (which incited me to download the demo, and the demo conviced me to buy the game, which I sporadicly played for over 3 years. The best 30$ spent on a game).

Do you plan on translating the game in french any time soon? Specially if you can create a french version of StarDock to download the game, and can convince the major french reviewers to link to your francophone website. Would do a huge boost on your sales.

Successfully updated karma reason!

I recentily got the new issue of game informer, main review gave it a 8, second opinion gave it a 7. Strange is that in the description of the game, they said a great 4x strategy game that could give Civ4 a run for it's money or something to that effect. They did say alot of the gameplay felt like all they were doing was hiting the next turn button.

Successfully updated karma reason!

Alstein

Join Date 07/2004

+55

Well, Game Informer is one of the sites I trust least for reviews- since the magazine is pretty much owned by a company whose main interest is moving their game product- which leads to overinflated scores and payola schemes.

Successfully updated karma reason!

Oh man you pointed out the main reason that I stopped watching G4. The whole stupidity of not using a wider scale just boggles the mind. To add to this remarkable stupidity, they give virtually EVERY game a 3 out of 5. It's almost as if any game that would require actual analytical analysis is too difficult for them to judge so they give up and give it a ...(wait for it)...Three out of Five. As an added insult to their listeners they made a song for one episode where they basically mocked their fans who called them out on only using a 3 out of 5 scale. Bizarre.

Well, Game Informer is one of the sites I trust least for reviews

No kidding. I read their mostly inane review score in the most recent issue (got stuck getting the magazine as part of my deal to get a discount on used games there) and just marvelled at their ineptitude and poor understanding of anything PC-related. The main problem with that magazine is that they're slanted completely towards console gaming. They wouldn't know a quality PC title if it bit them in the @$$.

There, I feel better now.

PS. Brad aka FrogBoy, I loved this article. Very interesting and full of cool insights into the world of reviewing.

Successfully updated karma reason!

One of the things about GalCiv II and is a lesson for GalCiv III is that the game does hand you enough rope to hang yourself.

It's very easy for a new user to set up a game where ther'es not a lot to do. You create a large galaxy, make planets relatively rare, only have a few opponents and wham.

One nice polish point of Civ IV is that the map size helps determine the recommended # of players. Moreover, they have a Pacing slider (Epic to fast) which is very helpful.

Most players of GalCiv quickly nail down what type of setup they like and go with that. But a reviewer, especially one who doesn't really care about enjoying the game or not but rather just wants to crank out a review, may not have any interest in playing around with the settings.

Because GalCiv II is a single player game, a lot more emphasis was put in to having a greater variety of starting conditions to match very different needs. For instance, we intentionally made it so that starting conditions aren't perfectly balanced. They are balanced within a X % tolerance. Users can hit Ctrl-N if they'd like to quickly jump through different game startups until they get the one they like. But that's intentional.

Successfully updated karma reason!

the good news is that once you do establish yourselves people take note of the name of your company and start to look for it. and outside 'reviewers' get the short shaft. ive yet to read a review about GS 2. i know stardock from GS 1 and was very happy with the game and more importiant the support.

i cant speak for everyone but i can say for myself that i only read reviews of companys ive never heard of before, and even then i tend to ignore the nit picking and read if for the over all flavor of the review. was it a good idea that was reasonably made or was it crap from the starting line?

ive had , time and again, a case of my opinion of a game not even close to the opinions of a reviewer. some they loved and i hated and some they hated i loved. as my grandfather used to say "opinions are like a@@holes, they stink".

at the end of the day ill say this. dont sweat the 'professonel' reviewers to much. they are truly a dime a dozen and you can lose the forest for the trees. one persons opinion wont cripple your company and shouldnt be causeing you to lose sleap over it. after all reviewes didnt make sony or EA, and reviewrs arent breaking them either despite giving them high scores on almost every POS sequil they put out. they are killing themselves with crappy sequils (EA madden 10,000 anyone?) or indifferent customer support (sony, im an ex EQ player).

we the customers will make or break you, and since customer support and game support is a rock solid 10 out of 10 (if i had to rate it) you got nothing too worry about as long as you keep those front and center to your whole company game plan. ive played litteraly hundereds of games, starting on the atari 2600, i lived the pac-man crazyness, i been flooded with so much hype since then from so many 'ubaleet' be all end all, this is a MUST HAVE SELL YOUR CAT SO YOU CAN AFFORD TO BUY IT, games in my 20 years of playing that its all just so much noise now.

but stardock is the real deal. no flash and razzle-dazzle only to fizzle when the install is done and the cut sceens are over. just a rock solid game, with outstanding support. and most of all fun.

i know its tempting to fall under the sway of one loud persons voice and to want the respect you feel your entitled too for all your work. im that kind of a person myself. its one thing to get forums full of people saying "great game' and dismiss them as 'fanboys'. its something else to get a good review from someone who clearly isnt a fanboy. it validats you in a way that forums full of straingers opinions cant. thing is. the reviewer isnt buying your game. we are. as long as WE are happy nothing but good can come from it for your company.

im 34 years old, i play games. i dont do the whole movies/tv/music entertainment thing. i play games. i play games like my parents watched TV/movies. my extra money is spent on games or upgrading my computer. and with taht resume in mind ILL say that your company is great and ill support you guys by buying your products and by simple word of mouth as long as you continue the quality of work you have shown so far. i know im only one man. but its clear from thses forums that im certianly NOT in the minority in this opinion.

as a parting suggestion. perhaps you should go to a company like ohhhhh firefly and read some of their forums. then you wouldnt be quite so worried about what the paid talking heads think. and take notice of the fact that your game IS a success with the most importiant reviewers of all. we the customers.

Successfully updated karma reason!

I give this article a 4.5/5

Successfully updated karma reason!