Scientific Method in Iterative Playtesting

Playtesting and iteration–or, updating your prototype to newer versions over time–is absolutely essential to making good games.  As designers, we learned that lesson the very first time we inflicted our first game upon our friends and families.

Regardless of how important we all know this process is, one of the most common trappings I see newer designers falling victim to is aimless playtesting.

Game design is an art, yes, but it is also a science.  Relying on your intuition is fine, even necessary, when evaluating your playtest sessions.  However, the best results are achieved when you combine that with a deliberate methodology.

Game Flow Chart

Like so.

So, why bother with a method?  After all, not everyone is under a deadline.  Many designers I know have their one pet project that they’ve been polishing as a hobby for years.

Method is important to iterative playtesting because it gets you the best information possible as fast as possible.  Even if you’re not under a crunch, efficiency is still important.  The less time you waste on broken versions of your game, the sooner your game will end up on a shelf.  For experienced designers who are working on multiple projects, time is money in a very real way.

As it turns out, the method for rapid prototyping is very similar to another familiar favorite–the Scientific Method.  Let’s take a look at how we can apply that method to the iterative process.


1) Make an Observation

So your testers have been playing your game, and maybe they’ve even liked it.  Nice work!  But something is still wrong…

Unfortunately you’ll need more than that.

“Players aren’t really paying attention to what’s going on, and they’re frequently forgetting rules that I thought were simple.”

Much better.  The first step to improving your game’s next iteration is identifying its problems as specifically as you can.

Layperson tester feedback is a fantastic way to tell if there’s a problem with your game–although they can’t always be relied on to tell you exactly what that problem is, or how to fix it.  Beyond verbal feedback, simply watching people play your game can yield some interesting revelations.

Always be prepared to engage in a post-session QA–not just with your testers, but also with yourself.  Some testers may be reluctant to offer feedback without specific prompting, or may not be accustomed to analyzing their own feelings about a game experience.  It may take a bit of work on your part to get the feedback you need.

Try some of these example questions for your playtesters, from the CGS baseline satisfaction surveys:

  • Are you satisfied with the experience that this game provided?
  • How did you feel about the theme?
  • Do you feel like the game took an appropriate amount of time?
  • Did you feel at all overwhelmed by the game’s rules or nuances?
  • Did you feel like the game was fair?

Then, based on our six pillars of game design, ask yourself some of these example questions:

  • Exploits – Are there any lines of play available to one or more players that offer an undue advantage in a fashion that is not in the spirit of the game?
  • Strategic Balance – In a game that incentivizes multiple routes to victory, do each of these routes offer equal opportunities to players?  Are they equally approachable and engaging?
  • Depth – Was the the result of the game appropriately reliant on the players’ decisions?  Are any false choices present in the game?
  • Effect Balance – Did any of the effects present in the game offer a value that was incongruous to its cost?  Was variance overly punishing or lacking in tension?
  • Time Index – Did the game end before the players wanted it to?
  • Engagement – Did players experience engagement loss?  Were they as attentive as you’d expect them to be as individuals?  Did you notice excessive mobile phone usage?  How many times did you hear “Who’s turn is it?”
  • Strategy/Time Ratio – Did players have enough time for any well-executed strategies to come together?  Did they have too much time?
  • Hook – Does my game do anything better than or different from other comparable games on the market?  Is that enough to make it interesting to the consumer?
  • Density Index – Were the game’s depth, weight, and complexity appropriate for the intended experience?
    • Did players feel as though they could solve the game?
    • Were players blindsided by the consequences of their decisions?
    • Were players trapped in analysis paralysis more often than you’d expect from them as individuals?
    • Did the number of quality choices present in the game fail to meet players’ expectations?
    • Did players find that the amount of time and cognitive load necessary to invest in the game paid off in its experience?

Beyond these general questions, you should always have specific questions that you’re looking to answer in playtesting…but we’ll touch more on that later.


2) Formulate an Hypothesis

Great work.  You’ve identified one or more problems with your game.  Next, we’ll try and figure out what went wrong.

“I believe that players are losing engagement, and that a lack of density and depth is the culprit.  My draft feels like there are too many false choices.”

The more experienced you become at game design, the sharper your diagnosis will be.  There’s a world of difference between “my game isn’t fun” and “players don’t feel engaged by the choices presented to them.”  The best way to make keener observations is to practice.

Regardless of how specific your observations were, now it’s time to ponder.


Think think think think…..

Try to attribute your game’s problems to specific elements by asking yourself why those problems exist.  This is a difficult part of the iterative process for two reasons.  First, it forces us to dissect something that we’ve put our love into building, and maintaining objectivity can be challenging.  Secondly, accurate hypotheses require accurate insights, and insight comes from experience that not everyone has.

A methodical approach can be of benefit here as well.  Starting at the session’s surface, ask yourself if there is anything that is likely causative of the problem at hand.  If you’re confident that there are no suspects at that level (or even if you find something), then move to the next level of depth and look again.

The game elements that I (generally) investigate are, in order: circumstance, dynamics, effects, operands, and structures.

Circumstance – Sometimes the game session itself is to blame.  Were your testers outside of the game’s intended audience?  Was blind playtesting contaminated in some way?  Did a social exchange unrelated to the game impact the players’ experience?  Did you start the game at the end of a long game night?Hierarchy

Dynamics – Dynamics are defined as the organic interactions between your players and the game.  For example, if giving an NPC gold restores your health, and stealing from an NPC gives you gold, both of these exchanges seem reasonable.  Together, however, they may lead to play behavior that isn’t in the spirit of the game.  Did an unforeseen interaction lead to an unintended game state?  Did your game turn out to just be too long, or drag?

Effects – Subjectively examine each individual effect present in the game, evaluating its value in the game experience.  Look at procedures, actions, and reactions present in the game.  Is the effect necessary?  Could it be more efficient?  Is it appropriate to the game?  Could a more fair or fun version of this effect be implemented instead?

Operands – Operands are the objective figures involved in your game’s balance and execution.  Does the math in your game come together?  Is your scoring appropriate?  Are the costs of your effects proportional to their value and throughput?  Is the variance in your game statistically appropriate?

Structures – Finally, look at the conceptual structure of the game.  Is it possible that there’s a flaw in your design, or that the design itself is bad?  Should you employ different mechanisms?  Do you need to reapproach your theme, turn sequence, or the timing of your effects?

It should go without saying, games are a frontier, so there’s no definitive method for evaluation.  Some of the questions in this section might even be completely absurd when applied specifically to your game.  The guide above is just a suggestion, but one that I’ve found useful in the past.  The take-away is, start from the top and work your way down.


3) Experiment to Prove or Disprove the Hypothesis

Now that we’ve made an educated guess as to why our game has problems, it’s time to put together a plan on how to fix them.

“Making the options in the my draft equally appealing will offer depth and greater engagement.  I’ll replace my turn-based structure with simultaneous play, so players have more to do than look at their phones.”

During this step of the process, you’ll get to exercise all of those creative problem-solving skills you’ve been building ever since you got into game design.  Unfortunately, offering a more detailed method for approaching this challenge is far beyond the scope of this blog entry.  Each game and problem will have its own unique solutions.

The most important advice I can give you is, don’t change too much at once unless you’re fully committed to an overhaul.  Remember, iterative playtesting is meant to give you information.  If you change several things at once to solve multiple problems, you may distort your ability to assess whether or not you fixed the problem to begin with.


4) Playtesting, Interpreting the Results of the Experiment, and Forming New Observations

You’ve plugged your new ideas into your game, made a new prototype to reflect the changes, and your friends are on their way over.

Each change you’ve implemented will set the stage for this test session.  Remember, the primary purpose of playtesting is to gather information.  As such, you should always have a question in your mind that playtesting can answer.  Without knowing the right questions beforehand, your test won’t be as valuable.

  • Does my update fix the problem as intended?
  • Does my update create any new problems?
“Players are looking at their phones much less often now, and they seem very engaged.  I believe the simultaneous-action turn structures are the reason.  However, now players are frustrated with the draft.  It seems like deepening the options resulted in my piling too much text on the cards.  Players seem more focused, but also less satisfied.”

From here, you springboard into your next round of iteration!


In Closing

Iterative playtesting is where science meets the art of game design.  It will challenge you to apply both the lessons you’ve learned and the insights you’ve developed.

angkorIt’s up to you to know when your game is done.  Iteration must come to an end, and there is definitely such a thing as a game being too refined.  In the words of designer Daniel Solis, “It is possible to polish a game until there is nothing left.  Instead of perfect smoothness, focus on finding the right texture for your game.”

Your game should have flaws.  There should be people who dislike it.  As a game designer, you are the creator of an experience, and not everyone can appreciate every experience.  Sometimes, it’s difficult to know when your game is ready for the world.

But, hey, that’s what blind playtesting is for, right?

Leave a Reply

Your email address will not be published. Required fields are marked *