Dice are magical little devices. Who would have thought a couple of small plastic cubes could generate that much excitement, tension and drama. They can also be deceptive to the game designer and the play-tester.

Play-testing is of course the crucible where game ideas are hammered out, broken or confirmed but especially with small sample sizes it can be very unpredictable, and most gaming groups will produce a small sample size due to the time constraints involved in setting up and playing a miniatures game. 

Let us say we are testing out a particular type of weapon in a science fiction game. The Super Laser (TM) is intended to be a powerful heavy weapon for attacking tough vehicles but the play-testers report back that it seems to be far too weak for the assigned points value. What happened? 

Well, first and foremost you may of course just have gotten the cost wrong or the stats do need to be tweaked, but what if the luck was awry? How many times does a single unit get to shoot in a normal miniatures battle? I wager 3-5 times in a lot of systems so if we have one unit with the Super Laser, that's how many changes we get to evaluate the effectiveness. Assuming a pretty conventional game system with hit dice and effect rolls how many of those shots hit? Missing three 3+ hit rolls in a row on a D6 is a 1 in 27 chance. Not very likely but also not impossible at all. If I offered 27 donuts and told you one of them was filled with ants you probably wouldn't take a chance on Weasels Donut Surprise Shop (Especially with a Dunkin' up the road).

Of the shots that hit, how were the rolls for effect? What happened on the damage tables? We often pay attention mainly to overall luck (and even then our perceptions are often heavily skewed) but even if a player has perfectly average luck throughout the game, not every roll is equally valuable. Playing a board game session recently, my opponent had a pretty representative mixture of good and bad rolls, except when attacking one particular city where he got three 1's in a row. This city happened to not only be an objective but was also pivotal to continuing the overall advance and thus be able to win the scenario. 

These factors can of course all apply in reverse. The dice can be hot and sometimes even the weakest unit can destroy the enemy elite. 

I hope this helps make it clear that you need to look a bit more closely, if you were present at the play session and can examine exactly what happened. You might even keep notes on a particular unit or character. This can be difficult of course, so trying to get data over a number of games is often more reliable. 

Once you have a grasp on the probabilities involved, you can then start looking at other questions such as how the unit was used. An expensive anti-tank gun that ended up spending the whole battle shooting at infantry is not "earning its keep" and an error in deployment might have stranded a strong assault unit too far from the objectives. 

These factors are much more difficult to control for and again requires more data to work with. It is worth noting that you can still learn something from such a battle. If the super weapon is so expensive in points that the rest of army is now too limited, that may be an indication that things need to be scaled back regardless.

Happy hunting.