The benefits and perils of doing user research on games –

It’s easy to understand how small indie games sometimes turn out badly. Whether the concept is poorly explained or the game suffers from poor design, it is often partly due to the fact that the developer lacks the money or time to do sufficient user research.

Which makes it all the more confusing when larger games, backed by publishers with big budgets, sometimes fall flat despite plenty of resources. Shouldn’t they – with all their careful testing and continuous feedback gathering – have seen it coming?

Earlier this year, Gamereactor spoke with Jonathan Bonillasm, who works for Nordisk Games and has nearly a decade of experience in user research. According to Bonillas, the simplest explanation for failed user research is that the people in charge of the game sometimes refuse to read the writing on the wall:

“I do run into selfish creative directors, and they don’t want to believe the data. They’re just completely against it and don’t want to change anything. You try to build trust, you try to turn it around, but no matter what you do, they don’t want to believe you or they want a relationship with user research.

Usually a game that is a flop despite much testing is not the fault of a single individual, but has to do with management issues, different creative visions or limitations in terms of time and money. Factors that can be difficult to manage when dealing with large projects. And of course, user research is not simply a magic formula that makes every game better. The answers – qualitative or quantitative – still need to be sorted and the relevant changes implemented by the developer.

Risk of leaks
For the team doing the actual user testing, there are also many elements to consider, Bonillas explains. First of all, the testers must be well informed, because it doesn’t help much if they give feedback on the difficulty level, if the developer wanted to improve the user interface. Another important element is to build trust between the team doing user research and the developer, if the feedback and testing are not done internally. Finally, it also helps to start early while changes can still be adopted relatively easily.

However, this opens up a new can of worms. With early playtesting, details about the game can risk being leaked, which can lead to problems down the road in terms of building hype or managing expectations.

“There are a lot of developers I’ve worked with where they don’t want to do user research because there’s this fear [voor lekken] is there. Frankly, that’s a fear you really have to get over. You can ask yourself, ‘What are we more afraid of? Are we afraid of the game leaking out, or are we afraid of the game being bad?’ “ says Bonillas.

That said, Bonillas mentions leaks “an ever-present concern” and explains how steps are often taken in terms of having strict NDAs (Non Disclosure Agreements), putting phones in lockers during testing sessions, and so on. But as he explains, very few gamers test to cause problems, and during his nine-year career he had only two instances of leaks emerging from user research.

Early Access and creative control
Over the past five years, a new way of testing games has emerged that in some ways bypasses both traditional user research and the risk of leaks. Early Access, which allows players to buy an unfinished version of a game, can help fund production and provide valuable feedback. But it’s not exactly a bulletproof method.

“I think Early Access is a good way, if you don’t have money for user research,” says one smiling Bonillas. “But what’s difficult about Early Access, and the only real drawback, is that feedback can come from anywhere. And it can be very muddled. Also, someone has to sort through all that data. That takes a lot of time and effort.”

Early Access also points out some of the broader concerns about user research in general. Do developers run the risk of losing creative control of their game if they receive constant feedback? And will they have to give in to the public?

Jonathan Bonillas explained that this is rarely the case because user research is often focused on improving specific elements of the game, compared to more general market research that tries to identify what is currently popular or trending. But, as he revealed with an anecdote, sometimes developers take feedback too much to heart, such as one time a small studio, after getting user research, simply ditched their existing concept and came up with an entirely new game!

Saved by marketing
As a final question, we asked Jonathan Bonillas if he had ever worked on a project where all user feedback was negative, but the game was still well received by players. Perhaps not surprisingly, he answered no, but he revealed that sometimes a game can still sell well even if the feedback from playtesters was mostly negative.

“There are games I worked on that we knew were going to be bad, but the marketing is so good that they still sell five million units. Sometimes the company sees it as a relief. ‘Great, we sold it, we got our money back and made a little profit. We knew it was going to be crap, but at least we got our money back.’ Sometimes that’s the case, especially with the big budget titles. And that’s, because it doesn’t really motivate me”, he laughingly explains.