First responders, disaster relief personnel, and members of the military depend on their teammates in order to succeed in their mission; however, the high costs of data collection have made understanding team dynamics difficult. Recent work by researchers at Rensselaer Polytechnic Institute is demonstrating how data from online games can help provide meaningful insights.

Using data from League of Legends, one of the world’s most popular online role playing games, the researchers have employed big data techniques to develop models that reveal how team membership, working history, and other factors affect performance outcomes.

These findings, which were recently published in Proceedings of the Human Factors and Ergonomics Society Annual Meeting, have broad and critical implications. The data — on hundreds of thousands of matches played by approximately 100,000 teams — has allowed the researchers to build and test advanced statistical models and explore complex methodological challenges in the study of teamwork.

Their research on League of Legends is aimed at determining how best to measure and monitor individual team members in order to improve performance and learning over time, particularly when it comes to more critical situations.

“Our approach enables us to ask questions that are exceedingly difficult to ask in ‘live’ environments such as military combat,” said David Mendonca, an associate professor of industrial and systems engineering at Rensselaer who is one of the lead investigators on the project. “Data on teamwork at this scope and scale are simply not readily available.”

With the League of Legends data, painstakingly compiled from publicly available sources, the researchers were able to test their instrumentation and design their studies so as to improve data collection in a high-stakes setting.

Typically, Mendonca said, it’s difficult to perform these types of team studies because a large number of research subjects are needed to get enough data. Team members also, ideally, need to have worked together before in order for researchers to explore their behavior over time. The League of Legends data set contains thousands of examples of teams whose members have played hundreds of games together.

“We’re trying to understand processes of learning and adaptation, particularly when teams are subjected to unforeseen or unusual events, such as the loss of team members,” said Mendonca.

This recent paper builds upon past research done by Mendonca and Wayne Gray, a professor of cognitive science at Rensselaer, as well as Joshua Eaton, a doctoral student in industrial and systems engineering at Rensselaer. They previously found that there are critical positions within a team that can significantly affect a group’s goals. For example, when a player filling a critical position is “down,” negative events within the game increase for their team.

Mendonca and Eaton also found that, if team members are familiar with their position, it can positively affect the effectiveness of a team. What this research has enabled is the development of metrics that will allow Mendonca and his team to try to answer one of their driving, and complex, questions: What is the effect of turnover and how can they measure it?

The Office of Naval Research supports this work through a grant, for which Gray is the primary investigator. He characterizes the overall effort as trying to find the “I” in “team.”

“Can we find the contribution of the individual in a team?” said Gray, whose work on performance in single-player games, like Tetris, has led to pathbreaking discoveries on learning by experts.

Ultimately, this work is strongly focused on uncovering which metrics are the most meaningful to measure and how to collect information that will prove useful in further study. With these results in hand, Mendonca said, subsequent research is expected to explore the timing of re-staffing in military and other organizations in order to support performance.