GAP: Metrics for Soft Skills Measurement

If we look at the approaches to measure soft skills, we can identify a number of approaches. The most common approach has been to rely on a range of More Usually, these surveys consist of questions like “I can work with someone who has different opinions than mine” and are rated with Likert scales. The surveys are answered by the manager herself in most cases also by co-workers. The problem with this approach, and with measuring soft skills in general, is that people give socially desirable answers. The measurement results are as a consequence largely subjective and only partial useful.

Another approach is to have group exercises, where one or more trained evaluators score the behaviour of the participants. The risk of social desirable answers is lessened but the risk of subjectivity is still there, as the evaluation is performed by a person. Also, group dynamics are at play, affecting an objective assessment of soft skills.

The question then is whether one can have more objective measures of key soft skills. Within the European Union funded Gaming for Peace-project this question has been explored in depth. Based on this, a specific approach has been developed, geared at increasing the objectiveness of the measurement by blending existing approaches and using the power of technology.

The starting point was of course the serious game for peacekeepers, focusing on situations where specific soft skills relating to culture, gender and communication are at play. An in game assessment is foreseen, allowing to rate the behaviour of players. However, this assessment only rates the response to the game’s actions and is thus limited as it does not provide a baseline and has limited capabilities to track progress.

Therefore, in the project a pre- and post-game assessment methodology has been developed. It consists of three elements:

(a) Pre-game assessment
(b) Post-game assessment
(c) Interviews by evaluators

The pre- and post-game assessment are similar, only differ in timing. One is before the game and the other is administered after the game. The assessments consist of a list of statements grouped into three areas: culture, gender and communication. The questions are statements such as ‘I enjoy the opportunity of eating other police services’ national cuisine’. Game players are asked to rate their agreement with these statements, based on the likhert scale (strongly agree to strongly disagree). This is done immediately before playing the game and repeated after having played the game. The order of the 70 statements is different in the pre and post assessment. Participants are asked to rate the behaviours quickly, without too much thinking.

The pre-game assessment allows to identify a baseline for the game players level of soft skills. The post-game assessment allows to identify the potential learning that has occurred by playing the game.

This learning is discussed in the interview after the post-game assessment. This interview is taken by an external evaluator and allows the game player to reflect on the game and learning points.

The advantage of this method is that it is user centric, as the game player rates behaviours. By having a substantial number of statements and by changing the order of the statements in the pre- and post-game assessment it introduces mechanisms to reduce the social desired answers. Also, it allows to measure whether some behaviours are rated differently after playing the game.

Finally, the interview with an external evaluator is also key to assess the learning. The pre- and post-game rating results are reviewed with the game player. The session is structured in a socratic way, allowing the game player to reflect on the game and explore possible behavioural response in terms of strengths and weaknesses.