How should I measure character, virtues, or personality more generally?

To a methodologist, these seemingly straightforward questions aren’t easy to answer. Any helpful response requires careful consideration of the aims and scope of the proposed study. As with any measurement question, there is typically a compromise between the researcher’s interests and available methods. There is the need to balance current standards in the field, specific areas of interest and the characteristics of the target populations.

There is a range of methodologies that are more or less suited for studies of morality and the virtues. For instance, the in-depth interview provides an ideal methodology for assessing the ways in which virtuous concepts are embedded within personality. However, this technique is less appropriate for studies asking normative questions and may not be particularly helpful in younger populations who may not be as reflective about the self.  Additionally, a reliance on verbal performance may be a limiting factor in marginalised populations.

To counter some of these shortcomings, the questionnaire has been successfully adapted to measure reasoning about, and attitudes toward, virtue and moral concepts. Paper and pencil (and now web-based) measurement systems are often used when time is at a premium and the researcher is interested in broad population trends.

However, consumers of this research often wonder whether the responses are truly reflective of participant capacities and express concerns about the test-taking set of the respondents. Do the participants take the measures seriously, or do they simply regurgitate what they think is expected?

The latter concern is pertinent when measuring concepts on which society places a value such as moral/character/virtue considerations. Although this is not a new problem and there are design elements in questionnaire studies to limit the impact of these concerns, the issue of honest responding remains.

I have come to appreciate a third option that uses short stories or dilemmas to highlight the application of concepts. Typically, the stories describe situations in which the protagonist must decide on a course of action. The participants place themselves in the protagonist’s shoes and decide what he or she ought to do – and why.

Much like word problems in mathematics instruction, story approaches ask participants to consider the concept of interest within a realistic context; stories engage the concept rather than simply reacting to item statements in a piecemeal fashion.

The story or dilemma approach represents a methodological middle ground. It provides more context and nuance than the traditional questionnaire approach and yet is adaptable to mass distribution (the questionnaire’s primary advantage). Additionally, dilemmas tend to minimise the effect of social expectations on participants’ responses as the focus shifts from the self to the fictional protagonist.

The success of a story-based measurement system hinges on whether the dilemma engages the participant. A story that is implausible, awkward or obvious is distracting and leads to a lack of focus.

The most straightforward way to ensure the story’s relevance is to enlist members of the population you wish to assess and get them to act as co-developers of the measure. For example, my colleagues and I developed a measure of the ability to apply the virtues within adolescent populations. We used focus groups of mixed-age adolescents to identified stories that emphasised a particular virtue.

Each story was tested multiple times with different groups and deemed realistic by the adolescents. We also collected the informant’s ideas about what the protagonist ought to do and why.

Once this process was completed, we arrived at a set of stories and a corresponding list of items representing the most frequently suggested actions and justifications.

As expected, some of the choices and justifications seemed plausible and appropriate while others seemed, at best, incomplete.

An expert panel well-versed in moral psychology and adolescent development confirmed our perceptions. The use of experts allowed us to develop and justify a key from which we could judge whether the participant applied the virtues using choices consistent with an established viewpoint.

Armed with our finished measure, we decided to assess the participants’ ability to identify best and worst choices and justifications. Respondents achieved high scores if they were able to identify items in much the same way as the experts. However, if the participants selected “bad” items as best and “good” items as worst, their scores declined.

In this way we were able to identify adolescents who had a good grasp of the targeted virtue concepts (at least insofar as they reflect the prevailing norms) and those who did not.

More interestingly, we were able to tease out which specific judgments were more or less difficult for adolescents and for whom. That is, we could address whether adolescents find it easier to identify appropriate choices as compared to good justifications, or whether they find it easier to identify good choices and justifications rather than worse ones, and whether these patterns change across the adolescent period.

Compared with other objective measurement systems, the range of information generated by a story approach provides a more detailed picture of how adolescents apply the virtues and offers the potential for more informed educational curricula and evaluations.

 

Steve Thoma

Professor of Moral Psychology & Psychometrics at the Jubilee Centre

Advertisements