A Probabilistic Approach to Card Sort Analysis

Date
2018-02
Authors
Bin Amer, Hadeel Hatim
Journal Title
Journal ISSN
Volume Title
Publisher
Faculty of Graduate Studies and Research, University of Regina
Abstract

The work in this dissertation was motivated by the desire to more fully understand the results of a card sorting study of facial photographs. Twenty-five participants were asked to sort 356 facial photographs (178 of Caucasian males and 178 of First Nations’ males) into an unconstrained number of piles according to their judgments of similarity. Researchers have used card sorting to understand different concepts, but not facial similarity. Therefore, the work presented in the dissertation is novel because it takes an existing method and applies it to a new context and adapts analysis tools to this purpose. Pairs of photos in the same pile were deemed to be judged “similar” and those in different piles were deemed to be judged “dissimilar”. There are 63,190 pairs possible from 356 photos. It is clear that participants did not, nor could not, make all these pairwise comparisons directly. The study was executed, analyzed, and described by other researchers at the University of Regina, but there remain unresolved questions from the data that it produced. Some participants made very few piles which others made very many: should the information provided by each participant be treated equally? Is there enough information in the sorting of the cards to uncover how participants have judged facial similarity, with different participants using possibly different strategies? Perhaps it would be more productive to work with a smaller number of photos, but how should these photos be selected and what number of photos is neither “too many” nor “too few”? If the majority of participants agree that a photo pair is similar or dissimilar, that pair may not help to discern different strategies that participants may be using. Therefore, it is possible to consider the data in terms of a three-way decision: if 16 or more of the 25 participants judged a pair to be similar, the pair is labelled as Similar. If 9 or fewer of the 25 participants judged a pair to be similar, the pair is labelled as Dissimilar. Finally, if between 10 and 15 participants judged the pair to be similar, the pair is labelled as Undecided. In order to explore the questions about information quality, the probability of each pair of photos was calculated, modelled as the first two photos drawn from a deck without replacement. If a participant placed the pair with many other photos, either together in one pile or apart in two piles, the probability of that pair was large. Alternatively, the probability of the pair was small if it was placed with few other photos. The probability of the pair is hypothesized to be an indicator of confidence: a small probability means high confidence and a large probability means low confidence. If the Undecided group contains photo pairs about which some participants are very confident of similarity and others are very confident of dissimilarity, these pairs may be most useful to study further. The eigenface method of evaluating facial similarity was used to provide a reference for the judgements made in the card sorting study. The Euclidean distance between each pair projected photos was used as a surrogate for the similarity judgements: pairs with small intra-pair distances were deemed to be judged “similar” and pairs with large intra-pair distances were deemed to be judged “dissimilar”. A second study was conducted, based the analysis of the first, that asked forty-three participants to rate the similarity of selected photo pairs on a scale from 0 (Similar) to 100 (Dissimilar), with the midpoint representing Undecided. Analysis of the second, pairwise, study found agreement with the first, card-sorting, study. The application of the 3-way decision framework also proved valuable. In conclusion, the work has provided a means, previously unavailable, to understand the importance of various photo pairs in participants’ judgment of similarity.

Description
A Thesis Submitted to the Faculty of Graduate Studies and Research In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Computer Science, University of Regina. xi, 133 p.
Keywords
Citation