We will use the example of our user study to illustrate how to recruit participants, how to collect data, and how to analyze data. We designed the accessibility and usability study for 64 BVI users across the country. The study was approved by the IRB; any study with human participants requires IRB approval.
Sixty-four participants were recruited representing BVI users across the US with different characteristics, e.g., sex, age, and race. Diversity is the key to recruitment. Since this is a special, targeted population, some conventional recruiting methods, such as flyers, do not work well. Two groups of users were studied: the onsite group and the diary group. For the onsite group (32 participants), a majority of participants were recruited at the NFB convention, and the remaining participants were recruited through local blind associations. For the diary group (32 participants), participants were recruited through local blind associations and advisory board members. It is more effective to recruit participants through organizations or units that BVI users trust and have some connections. Requirements necessitated that participants must: (a) use a screen reader to access the Internet, (b) have at least three years of experience in using the Internet, and (c) be 18 years and older. Potential participants were prescreened via a prescreening form. You can define your own qualifying criteria based on the purpose of your user study. Each participant received a $100 gift card as an incentive for completing the study.
For the onsite group, multiple data collection methods were applied to explore BVI users’ help-seeking situations: pre-questionnaires, think-aloud protocols, transaction logs, and post-questionnaires. Pre-questionnaires were used to solicit demographic information. A popular screen reader (JAWS) was used, and Morae software captured participant verbalization, screenshots, and transaction logs. Think-aloud protocols provided information about BVI users’ perception of problems in interacting with the DLs and desired features. Transaction logs recorded the unique help-seeking patterns of BVI users. In post-questionnaires, participants rated their perceptions of the helpfulness of DL features in the search process. If your participants are local, conducting an on-site study allows you to engage in in-person observation more easily. You can design your data collection instruments based on the types of data that you need to improve the design of your digital library.
For the diary group, multiple data collection methods were also applied to explore BVI users’ help-seeking situations: pre-questionnaires, diaries, and post-questionnaires. Pre-questionnaires were used to solicit demographic information. Participants were allowed to choose their own preferred screen reader, and diaries captured participants’ statements about their experiences, any problems they encountered, associated factors in interacting with the DLs, and desired features. In post-questionnaires, participants rated their perceptions of the helpfulness of DL features in the search process.
For both groups of users, all participants searched Library of Congress Digital Collections. At the same time, four groups of eight participants interacted with each of the following four digital libraries: Digital Public Library of America, The New York Botanical Garden Mertz Library, HathiTrust, and Artstor. These DLs represent different types of content as well as design. In your user study, participants should use your own digital library.
Data were analyzed qualitatively and quantitatively. First, qualitative data were analyzed using open coding, which is the process of breaking down, examining, comparing, conceptualizing, and categorizing unstructured data. Then, taxonomies of help-seeking situations were specified and categorized into help-seeking situations unique to BVI users at the physical level and the cognitive level. Based on this open coding, the team identified about 40 types of help-seeking situations that BVI users encountered during the search process. Quantitatively, correlation tests were conducted to identify the specific types of factors that led to the specific types of help-seeking situations. After that, the researchers developed digital library accessibility and usability guidelines grounded in the help-seeking situations that BVI users faced in their interactions with digital libraries. For your own study, you will need to analyze the data based on your own research questions and associated hypotheses, if any.
You can find the four transcripts of a user study in Appendix I-b.