Methodology

Sampling

The authors conducted a usability study with 40 BVI subjects in the United States, mainly in Wisconsin, featuring two separate groups of 20 BVI users each. Subjects at least 18-years of age were recruited from the Midwest. These subjects had as a minimum three years of experience in using a search engine with a (SR) and had basic information search skills to retrieve information from a DL. Purposive, convenience, and snowball strategies were applied for recruitment. To guarantee success in recruitment, the researchers worked closely with BVI-serving support organizations to recruit subjects. A pre-questionnaire was used to prescreen all subjects, ensuring both groups were comprised of similar demographic characteristics and information search skills. Forty subjects were divided into two groups with similar demographic data, system knowledge, and screen reader familiarity based on the data generated from the pre-questionnaires. The similarities of subjects’ demographic characteristics between the control and experimental groups enabled researchers to compare the differences in usability of the DLs and their associated help features. Each subject was paid $75 as an incentive for participation in the study.

Data collection 

The control group (n = 20) used the real digital collection equipped with existing help features, and the experimental group (n = 20) worked on the testing digital collection equipped with new help features. Subjects conducted three search tasks that BVI users typically perform, which represent known item search, specific information search, and exploratory search. These tasks help to reveal different types of help-seeking situations BVI users experience in their DL interactions. Multiple data collection methods including pre-questionnaires, think-aloud protocols, transaction logs, and pre and post search interviews, were employed in the experimental design. Pre-questionnaires were utilized to solicit subjects’ demographic information, search and system knowledge, and assistive technology use. Pre-search interviews were conducted to obtain their levels of subject knowledge for each assigned task. Laptops with JAWS Screen Reader and Morae software were used for this study as JAWS is the most popular SR in the BVI community, and Morae software captures participant verbalization, screen shots, and transaction logs. Think-aloud protocols and transaction logs recorded subjects’ behaviors and their thoughts behind these behaviors, including the help-seeking situations they encountered, the DL features used (or did not use), and user feedback on these features during the search process. Finally, post-search interviews solicited additional information regarding subjects’ perceptions of ease of use of the DL, the usefulness of the help features, and satisfaction levels with the DL.

Data analysis

The researchers applied both quantitative and qualitative approaches to examine the proposed research questions and corresponding hypotheses. First, types of help-seeking situations were analyzed based on the coding scheme developed by the authors’ previous work on help-seeking situations (Xie et al., 2018). The researchers performed content analysis to code and calculate frequency of each of the five types of help-seeking situations. Table 4 illustrates the five types of help-seeking situations and their definitions. Two independent coders analyzed these situations, selecting a random sample from the 40 subjects. Based on Holsti’s (1969) formula, the inter-coder reliability of identified types of help-seeking situations between the two coders was 91.9%. The coders calculated the frequency of each type of help-seeking situation, and further applied these to the statistical analysis for hypotheses testing associated with research questions 1–4.

Findings

In an overall comparison of frequency of nine help-seeking situations, key differences were observed between the two groups for confusion about multiple programs and structures and difficulty with help. There was a reduction in situations related to confusion about multiple programs and structure in the experimental group. The control group was more confused about the overall structure of pages, search results, and content layout. Help features that may have reduced these situations for the experimental group included: 1) an introductory task to visit the help page which outlined the overall structure, 2) individual descriptions about the page contents and structure, or 3) context-sensitive help with features. The experimental group experienced more situations related to difficulty with help. It is possible that the added instruction with the search limiters encouraged the experimental group to use the search limiters, but the feature itself was extremely difficult to use and navigate. For this situation, the experimental group also experienced difficulties recognizing the meaning of labels and features that were not modified from the original site. Mann-Whitney U tests were conducted to compare the time spent for task completion between the control group and the experimental group. There was no significant difference in mean time between the two groups in completing all search tasks. This is likely because time limits were placed on each task. Actual differences in time may be better understood if time was not restricted.

Differences were not significant for perceive ease of use or perceived satisfaction. However, descriptive data showed that the experimental group found the DL to be easier to use and were more satisfied. Although differences are not significant, users reported what made them satisfied and dissatisfied with using the DL. The main reasons for high satisfaction level in the control group included coverage of the DL along with browse options and navigation. The experimental group reported high satisfaction due to the availability of multiple help features, clear headings, DL coverage, clear labels, and easy navigation. Main reasons for low satisfaction levels in the control group included inaccessible content or no alternative text, unclear labels/description, and a complicated structure, whereas main reasons for low satisfaction level in the experimental group included difficulty accessing information, no or irrelevant results, and multimedia problems. There is a significant difference between the controlled group (M=3.63, SD=1.18) and the experimental group (M=4.61, SD=0.69) in the perceived helpfulness levels of system help; t(18)=-2.275, p=0.035. This indicates that the experimental group perceived a higher level of helpfulness of system help than the control group. Specifically, the experimental group reporting higher helpfulness levels for nearly all system features.

The experimental group was asked about the helpfulness of new features according to three categories: navigation, description, and search display options. Users found it easier to skip to content or results, browse, and appreciated the clear headings. Experimental subjects who encountered problems with using new features expressed time limitations or difficulty in understanding how to use features. Subjects who did not use the new features indicated that they were unclear about the features, they found them difficult to use (specifically search limiters), or experienced time limitations and would have explored them if given more time. Users in the experimental group were asked to identify their top features. Browse, Skip buttons or links, and headings were the top features preferred by users.

Design

Our results show the feasibility of designing DL features that serve the unique needs of BVI users for specific IR tasks. It proves that a careful examination of the DL interaction experiences of BVI users affords a deep understanding of their help-seeking situations in their information search process. Analysis of these situations reveals what system help is needed to retrieve information by listening to SR renditions. This confirms the findings of previous research which reveals that many situations BVI users face in using the web are due to poor usability, creating obstacles during their IR system interactions (Power et al., 2012; Rømen & Svanæs, 2012; Vigo & Harper, 2013, 2014). In turn, this research allows the authors to define the function and placement of help features offering description, instruction, navigation, format, search function and multimedia required to address the need.

Our analysis uncovered several examples:

Menu bar (Feature: Navigation; Situation: Difficulty with help): To launch their DL interaction, BVI users need immediate help about how to proceed. Considering this need, we designed a menu bar with the Help Menu as its first item and placed this menu bar at the top of the page (the location where the screen-reader focus typically lands after arriving on a new web page).Labels (Feature: Description; Situation: Multiple situations): It is noted that to recognize the available functions of the DL, BVI users rely on the meaningfulness of labels for individual menu items. Accordingly, when assigning labels to the items in the top menu bar, we made sure each label unambiguously described the purpose of the relevant menu item.

Transcript (Feature: Description/Format; Situation: Difficulty accessing information): To access information embedded in documents, including images, BVI users need the capability to transcribe the embedded content into descriptive text. Accordingly, the transcription functionality of the content management system was enabled, thereby offering a Transcript feature in the DL. Search results (Feature: Navigation; Situations: Difficulty locating information and difficulty evaluating information): To tell what the outcome of a search query was, BVI users need direct access to the text describing its status (e.g., whether search results were generated or altered). Correspondingly, two features were created. First, a title was added to the search results section describing the status of the executed search. Second, a heading marker was added to this title to enable section header navigation.

Item (Feature: Navigation/Description; Situation: Difficulty locating information): To determine if a searched item is immediately available or reachable, BVI users need step-by-step directions to locate that item in the DL. Consequently, three features were incorporated into the DL. First, a breadcrumb was added, indicating the path to the searched item. Second, meaningful labels were assigned to each breadcrumb element. Finally, a hyperlink was embedded in each breadcrumb element to lead to the destination page.

Most important, the results demonstrate that users encounter fewer difficulties when interacting with a DL equipped with help 11 features than one whose features do not take into account their specific needs. The findings strongly support this finding in two types of help-seeking situations: (1) accessing DL information, and (2) using system help. BVI participants who interacted with the experimental DL interface reported significantly fewer help-seeking situations when attempting to complete tasks relevant to in- formation access and help use compared to those who used the baseline DL. At the same time, the authors observed a positive difference in the frequencies of the three types of help-seeking situations between the experimental and controlled conditions (e.g., evaluating information, locating specific information, items, or features, and constructing or refining search statements). Ultimately, these differences were not significant, pointing to possible inadequacies in the design of features to address the help needs identified and confirming that the authors need to return to the drawing board and redesign the features after analyzing qualitative feedback provided by BVI subjects. Nevertheless, the results of our analysis establish that DL usability is positively impacted by the degree of alignment between help features and help needs of this vulnerable population.

Interestingly, this study also identified a crucial problem in designing DLs for universal use. Just as previous research has pointed out that linear nature of SR navigation and information architecture elicit problems for BVI users in navigating web interfaces (Borodin et al., 2010; Brophy & Craven, 2007; Lazar et al., 2007; Vigo & Harper, 2014), this problem also exists in DL design and needs to be addressed. Nevertheless, some of the newly added features that benefit BVI users might cause confusion for sighted users. For example, BVI users have to rely on a SR to navigate the interface. Before they can access the search results, they may have to listen to additional irrelevant information. In order to help BVI users quickly locate the first search result, a feature labeled as “Skip to first search result” was incorporated into the interface design. However, sighted users do not need this feature since they can quickly identify the location of the search results, and this new feature may cause added confusion. For this reason, the research team decided to make this feature visible to SRs, but invisible for sighted users. The same design principles also apply to “Skip to event details” and “Return to thumbnail” for the DL timeline browser. The invisible feature design demonstrates an effective approach for DL design to satisfy the needs of both sighted and BVI users.

To truly promote universal access to information, DL designers should endeavor to remove barriers that prevent users from accessing information, rather than expect those with impairments to adapt their behaviors to overcome such barriers (Brebner & Parkinson, 2006). At the same time, we continue to work on understanding the barriers these users face as we strive towards a more inclusive design and equitable society.

Publications

Xie, I., Babu,  R., Lee, T., Castillo, M., You, S. & Hanlon, A. (2020). Enhancing usability of digital libraries: Designing help features to support blind and visually impaired users. Information Processing and Management, 57(3), https://doi.org/10.1016/j.ipm.2019.102110

Conference

Xie, I. (2017). Keynote speaker. Support Accessibility, Usability, and Utility: An investigation of blind and visually impaired users’ interaction with digital Libraries. Presented at iConference Workshop 2017: Vulnerable Communities in the Digital Age: Advancing Research and Exploring Collaborations, Wuhan, China.

Xie, I., Babu, R., Castillo, M., Lee, TH., You, S. & Hanlon, A. (2017). Designing Help Features to Support Blind Users’ Interaction with Digital Libraries. Presented at ALISE 2017, Atlanta, GA.

Team

Principal investigators

1. Iris Xie, PI, Professor, UWM-SOIS, hiris@uwm.edu
2. Rakesh Babu, Co-PI, PhD, Assistant Professor, UWM-SOIS, babu@uwm.edu

Research assistants

1. Melissa Davey Castillo, PhD student, UWM-SOIS
2. Tae Hee Lee, PhD student, UWM-SOIS
3. Sukwon Lee, PhD student, UWM-SOIS
4. Sukjin You, PhD student, UWM-SOIS

Consultant for OCLC/ALISE project

Ann Hanlon, UWM Libraries, Digital librarian University of Wisconsin – Milwaukee