Methodology

Sampling 

Thirty blind and visually impaired participants were recruited throughout the United States by distributing the recruitment flyer to the National Federation of the Blind (NFB) listserv. Prior to joining the study, potential participants received a brief pre-screening questionnaire and an informed consent form. Participants needed to meet the following requirements: (1) using iPhone 6S (or newer) with iOS 11 (or later); (2) using iPhone non-visually by listening to VoiceOver; (3) having at least three years of experience searching for information on the Internet via iPhone; (4) feeling comfortable verbalizing thoughts in English; and (5) being willing to install the Microsoft Teams software and the Library of Congress Digital Collections app. iPhone users were chosen because iOS devices (Apple iPhone, iPad, or iPod) are the most widely used mobile devices (71.9%) by people with disabilities (WebAIM, 2021). The Library of Congress Digital Collections were selected for this study since it is one of a few digital libraries that provides both mobile website and mobile app. As a national level digital library, the Library of Congress Digital Collections consist of multiple digital collections covering a wide range of topics of interest to blind and visually impaired users. Participants were asked to use Safari to access the mobile website because it is the mobile browser most commonly used by blind and visually impaired users (WebAIM, 2021). Thirty participants were recruited to use both the mobile app and the mobile website. Required materials and forms about the study were submitted to the Institutional Review Board of a state university, and ethical approval was obtained. Each participant received a $100 electronic gift card as an incentive for participating in the study.

Data collection

Multiple data collection methods were used, including pre-search interviews, think-aloud protocols and transaction logs, post-platform questionnaires, post-platform interviews, and post-search interviews. Since this is a within-subjects design, following the Latin Square design, the first participant (P1) was assigned to use the mobile app first, and the second participant (P2) was assigned to the mobile website first. The rest of the participants were assigned to either platform first accordingly. For each platform, participants performed one orientation task, one specific search task, and one subject search task. After successfully installing the required app, each participant spent around 3.5 hours completing all of the study activities. Microsoft Teams was employed to collect data. Think-aloud protocols and transaction logs were applied to capture participants’ thoughts and movements when working on the tasks. All recorded audio and video files were later transcribed verbatim for further analysis. For each platform, a questionnaire was employed to help researchers to measure participants’ perceptions of each platform’ accessibility and usability as well as their satisfaction towards the platform by using a 7-point Likert scale with 1 indicating not at all and 7 extremely. For example, participants rated their perceptions, such as “On a scale of 1 to 7, how would you rate the ease of access to the features of the DL when using the mobile app?”  Different types of interviews, including post-platform interviews and post-search interviews, allowed the researchers to record participants’ responses related to research questions. During Post-platform interviews, participants provided the reasons for their rating of accessibility and usability of each platform as well as suggestions for how to enhance the digital library in mobile platforms. For example, participants were asked, “What made the mobile site of digital library easy or difficult to learn?” and “What made the digital library mobile app easy or difficult to use?” In the post-search interviews, participants provided their assessment regarding the comparison of the two platforms of the digital library and their final thoughts.

Data analysis

There are two steps of analysis. First, quantitative analysis was applied to examine null hypotheses associated with research questions. Quantitative data from the questionnaires were used for quantitative analysis. Before performing a T-test, we checked the normality of the Likert scale data collected from the questionnaires. Specifically, skewness was used to check for normality of the data related to accessing a digital library, accessing a navigation component, accessing a feature, accessing information/objects, ease of learning, ease of use, efficiency, and satisfaction of the two digital library platforms. A value for skewness between -2 and +2 is considered to be normally distributed. The collected data had a skewness value between -2 and +2 indicating normal distribution, so a T-test was applied to test the null hypotheses associated with the two research questions. After statistical analysis, interview transcripts, think-aloud protocols, and transaction logs were analysed as supporting evidence for the similarities and differences of variables between the mobile app and the mobile website. Qualitative data were examined to identify types of design factors corresponding to the two research questions. Open coding, which is the process of analyzing and categorizing the unstructured textual transcripts into meaningful concepts, was utilized for the analysis of design factors. Nine types of design factors were identified. Two coders coded the data independently, and the inter-coder reliability was 0.94 based on Holsti’s (1969) method. The codes were discussed within the research team until an agreement was reached, and disagreements or questions were resolved by group discussions to ensure the reliability of data analysis.

 Findings

As for accessibility (Research question 1), the results showed that there were significant differences between the mobile app and the mobile website for all perceived variable levels except accessing information/objects.

H₀1:1 – 3: An independent-samples t-test was applied to compare the perceived levels of accessing a digital library, accessing a navigation component, and accessing a feature between the mobile app and the mobile website. The perceived level of accessing a digital library for the mobile app (M = 6.53, SD = .90) was significantly higher than the mobile website (M = 5.4, SD = 1.65); t(58) = 3.30, p < .05. The perceived level of accessing a navigation component for the mobile app (M = 5.90, SD = 1.32) was significantly higher than the mobile website (M = 4.43, SD = 1.72); t(58) = 3.71, p < .05.  The perceived level of accessing a feature for the mobile app (M = 6.00, SD = 1.17) was significantly higher than the mobile website (M = 4.4, SD = 1.43); t(58) = 4.74, p < .05. Therefore, null hypotheses H₀1:1 – 3 were rejected.

H₀1:4: An independent-samples t-test was applied to compare the perceived levels of accessing information/objectsbetween the mobile app and the mobile website. The perceived level of accessing information/objects for the mobile app (M = 4.23, SD = 2.14) was not significantly higher than the mobile website (M = 3.73, SD = 1.86); t(58) = .97, p > .05, so null Hypothesis H₀1:4 was accepted.

The design factors identified through open coding analysis are presented with associated examples. For each data type, TA, PS, and PI stand for think-aloud protocols and transaction logs, post search interview, and platform interview data, respectively.

Accessing a feature shows the largest mean difference between the mobile app and the mobile website. The design factors notably associated with the different perceptions on accessing features are adequacy of a feature label and stability of keyboard focus. Additionally, the results showed that shifting the input cursor influenced accessing features in the mobile website.

Following accessing a feature, accessing a navigation component shows the second-largest mean difference between the mobile app and the mobile website. For this variable, logic of structure and simplicity of layout and availability of navigation shortcut were the design factors behind their perceptions of the two platforms. Furthermore, the participants complained that the mobile website lacked shortcuts requiring them to make more efforts in navigating within a digital library page.

Accessing information/objects shows the lowest mean difference between the mobile app and the mobile website. The participants encountered difficulty accessing information/objects such as photos, manuscripts, and newspapers in accomplishing their tasks for both the mobile app and the mobile website. Adequacy of alt text and adequacy of item metadata and collection description were mentioned in the participants’ explanations. Furthermore, participants noted the absence of a meaningful title or descriptions needed for understanding the presented item.

As to usability (Research question 2), the results showed there were significant differences in perceived levels of all variables between the mobile app and the mobile website.

H₀2:1 – 4: An independent-samples T-test was applied to compare the perceived level of ease of learning between the mobile app and the mobile website. The perceived level of ease of learning for the mobile app (M = 6.40, SD = .62) was significantly higher than the mobile website (M = 4.83, SD = 1.72); t(58) = 4.68, p < .05. The perceived level of ease of use for the mobile app (M = 5.87, SD = 1.36) was significantly higher than the mobile website (M = 3.90, SD = 1.67);t(58) = 5.01, p < .05. The perceived level of efficiency for the mobile app (M = 5.13, SD = 1.38) was significantly higher than the mobile website (M = 3.07, SD = 1.84); t(58) = 4.92, p < .05. The perceived level of satisfaction for the mobile app (M = 4.73, SD = 1.62) was significantly higher than the mobile website (M = 3.60, SD = 1.99); t(58) = 2.42, p < .05. Therefore, null hypotheses H₀2:1 – 4 were rejected.

Efficiency shows the largest mean difference between the mobile app and the mobile website in fulfilling their tasks. The participants mentioned that logic of structure and simplicity of layout and logical location of a feature affected their ratings for perceived efficiency. The structure of the mobile website was not clear for her to locate the wanted information, which hampered her search efficiency in the digital library. In particular, some participants emphasized that the location of a feature hindered efficiency while using the mobile website.

Ease of use shows the second-largest mean difference among usability variables. Logic of structure and simplicity of layout, availability of contextual help information, and availability of navigation shortcut were identified as design factors causing the different ratings for ease of use between the mobile app and the mobile website. Similar to other variables, the digital library structure and the layout played a role in participants’ perceived ease of use.

Furthermore, perceived ease of use was related to the existence of contextual help information.In addition, availability of navigation shortcut impacted blind and visually impaired users’ ratings for ease of use.

Satisfaction shows the lowest mean difference between the mobile app and the mobile website. Logic of structure and simplicity of layout explains the difference in the ratings for the two platforms, while adequacy of alt text illustrates the low mean gap between the two. In contrast, the insufficient alt text was a negative factor when explaining their perceived satisfaction with both platforms.

Design

Among all the design factors, logic of structure and simplicity of layout, availability of navigation shortcut, and adequacy of alt text influenced both accessibility and usability of the mobile app and the mobile website. This section concentrates on the design implications for the mobile website because it has many more problems than the mobile app of the digital library. Logic of structure and simplicity of layout is the design factor affecting five of the accessibility and usability variables. For blind and visually impaired users, a shallower and wider navigational structure is the best option. Considering the complex design of a mobile website, it would be helpful to first provide an overview of the digital library structure and give blind and visually impaired users an idea of the digital library’s organization, such as options for finding and accessing relevant collections/items as well as how to use Help. Second, an overview of search results is critical for these users to understand the structure of search results. Possible structures include 1) two levels of search results consisting of collection-level and item-level, 2) only item-level, 3) or mixed results. Moreover, these overviews should be able to turn on and off so that experienced blind and visually impaired users and sighted users can skip that option.

Third, it would be helpful for a mobile website to present the main functions of a digital library that users most frequently use rather than a list of the collections in an alphabetical list. Examples of recommended functions include Search digital collections, Browse digital collections, About the digital library, etc. Fourth, the digital library layout must be adaptable without losing structure and information, ensuring that blind and visually impaired users can perform two-dimensional navigation. As to availability of navigation shortcut, creating shortcuts, such as assigning headings and/or landmarks, is an effective approach for blind and visually impaired users to jump from one page/section to another. Simultaneously, assigning more headings and/or landmarks, in particular in the results section, enables them to orient themselves to a page and to navigate easily to various sections of a page. In terms of adequacy of alt text, alt text is needed for visual items, and more importantly, descriptive alt text helps blind and visually impaired users differentiate one collection/item from another. As mobile devices hide some content because of the screen size, it is difficult for them to select a collection/item from multiple collections/items with similar titles. Furthermore, alt text needs to provide both content and format of a collection/item.

Team

Principal investigators

1. Iris Xie, PI, Professor, UWM-SOIS, hiris@uwm.edu

Research assistants

1. Tae Hee Lee, PhD student, UWM-SOIS
2. Shengang Wang, PhD student, UWM-SOIS
3. Hyun Seung Lee, PhD student, UWM-SOIS

Consultant for DIG project

1. Rakesh Babu, Co-PI, PhD, Envision, Rakesh.babu@envisionus.com