Project stage

This section presents the three-year project work plan, including document analysis, user study, guidelines creation and testing, and results dissemination. IRB approval will be secured at the start of the project. In order to address the research questions proposed above, the proposed project consists of four stages.

Stage 1 Build a foundation for mDLAUG (08/01/22–01/31/23, Research Questions 1, 2 & 3)

Identify unique help-seeking situations in mobile interactions by document analysis. A thorough and comprehensive literature search for the last 20 years and document analysis will be conducted to identify a list of help-seeking situations that BVI users encounter in their mobile interactions since very little research has been conducted in the DL environment. Both physical and cognitive levels of situations will be analyzed. These findings will enable the team to create a draft of design guidelines that address the variety of help-seeking situations that BVI users may experience during their DL interactions in using mobile devices. The open coding technique, which is the process of breaking down, examining, comparing, conceptualizing, and categorizing unstructured textual transcripts, will be utilized. The document analysis will build on the literature review conducted in LIS and other related fields during the pilot study described in Project Justification. Four types of categories will be generated in the mobile environment: unique needs of BVI users, their help-seeking situations, factors leading to the situations, and their desired help needs. Findings will be verified and enhanced by data gathered from the user study in Stage 2.

Survey existing guidelines and papers by document analysis. Two types of analyses will be conducted to identify the current status of and problems with the existing guidelines as they relate to accessibility and usability in mobile contexts: 1) analysis of existing accessibility and usability guidelines and 2) analysis of associated research on the guidelines. A comprehensive search will be conducted to identify the existing guidelines and associated papers through web search engines and major online databases for the last 20 years. The scope of these guidelines will cover both US and international territory. The inclusion criteria are: 1) guidelines, standards, or policies that are related to accessibility and usability for information systems, Web pages, software, etc., and 2) papers that address the coverage, components, structure, problems or future directions of guidelines, standards, best practices and policies. Next, the team will describe various aspects and components of the guidelines in detail, conduct content analysis of guidelines that provides an overview on the coverage, components, and structure of the existing guidelines, and identify types of problems and future directions. Finally, BVI users’ help-seeking situations and needs identified in the literature will be compared to the document analysis of the guidelines to help inform the user study and guidelines development in Stage 2. These two types of document analyses will complement each other, providing a comprehensive overview on how, where, and why existing guidelines fail to address the help-seeking situations and needs of BVI users of DLs in the mobile environment, with the primary goal of identifying the types of guidelines needed to fulfill these needs.

Stage 2 Develop a draft of DL guidelines (02/01/23-04/30/24, Research Questions 2 & 3)

Recruit participants and prepare IRB. Participant recruitment will occur for three different communities participating in various stages of the DL design guidelines development. First, 120 BVI participants will be recruited for a user study in DL mobile contexts in order to identify help-seeking situations. Second, 150 participants will be recruited for two-round Delphi surveys to provide feedback for the draft of guidelines. Third, 30 DL developers will be recruited for the assessment of 6 DLs and participation in focus groups to identify the challenges and associated solutions in implementing and adopting mDLAUG.

Since the BVI population comprises a low-incidence user population, recruiting a sufficient number of participants can pose unique challenges. To ensure success in recruitment, participants will be recruited from BVI partner organizations (see Partnercommitment.pdf), Participants will also be recruited from mailing lists, such as NFB Net, Blind geek zone, The New Blind Tech, Blind Cool Tech, Blind Webbers, Blind World, and NOBE (National Organization of Blind Educators). Additionally, the consultants and advisory board will also make recommendations for and help with participant recruitment. In addition, they will be recruited at different BVI conventions. One hundred and twenty participants will be recruited representing BVI users across the US with different characteristics. Diversity is the key to recruitment. Each participant will receive $100 as an incentive for completing the study. Participants must: (a) use a screen reader to access the Internet, (b) have at least three years of experience in using mobile devices to search for information, and (c) be 18 years and older. Potential participants will be prescreened via a pre-questionnaire.

To reflect various opinions of heterogeneous stakeholders involved in the research, development, and use of DLs, a variety of stakeholders will participate in two-round Delphi surveys. 150 participants will be recruited to represent the following three groups: scholars/experts, digital library developers, and BVI. The number and the selection criteria of each group are specified as: Scholars/experts (N=50): Scholars (N=25): scholars who have conducted research on accessibility and usability with high citations. Experts (N=25) are people who perform accessibility and usability tests for BVI users. Experts will be recruited through organizations and units, such as the American Foundation for the Blind (AFB) Tech Lab, Microsoft Accessibility unit, Google Accessibility lab, and The Trace Research and Development Center. They will also be recruited at conventions, including the CSUN International Technology & Persons with Disabilities Conference, and the International Conference on Computers Helping People with Special Needs. Digital library Developers (N=50): Digital librarians who have been in charge of DL development (interface, content, metadata, etc.) for several years. Participants will be recruited through DL partners and advisory board members. Additionally, recruiting messages will be sent to related listservs (e.g., Diglib, Imagelib, CONTENTdm-L). End users (N=50): BVI users. All participants will be instructed to fill in pre-questionnaires, consisting of three parts: 1) demographics information; 2) experience in researching, developing, or using DLs; and 3) assessment of the draft of the mDLAUG.

Thirty of the 50 DL developers who participated in the Delphi surveys will be recruited to participate in the DL assessment and guidelines feasibility assessment including focus groups. The reasons for the selection of this group are: 1) they are the implementers of the guidelines for the design and improvement of DLs; and, 2) they offer feedback for the draft guidelines, and are familiar with guidelines. After completing the assessment, these members will participate in focus groups to discuss their experiences in applying the guidelines to DLs. They will each be compensated $200 for their time commitment and input. An IRB Protocol form including all instruments, consent form, and recruitment flyer, will be prepared and submitted to University of Wisconsin-Milwaukee Institutional Review Board (IRB) board for approval.

Identify help-seeking situations through a user study. In Stage 2, the team will develop an understanding of various help-seeking situations BVI users encounter in DLs in using various mobile devices when performing specific search tasks, and the types of help needed to resolve these situations in mobile contexts. In addition, the team will investigate the factors that lead to the help-seeking situations. By identifying unique help-seeking situations and factors, the team will uncover the specific needs of BVI users in order to prepare for the development of DL design guidelines in the mobile environment.

To be selected, a DL must include a wide variety of content and media formats in which BVI users might be interested. Efforts will be made to choose a diverse set of DLs for the study, such as a stand-alone DL with multiple digital collections in both app and web platforms, federated DLs, visual digital libraries, DLs offers STEM information, etc. To identify diverse types of interactions, a combination of two types of search tasks (one assigned and one self-generated) will be employed in the study, including specific information search and subject-oriented search. In a specific information search, a user looks for exact data or facts. In a subject-oriented search, a user looks for items with common characteristics. The participants will conduct each search task in two different DLs, and the 12 representative DLs, including M. Web and M. App of DLs, will be selected for the user study. These tasks will help investigate the different types of help-seeking situations BVI users experience in mobile contexts when accomplishing search tasks.

One hundred and twenty BVI subjects will be divided into four groups (iOS phone, Android phone, iOS tablet, and Android tablet) with 30 subjects in each group. Multiple data collection methods will be applied to explore BVI users’ help-seeking situations: pre-questionnaires, pre-search interviews, think-aloud protocols, transaction logs, and post-search interviews. Pre-questionnaires and pre-search interviews will be used to solicit demographic information. Mobile usability testing software, which captures participant verbalization, screenshots, and transaction logs, will be used for this study. Think-aloud protocols will provide detailed information about BVI users’ perception of their help-seeking problems and desired features. Transaction logs will show the unique help-seeking patterns of BVI users. In post-search interviews, the team will ask participants to identify: typical problems in fulfilling the search tasks, factors leading to the problems, interface features used, desired features, etc. The pilot study demonstrates that these data collection methods work for the BVI participants. The team will modify the pilot study instruments (Supportingdoc1.pdf) for the proposed project.

Data will be analyzed both qualitatively and quantitatively. Based on open coding, the team will identify types of help-seeking situations that BVI users encounter during the search process and associated factors as well as desired help needs. First, qualitative data will be analyzed by using open coding, specified in stage 1. Taxonomies of help-seeking situations will be specified and categorized into help-seeking situations unique to BVI users at the physical level and the cognitive level. Second, open coding will also be applied to specify associated help needs for each type of help-seeking situation. Third, the comparison of the frequency of help-seeking situations in the four groups will be analyzed by applying ANOVA or the Kruskal–Wallis test depending on the data distribution. Fourth, relationships between factors (user, system design, task, interaction outcomes, etc.) and different types of help-seeking situations in diverse groups will be analyzed by applying multivariate analysis and/or multiple regressions.

Develop the draft of guidelines. Based on Stages 1.1, 1.2, and 2.2, a draft of mDLAUG will be developed based on the DLAUG structure developed by the team and WCAG structure. The document analysis including a review of WCAG and other guidelines that will be reviewed against BVI users’ help-seeking situations and challenges in the mobile environment. Similar to the analysis in Stage 1.2, design guidelines will be organized by types of help-seeking situations associated with accessibility and usability, and consist of Definition of the type of help-seeking situation, Factors that lead to the situation, Guideline or Design Recommendations to solve the situation, Rationale and Objective that provide a set of reasons behind the creation of guidelines, Techniques and Methods to comply with a specific DL design guideline, Recommended Features to implement a specific technique or method, Examples that show good and/or bad designs for a recommended feature, Related Resources that provide extra resources corresponding to the situation, and See Also to link to the corresponding category or categories of situations that the current situation belongs to. In addition, a Glossarywas created to present the definitions of the key terms used therein. The following Appendix will be provided: How to conduct a user study, Types of factors, Level of conformance recommendation, Similarities and differences between the mDLAUG and WCAG, Keyboard shortcuts, and Accessibility assessment tools.

In order to develop the draft of guidelines, the research team will analyze and re-analyze the results of the user study using open coding and compare findings to the existing guidelines. The key DL design guidelines will be identified in association with the two overarching areas of accessibility and usability. Each of the help-seeking situations of BVI users will be grouped together. Definition of each type of situation will be presented. Simultaneously, types of factors that lead to the situation will be identified. Next, specific design guidelines will be developed to address each situation. These guidelines will also be developed from the findings of the document analyses, which is the comprehensive review and analysis of relevant literature on help-seeking and development of guidelines on accessibility and usability in mobile contexts. Accordingly, the rationales and objects of these design guidelines will be offered. After that, associated techniques and methods that comply with design guidelines will be suggested. As a result, relevant features will be recommended based on the techniques and methods. Good and bad examples for the recommended features will also be provided. Additionally, related resources of theoretical research and practical best practices of how to design DLs on accessibility and usability to support BVI users in the mobile environment that the team used to create the guidelines will be shared. To determine how well mDLAUG supports the help needs of BVI users, three levels of conformance recommendation will be built for each of the guidelines: A minimum compliance, AA partial compliance, and AAA full compliance based on user studies.

Stage 3 Refine DL guidelines (05/01/24-11/30/24, Research Question 4)

Two-round Delphi surveys will be administered to 150 participants representing three groups of stakeholders to provide feedback for the draft of guidelines. Each participant will only respond to the survey that is applicable to his/her group. Data obtained in Stages 1.1, 1.2, 2.2, and 2.3 will be incorporated in the development of the Delphi surveys. The purpose of the first round is to solicit qualitative and quantitative feedback on the guidelines drafted in Stage 2. The survey will instruct participants to review the current guidelines and suggest additional changes they perceive to be important. This option will ensure the design guidelines and their components relevant to different groups of stakeholders. Quantitatively, the Delphi survey will instruct participants to fill out the survey on a “1 to 7” Likert scale for each component of the guidelines. For each component, five types of questions will be asked related to: importance, relevance, clarity, feasibility, and usefulness. Importance will serve as a key variable to rank each element. Qualitatively, suggestions to modify each component of the guidelines will be analyzed by using open coding. The results from the previous round will be incorporated into the second round. Quantitatively, ANOVA or the Kruskal–Wallis test depending on the data distribution will be applied to compare the assessment from the three groups.  In the second round, participants will be instructed to review the updated guidelines quantitatively and qualitatively in the same format as the first round. The findings of the second round will help the research team modify mDLAUG for BVI users.

Stage 4 Test and finalize the guidelines (12/01/24-07/31/25, Research Questions 5 & 6)

Apply guidelines for DL Assessment and finalize the guidelines. At this stage, the guidelines will be used to assess the same 12 DLs representing different types of DLs selected for the user study. Each of the thirty (30) DL developers, selected from Stage 3, will be instructed to assess two (2) of the selected DLs based on the guidelines. In total, each DL will be evaluated by 5 DL developers. The objectives of D4.1 are two-fold. First, the guidelines will be tested to see whether they can be used to assess DLs and further enhancements will be suggested to improve the DL design guidelines. Second, the guidelines will be used to assess the current status and conformance levels of the six DLs in terms of whether they meet conformance criteria for accessibility and usability for BVI users. The DL selection criteria in Stage 2 is adopted for this stage as well. Each DL will be assessed qualitatively (strengths and problems by open coding) and quantitatively (the extent of DL’s conformance by descriptive analysis) to review the current status of DLs.  Each DL will be assessed based on the conformance criteria, techniques, and failures for each of the DL guidelines. Simultaneously, each DL developer will also record his/her problems in applying the guidelines and make suggestions for the improvement of the guidelines. Each component of DL guidelines will be rated numerically for its importance, relevance, clarity, feasibility, and usefulness for its application.

Following the DL assessment, four focus groups with 7-8 participants in each group will be formed to discuss participants’ experience in assessing the DLs according to the new guidelines. Focus groups will provide suggestions on how to best finalize the guidelines. Furthermore, participants will discuss the challenges and solutions in their implementing and adopting mDLAUG. Considering the participants will come from all over the country, these focus groups will be taken place asynchronously. D2L, an online learning platform, in particular, its Discussion will be used to facilitate and record the discussions of the focus groups. Focus groups will include open-ended questions and seek modification suggestions to address the problems. Focus groups will be analyzed using open coding discussed in Stage 2. Taxonomies of problems with the guidelines and suggested modifications will be identified from the data. Based on the suggestions and feedback from the 30 DL developers, the team will work with advisory board members to finalize the guidelines. Finally, types of challenges and associated solutions will be identified. Finally, the team will compare the mDLAUG to dDLAUG to highlight all the unique components in the mobile DL environment, such as situations, factors, design guidelines, techniques and methods, and recommended features.

Project products/findings

DLs are increasingly becoming the preferred resource for searchers, replacing or supplementing physical interactions with traditional libraries. Simultaneously, more and more BVI users access and use DLs via their mobile devices. The proposed project innovatively addresses the issue of mobile DL accessibility and usability for one of the key underserved groups by creating design guidelines to address BVI users’ help-seeking situations. As previously reviewed, it is difficult for BVI users to adapt to different types of DLs. Limitations of existing design guidelines that do not holistically address these problems result in poor DL design and, subsequently, hinder BVI users’ ability to use DLs. Existing guidelines must be assessed for their gaps, and new guidelines need to be developed to make mobile DLs accessible, usable, and useful. None of the research has addressed building guidelines for mobile DL design that supports the accessibility and usability needs of BVI users. The research will impact DLs across the country by filling gaps; new guidelines will ensure that DLs are meeting the needs of diverse communities.

The innovative project addresses the issue of mobile DL accessibility and usability for one of the key underserved groups—BVI users by creating a new framework for designing DLs, in particular the development of mDLAUG to address BVI users’ help-seeking situations in the mobile environment. When such problems, or “help-seeking situations,” arise, BVI users seek assistance in order to complete their IR tasks. As BVI users have unique needs, challenges, strategies, and preferences in their interactions with DLs, especially when using mobile devices, this project focuses on those BVI users who are unable to see information presented on a display screen and rely on screen-reader software to interact with DLs. To offer accessible and usable DLs to BVI users, DL developers need to have design knowledge that considers the unique help-seeking situations and help needs of BVI users. Previous research has identified diverse types of help-seeking situations when BVI users use mobile devices, but not within DLs.

The project will address research questions in relation to limitations of existing design guidelines, unique help-seeking situations that BVI users encounter in mobile DL environments, types of needed design guidelines, the perspectives of key DL stakeholders on mDLAUG, the current status of DL design in compliance with mDLAUG and the challenges that developers face in implementing mDLAUG and associated solutions.

Results from this project will inform DL researchers and practitioners about the impact of DL design problems on BVI users and enable DL developers to enhance DLs for universal access, thereby increasing BVI users to access to and use of DLs in mobile contexts. The results also can be applied to generate guidelines for non-BVI users such as people with other types of disabilities and elderly people. This project can serve as a model for developing design guidelines for different user groups in diverse library/museum digital platforms. The design guidelines can be implemented into systems such as web search engines, online databases, and online public access catalogs. The project team will submit the guidelines to W3C and other organizations for adoption. As external environments and related technologies change continuously, the PIs plan to conduct future studies to ensure guidelines remain relevant to support BVI users.

Your content goes here. Edit or remove this text inline or in the module Content settings. You can also style every aspect of this content in the module Design settings and even apply custom CSS to this text in the module Advanced settings.

Team

Principal investigators

1. Iris Xie, PI, Professor, UWM-SOIS, hiris@uwm.edu
2. Wonchan Choi, Co-PI, Assistant Professor, UWM-SOIS,wchoi@uwm.edu

Research assistants

1. Shangang Wang, PhD student, UWM-SOIS
2. Hyun Seung Lee, PhD student, UWM-SOIS
3. Tae Hee Lee, PhD student, UWM-SOIS

Consultants for IMLS project

1. Rakesh Babu, PhD, Lead Accessibility Scientist, Envision, Rakesh.babu@envisionus.com
2. Jim Allan, Accessibility Coordinator, Webmaster, Texas School for the Blind and Visually Impaired; Chair, W3C User Agent Accessibility Guidelines Working Group, Web Accessibility Initiative, jimallan@tsbvi.edu
3. Krystyna Matusiak, PhD, Assistant Professor, University of Denver, matusiak@du.edu

Advisory Board Members

1. Mary Alexander, Senior Director, Vocational Rehabilitation and Programs and the College Success Program, Envision, Inc., alexander@envisionus.com
2. Michael Della Bitta, Director of Technology, Digital Public Library of America, michael@dp.la
3. Lou Ann Blake, Director of Research Programs, Blindness Initiatives, National Federation of the Blind, lblake@nfb.org
4. Michelle Brennan, Product Manager, OER Commons, michelle@iskme.org
5. Jennifer Ferretti, DLF Senior Program Officer, jferretti@clir.org
6. Mike Furlough, Executive Director, HathiTrust, furlough@hathitrust.org
7. Beret Balestrieri Kohn, Manager of Digital Assets, Milwaukee Art Museum, balestrierikohn@mam.org
8. Shane Huddleston, Product Manager, OCLC, huddless@oclc.org
9. Jonathan Lazar, Professor, Director of the Trace Center, the University of Maryland, jlazar@uwm.edu
10. Shaneé Yvette Murrain, Director of Community Engagement, Digital Public Library of America, shanee@dp.la
11. Serena Rosenhan, Vice President, User Experience Design, ProQuest, Rosehan@proquest.com
12. Bengisu Tulu, Professor of Information Systems at Worcester Polytechnic Institute, bengisu@wpi.edu
13. Marcia Zeng, Professor, Kent State University, mzeng@kent.edu