2. User study instruments

2.1. Mobile digital library accessibility and usability guidelines (mDLAUG) funded by the Institute of Museum and Library Services (IMLS)

We selected six DLs for the user studies. Even though participants of different groups worked on search tasks corresponding to each of the DLs, the search and interview instruments are identical.

2.1.1. Pre-questionnaire

The main components of the pre-questionnaire are listed below:

      • Demographic information (e.g., age, gender, vision condition, ethnicity/race, and education)
      • Internet use experience
      • Search skills
      • DL use experience
      • Reasons for using information retrieval systems
2.1.2. Tasks and think aloud

The table below shows information related to different types of tasks and associated duration and task examples.

Table 1. Task types and examples

Task Type Task Duration Example (DL: Task Information)
Orientation task 10 minutes LoC, DPLA, HathiTrust, ArtStor, OER Commons, and the National Museum of African American History and Culture: You will have 10 min to explore the features and functions of this digital library. Please talk continuously about your thoughts and actions in relation to your interactions with the digital library during this task including its structure, features, content, format, search results, etc. Please specify your intentions for each action, the problems you encountered, and your solutions.
Specific search 15 minutes ArtStor: Find a World War II poster on war dogs in 1943. Find out the name of the repository that houses it. What were the two figures portrayed in the poster?
Exploratory search 15 minutes OER Commons: Find interactive items or graphics related to relationships between position, velocity, and acceleration. Each should represent different formats (e.g., interactive item, image, and text) or content (e.g., the relationship between velocity and acceleration, the relationship between position and velocity) of this search topic.
2.1.3. Data collection process

Below is an example that shows the process of BVI users working on search tasks and thinking aloud when using a DL. The sentences in quotation marks are read by research assistants.

  1. The Library of Congress Digital Collections (https://www.loc.gov/collections/)
    • Orientation task (10 minutes)

Visit the website of the Library of Congress Digital Collections available at: https://www.loc.gov/collections/

      • Send DL link via the chat box on Teams (they can click the link)

“We will begin with the Library of Congress Digital Collections. I will send the digital library link via Team chat box.”

https://www.loc.gov/collections/

      • Make sure that participants are sharing their browser not Teams.
      • Ask participants to explore the features and functions of this site, thinking aloud, to gain a basic understanding of how it works.

“Our first task is the orientation task. You will have 10 minutes to explore the features and functions of this digital library. Please talk continuously about your thoughts and actions in relation to your interactions with the digital library during this task including its structure, features, content, format, search results, etc. Please specify your intentions for each action, the problems you encountered, and your solutions. I will remind you when you have 2 minutes left. Are you ready to start the orientation task? I am going to set the timer now.”

    • Start the timer (10 minutes)
        • Keep quite when participants are exploring DLs and screen reader is talking
        • Ask participants to keep talking if they pause for a while
        • “Please keep talking.”
    • 2 minutes reminder

“You have 2 minutes left for this task.”
“That was the orientation task. Let’s begin Task 1. You will have 15 minutes. I will remind you when there are 2 minutes left”

    • Search task 1 for LOC (15 minutes)
        • Send the task in the chat box and ask participants to use the task information if they need to review the task again

“I am going to send two messages in the chat box, the first one is the task, the second one is the link to the digital library. You can use the task information if you need to review it again”

      • Copy the task information and link below separately.

LOC Task 1: Find a copy of the United States Declaration of Independence, formally the unanimous Declaration of the thirteen United States of America. When was it passed by the congress? What are the unalienable rights mentioned in the document?

https://www.loc.gov/collections/

“To what extent are you familiar with this search topic? 1 Not familiar at all to 7 Extremely familiar”
“Are you ready to start LOC Task 1?

Please tell me whenever you find a relevant item and explain why the item is relevant. Please talk continuously about your thoughts and actions in relation to your interactions with the digital library during this task including its structure, features, content, format, search results, etc. Please specify your intentions for each action, the problems you encountered, and your solutions. I will remind you when you have 2 minutes left. Are you ready to start the orientation task? I am going to set the timer now.”

    • Start the timer (15 minutes)
      • Ask/remind participants to keep talking about their thoughts and actions.

“Please keep talking.”

      • Ask participants to specify relevant items based on the content of items when they are browsing items during the search tasks (transcriber will make notes)

“Please tell me whenever you find a relevant item and explain why the item is relevant”.

      • Take notes if anything worth noting happens during the task.
    • 2 minutes reminder.

“You have 2 minutes left for this task.”
“That was Task 1. Let’s begin Task 2. You will have 15 minutes. I will remind you when there are 2 minutes left”

    • Search task 2 for LOC (15 minutes)
      • Send the task in the chat box and ask participants to use the task information if they need to review the task again

“I am going to send two messages in the chat box, the first one is the task, the second one is the link to the digital library. You can use the task information if you need to review it again”

      • Copy the task information and link below separately.

LOC Task 2: Find items related to battles in World War 2 in the digital collection. Please find as many items as you can within 15 minutes. Each should represent different formats (e.g., image, text, etc.) or content (e.g., different battles) of this search topic. Please let us know whenever you find a relevant item.

https://www.loc.gov/collections/

“To what extent are you familiar with this search topic? 1 Not familiar at all to 7 Extremely familiar”
“Are you ready to start LOC Task 2?

Please tell me whenever you find a relevant item and explain why the item is relevant. Please talk continuously about your thoughts and actions in relation to your interactions with the digital library during this task including its structure, features, content, format, search results, etc. Please specify your intentions for each action, the problems you encountered, and your solutions. I will remind you when you have 2 minutes left. Are you ready to start the orientation task? I am going to set the timer now.”

    • Start the timer (15 minutes)
      • Ask/remind participants to keep talking about their thoughts and actions.

“Please keep talking.”

      • Ask participants to specify relevant items based on the content of items when they are browsing during the search tasks (transcriber will make notes)

“Please tell me whenever you find a relevant item and explain why the item is relevant”.

      • Take notes if anything worth noting happens during the task.
    • 2 minutes reminder.

“You have 2 minutes left for this task.”
“That was Task 2. You are done with this digital library. I will ask you a few questions based on your experience of using this digital library.”

2.1.3. Post-system survey and interview

The main components of the post-system survey and interview are listed below:

      • Assessment of accessibility (e.g., detectability, predictability, activatability, compatibility)
      • Assessment of usability (e.g., learnability, satisfaction, usefulness, ease of use)
      • Technology acceptance model (TAM) questions (e.g., relevance of information, help, enjoyment, and confidence in using DLs)
2.1.4. Post-search interview

The main components of the post-search interview are listed below:

      • Overall feedback for the DLs, preference, and rationale
      • Main problems and challenges encountered, and actions taken to deal with them
      • Willingness to continue suing DLs in the future and justification
      • Challenges and benefits of using DLs

2.2.  Universal accessibility and usability of digital library project funded by University of Wisconsin-Milwaukee Discovery and Innovation Grant (DIG)

2.2.1. Pre-search interview

The main components of the pre-search interview are listed below:

      • Demographic information (e.g., age, gender, vision condition, ethnicity/race, and education)
      • Internet use experience
      • Search skills
      • Information retrieval system use experience
2.2.2. Mobile website/app tasks
    • Orientation task for the mobile website (10 minutes)

Visit the website of the Library of Congress Digital Collections available at: https://www.loc.gov/collections/

Ask them to freely explore the various features and functions of this site, thinking aloud, to gain a basic understanding of how it works.

    • Search task 1 for the mobile website (15 minutes)

Locate a copy of Abraham Lincoln’s Gettysburg address available in the Library of Congress. What format (e.g., image, text, etc.) is this copy in? When was this address made?

    • Search task 2 (15 minutes)

Find items related to the women’s rights movements during the 1800s and 1900s in the digital collection. Make sure each item either represents one distinct format (e.g., image, text, etc.) or one distinct aspect of this search topic. Please find as many items as you can within 15 minutes.

2.2.3. Post-platform interview

The main components of the post-platform interview are listed below:

      • Assessment of accessibility of DLs (e.g., ease of access to DLs, ease of navigation of DLs, ease of access to DL features, ease of access to information, and compatibility)
      • Assessment of usability of DLs (e.g., ease of learning, usefulness, efficiency, and satisfaction)
2.2.4. Mobile app tasks
    • Orientation task for the mobile app (10 minutes)

Ask them to freely explore the various features and functions of this site, thinking aloud, to gain a basic understanding of how it works.

    • Search task 1 for the mobile app (15 minutes)

Locate a copy of Abraham Lincoln’s Gettysburg address available in the Library of Congress. What format (e.g., image, text, etc.) is this copy in? When was this address made?

    • Search task 2 for the mobile app (15 minutes)

Find items related to the women’s rights movements during the 1800s and 1900s in the digital collection. Make sure each item either represents one distinct format (e.g., image, text, etc.) or one distinct aspect of this search topic. Please find as many items as you can within 15 minutes.

2.2.5. Post-platform interview

The main components of the post-platform interview are listed below:

    • Assessment of accessibility of DLs (e.g., ease of access to DLs, ease of navigation of DLs, ease of access to DL features, ease of access to information, and compatibility)
    • Assessment of usability of DLs (e.g., ease of learning, usefulness, efficiency, and satisfaction)
2.2.6. Post-search interview

The main components of the post-search interview are listed below:

    • Preference between the mobile app and the mobile website and rationale
    • Overall assessment of the two DL platforms
    • Final thoughts about accessibility and usability of the two DL platforms