USE1. Difficulty using screen readers and voice activated commands

Definition

A situation that arises from difficulty utilizing screen readers (text with tooltip) A software program that reads textual information through synthesized speech and offers specialized keyboard commands to operate a computer interface. and voice activated commands (text with tooltip) A feature that allows users to control devices, applications, or systems by speaking instead of using manual input methods. within DLs (text with tooltip) The acronym for digital library (DL)  due to incompatibility of DLs with these assistive technologies.

Factors Leading to the Situation

  • Incompatibility: incompatible with assistive technologies

Guidelines

  1. Ensure that the design of DLs is compatible with assistive technologies.

Rationale for Suggesting the above Guidelines

To interact with DLs in the mobile environment, BVI (text with tooltip) The acronym for Blind and Visually Impaired. It refers to BVI users who rely on screen readers to interact with digital libraries (DLs).  users have to use screen readers (text with tooltip) A software program that reads textual information through synthesized speech and offers specialized keyboard commands to operate a computer interface. and other types of assistive technologies via their mobile devices. Meanwhile, voice activated commands (text with tooltip) A feature that allows users to control devices, applications, or systems by speaking instead of using manual input methods.  can make it easier for BVI users to access and use DLs without having to touch the screen to navigate DLs. However, it will be frustrating for BVI users when screen readers and voice activated commands are incompatible with DLs in the mobile environment. Therefore, the suggested guidelines ensure the compatible voice activated commands and relevant AI technologies are integrated to enhance BVI users’ experiences using DLs.

Techniques and Methods to Comply with a Specific Design Guideline

1.1. Integrate voice activated commands (text with tooltip) A feature that allows users to control devices, applications, or systems by speaking instead of using manual input methods.  to enhance users’ search experiences.
1.2. Integrate AI technologies to enhance users’ search experiences.

Features Suggested for Users

1.1.1. Voice Control / Voice Access
1.2.1. Siri / Apple AI / Google Assistant / Google Gemini

Examples of Best Practice

1.1.1.a1. Voice Control: Open a search feature with “Tap search” command

In USE1 Figure a1, users can activate a search feature in a DL by saying, “Tap or double tap search” command.

This screenshot shows an open search filter/box using Voice Control: “Tap Search.”
USE1 Figure a1. An example of using voice command with Voice Control

1.1.1.a2. Voice Access commands

These commands are examples of voice control interactions that enhance accessibility by enabling hands-free navigation and operation of web pages or applications.

1.1.1.a2.1. Create a search

The command “search” allows users to activate the search function, input search terms using their voice, and execute the search without requiring manual input. In the USE1 Figure a2, Voice Access is utilized to perform a search by saying, “search coronavirus.” The search term is automatically populated in the search box (text with tooltip) A design element that allows users to enter search terms into a blank field.  and processed to generate the results.

Access allows users to perform a search by saying a command like "search coronavirus," which is automatically entered into the search box and processedUSE1 Figure a2. An example of creating a search using voice access

1.1.1.a2.2.  Navigate within a page

“Swipe down/right”, “scroll down/up” and “move up/down” commands allow users to move between different sections or elements on the page (e.g., headings, links, or form fields) by issuing voice commands.

1.1.1a2.3. Go back previous page

“Go Back” command allows users to return to the previously viewed page using a simple voice command, equivalent to pressing the browser’s back button.

1.2.1.a1. Siri: ChatGPT integration

Siri can tap into ChatGPT to help users with certain requests when it determines ChatGPT can provide a helpful answer. In USE1 Figure a3, when a user requests to open the Library of Congress, ChatGPT provides a link along with brief information about the Digital Library.

Siri can use ChatGPT to help users with certain requests when it determines that ChatGPT can provide a helpful answer.USE1 Figure a3. An example of ChatGPT integration with Siri

1.2.1.a2. Google Gemini: Applicable command “Ask about this screen”

Users can ask questions through choosing “ask about this screen” to get the specific information within the page. In USE1 Figure a4, Google Gemini will automatically generate the screenshot of the page once users use “ask about this screen” and then Google Gemini will analyze the generated screenshot.

Users can ask questions through choosing “ask about this screen” in Google Gemini to get the specific information within the page.
USE1 Figure a4. An example of Google Gemini command “Ask about this screen”

1.2.1.a2.1. Google Gemini: Applicable command “List all the headings on this webpage”

In USE1 Figure a5, Google Gemini will list all the headings on specific webpage when users ask it to “list all headings on this webpage”.

The image shows that Google Gemini can execute the command to list all the headings on this webpage.

USE1 Figure a5. An example of Google Gemini command “List all the headings on this webpage”

1.2.1.a2.2. Google Gemini: Applicable command “Summarize this webpage”

In USE1 Figure a6, Google Gemini can summarize the page when users ask it to “summarize this webpage”.

The image shows that Google Gemini can execute the command to summarize the page.

USE1 Figure a6. An example of Google Gemini command “Summarize this webpage”

1.2.1.a2.3. Google Gemini: Applicable command “What are the key sections of this site?”

In USE1 Figure a7, Google Gemini can explain what those sections are about when users would like to know the key sections of the page.

The image shows that Google Gemini can execute commands for the key sections of the site.

USE1 Figure a7. An example of Google Gemini command “What are the key sections of this site?”

1.2.1.a2.4. Google Gemini: Applicable command “Where is the “Contact us” link?” or “Where can I find the search field?”

In USE1 Figure a8 and USE1 Figure a9, Google Gemini can execute the command for users about the location of link and specific features.

The image shows that Google Gemini can execute the command to ask for the "Contact US" link on the page.

USE1 Figure a8. An example of Google Gemini command “Where is the “Contact us” link?”

The image shows that Google Gemini can execute commands about where to find the search field.

USE1 Figure a9. An example of Google Gemini command “Where can I find the search field?”

1.2.1.a2.5. Google Gemini: Applicable command “List all the buttons available.”

In USE1 Figure a10, Google Gemini can list all the buttons within the webpage when users would like to identify quickly what those buttons are.

Google Gemini can execute the command to list all the buttons available.
USE1 Figure a10. An example of Google Gemini command “List all the buttons available”

1.2.1.a2.6. Google Gemini: Applicable command “Describe the images on this page”

In USE1 Figure a11, Google Gemini can analyze the image of the collection item and provide a description of its contents.

Google Gemini can execute the command to describe the images on this page.

USE1 Figure a11. An example of Google Gemini command “Describe the images on this page”

Examples of Poor Practice

1.1.1.b1. Voice Control: Interference with Siri

Siri processed comments/commands from VoiceOver and Voice Control. In USE1 Figure b1, when both Siri and Voice Control are enabled, Siri override the command “Tap advanced search” and displayed the search result instead of interacting with the advanced search.

A screenshot when both Siri and Voice Control are on, Siri takes over the commands and shows the search result instead of tapping advanced search.
USE1 Figure b1. An example of Siri takes Voice Control commands “Tap advanced search”

1.1.1.b2. Voice Control: difficulty closing a pop-up

In USE1 Figure b2, when users have difficulty closing a pop-up window, the system suggests, “Go Back.” However, when users say “Go Back,” it takes them to the previous search engine page.

When users encounter difficulties closing a pop-up window, the system offers the suggestion, "Go Back." However, selecting "Go Back" takes them to the previous search engine page instead.USE1 Figure b2. An example of a confusing suggestion for Voice Control

Resources