Instruction for Search Tasks

Overview

Our goal is to examine the experiences of blind and visually impaired (BVI) people in searching for information using digital libraries. We first ask you to explore two digital libraries, and then perform two search tasks in each of the two digital libraries while thinking aloud. We will audio-record your verbalizations, and later analyze it to identify what aspects of the design are not BVI friendly. We will then try to identify design improvements that will make information seeking from a digital library barrier-free for BVI people. This is not a test of your skills. Rather, it is a test of how well the system is designed.

As you perform the tasks, we ask that you verbalize your thoughts during your entire interaction process with the digital libraries, in particular the problems you encountered, DL feature(s) used when encountering a problem, the cause of the problem, DL feature(s) used when overcoming the problem, the desired features that would have helped you overcome the problem, and the outcome of the problem.

If you get stuck at any point while performing a task, and need assistance to continue, please ask the researcher.

Digital Library 1 (Library of Congress Digital Collections)

Exploration (10 minutes)

Visit the website of the Library of Congress Digital Collections. Please think aloud during this exploration and perform the following steps:

1. Explore the various features and functions of this site, specifically the navigation region, to gain a basic understanding of how it works.

Tasks (15 minutes per task, total 30 minutes)

When you are done exploring, please return to the Library of Congress Digital Collections, and complete the following tasks while thinking aloud. Please limit your search for each of the tasks to 15 minutes. The researcher will remind you when you complete your search task in 15 minutes.

Rate your level of subject knowledge of the below topic on a scale from 1 to 7, (1 = Not at all Knowledgeable and 7 = Extremely knowledgeable):

1. Find the date when Rosa Parks was arrested. When did Rosa Parks take a leadership role in a branch of NAACP and what was her role?

Rate your level of subject knowledge of the below topic on a scale from 1 to 7, (1 = Not at all Knowledgeable and 7 = Extremely knowledgeable):

2.  Find items related to the Great Earthquake and Fire in San Francisco in the early 1900s.

Make sure each item either represents one distinct format or one distinct aspect of this search topic. Please find as many items as possible from the digital library as you can.

Digital Library 2 (ArtStor)

Exploration (10 minutes)

Visit the website of ArtStor. Please think aloud during this exploration and perform the following steps:

1. Explore the various features and functions of this site, in particular the navigation region, to gain a basic understanding of how it works.

Tasks (15 minutes per task, total 30 minutes)

When you are done exploring, please return to ArtStor, and complete the following tasks while thinking aloud. Please limit your search for each of the task to 15 minutes. The researcher will remind you when you complete your search task in 15 minutes.

Rate your level of subject knowledge of the below topic on a scale from 1 to 7, (1 = Not at all Knowledgeable and 7 = Extremely knowledgeable):

1. Identify a work of art by Josef Albers focusing on geometric abstraction in the 1930s. Find out the material type and the name of the art gallery that houses it.

Rate your level of subject knowledge of the below topic on a scale from 1 to 7, (1 = Not at all Knowledgeable and 7 = Extremely knowledgeable):

2. Find images or objects /artifacts related to Apollo that were created in the 4th century in Greece. Make sure each item either represents one distinct type of artwork or one distinct aspect of this search topic. Please find as many items as possible.

Post-task Interviews

DL1 Exploration Follow-Up

[Learn how to use]

What approach did you follow to become familiar with the DL1? Describe the main three difficulties or problems that you encountered when exploring DL1. For each problem that you encountered, which features did you use? Which features were useful or not useful? Are there any desired features that would have helped you in the exploration?

DL1 Task Follow-Up (follows each task)  

[Problems encountered – help situations] [Problem solving – help situations]  

[Observer may inform of successful or unsuccessful interactions if the subject cannot remember.]

1. What approaches did you take to find information during Task 1?

Describe the main three problems or difficulties you encountered in performing this task? For each problem that you encountered, what features did you use? Which features were useful or not useful? Are there any desired features that would have helped you complete this task?

2. What approaches did you take to find information during Task 2?

Describe the main three problems or difficulties you encountered in performing this task? For each problem that you encountered, what features did you use? Which features were useful or not useful? Are there any desired features that would have helped you complete this task?

DL2 Exploration Follow-Up

[Learn how to use]

What approach did you follow to become familiar with the DL2? Describe the main three difficulties or problems that you encountered when exploring DL2? For each problem that you encountered, which features did you use? Which features were useful or not useful? Are there any desired features that would have helped in the exploration?

DL2 Task Follow-Up (follows each task)  

[Problems encountered – help situations] [Problem solving – help situations]

[Observer may inform of successful or unsuccessful interactions if the subject cannot remember.]

1. What approaches did you take to find information during Task 1?

Describe the main three problems or difficulties you encountered in performing this task? For each problem that you encountered, what features did you use? Which features were useful or not useful? Are there any desired features that would have helped you complete this task?

2. What approaches did you take to find information during Task 2? Describe the main three problems or difficulties you encountered in performing this task? For each problem that you encountered, what features did you use? Which features were useful or not useful? Are there any desired features that would have helped you complete this task?

Digital Library Used: ArtStor

Task Number: 1       

1. Visit 1(15 minutes) – https://library.artstor.org/library/

2. Identify a work of art by Josef Albers focusing on geometric abstraction in the 1930s. Find out the material type and the name of the art gallery that houses it.

3. Rate your level of subject knowledge of the above topic on a scale from 1 to 7,

(1 = Not at all Knowledgeable and 7 = Extremely knowledgeable):

4. The Diary forms are blank templates for you to fill in information about the problems you encountered, along with other information.

For this task, please complete the following information.

Date:       

Browser:

Screen reader and version:

Beginning time for the task:     

Ending time for the task:

Results for the task:

Please indicate the results or information found, along with URLs, for this task.

Please email your Search Diary to xxx at xxx@uwm.edu as soon as it is completed.  If you have any questions related to the diary, please call xxx at xxx-xxx-xxxx.

Thank you for your participation.

Diary Form

Please provide as much of the following information as possible for each example:

• • Problem encountered
  • Digital collection selected
  • Digital Library and/or screen reader feature(s) used when encountering the problem
  • Factor(s) behind the problem
    
  • Digital Library and/or screen reader feature/human help used when overcoming the problem
  • Desired Digital Library feature(s)
  • Outcome
  • Amount of time

a)     less than a minute,

b)    1-3 minutes

c)     More than 3 minutes

Q2 Please specify your age.

Q3 Please specify your gender.

Q4 Please specify your ethnicity.

Q5 Please indicate the highest educational level you have achieved.

Q6 Please specify your job title.

Q7 What is your vision condition? Select one.

• • Blind (1)
  • Visually impaired (2)

Q8 Please indicate how many years you have used the Internet to search for information.

Q9 How often do you use the Internet? Please rate on a scale of 1=never use to 7=use all the time).

Q10 Please specify three main purposes that you use the Internet for.

Q11 How would you rate your information search skills on the Internet? Select one.

• • Beginner (I need some help to search something) (1)
  • Intermediate (Fluent with using commercial search engines like Google and Yahoo) (2)
  • Advanced (Fluent with using advanced search functions) (3)
  • Expert (Good at using advanced search functions and understanding back-end information retrieval mechanisms) (4)

Q12 Please specify the main screen reader you use to access Internet via computers.

Q13 Please specify the length of time that you have used this Screen Reader.

Q14 Please specify three or four ways that blindness affects your use of the Internet.

Q15 This next series of questions will ask about your use of information retrieval systems. An information retrieval system is an online system that allows you to search for information in a variety of formats.

Q16 How frequently do you use Web search engines (e.g., Google)? Please rate on a scale of 1=never use to 7=use all the time).

Q17 How frequently do you use Multimedia retrieval systems (e.g., YouTube)? Please rate on a scale of 1=never use to 7=use all the time.

Q18 How frequently do you use digital libraries? A digital library contains items that are digitized and available through online collections in various formats (e.g., NLB BARD, Bookshare, or Digital Public Library of America).  Please rate on a scale of 1=never use to 7=use all the time.

Q19 Please specify what digital libraries you use, or have used in the past.

Q20 Please specify your preferred search engine to search for information on the Internet (e.g., Google, Bing, Dogpile).

Q21 This next series of questions will ask you to rate the helpfulness of various system features in your preferred search engine.

Q22 To what extent do features of your preferred search engine assist you or not to create and submit keyword search terms? Please rate on a scale of 1=not at all helpful to 7=extremely helpful.

Q23 To what extent do features of your preferred search engine assist you or not to change your keyword search terms? Please rate on a scale of 1=not at all helpful to 7=extremely helpful.

Q24 To what extent do features of your preferred search engine assist you or not to find relevant or useful items from search result pages? Please rate on a scale of 1=not at all helpful to 7=extremely helpful.

Q25 To what extent does your preferred search engine assist or not assist you to find relevant topics or items from browsing collections or categories? Please rate on a scale of 1=not at all helpful to 7=extremely helpful.

Q26 To what extent do features of your preferred search engine assist you or not to access the items in the system, such as opening documents or accessing alternative text? Please rate on a scale of 1=not at all helpful to 7=extremely helpful.

Q27 To what extent do features of your preferred search engine assist you or not to go back to previous pages? Please rate on a scale of 1=not at all helpful to 7=extremely helpful.

Q28 To what extent do features of your preferred search engine assist you or not to know where you are in the system? Please rate on a scale of 1=not at all helpful to 7=extremely helpful.

Q29 To what extent do features of your preferred search engine assist you or not to judge relevance or usefulness of individual items after you open or access the items? Please rate on a scale of 1=not at all helpful to 7=extremely helpful.

Q30 To what extent do features of your preferred search engine assist you or not to judge relevance or usefulness of images? Please rate on a scale of 1=not at all helpful to 7=extremely helpful.

Q31 To what extent do features of your preferred search engine assist you or not to judge relevance or usefulness of video or audio items? Please rate on a scale of 1=not at all helpful to 7=extremely helpful.

Q32 Please specify if there are any features missing from your preferred search engine that you would like to be available. If yes, what are these features?

Q33 Please explain what you do when you encounter a problem in using your preferred search engine.

Q34 Please explain the major problems you see in your preferred search engine. Do you have any suggestions or potential solutions to fix these problems?

Q35 Please click the next button to submit the survey.

Q2 Please tell us whether you visited the help page? Why or Why not?

Q3 If you visited the help page, please  rate the helpfulness of the help page on a scale of 1=Not at all helpful to 7=Extremely helpful.

Q4 The next section includes  ten 3-part questions related to different types of features offered by the Library of Congress Digital Library.

Q5 Please rate to what extent did the Library of Congress digital library assist or not assist you to create and submit keyword search terms? Rate on a scale of 1=Not at all helpful to 7=Extremely helpful.

Q6 How did the Library of Congress digital library features assist or not assist you to create and submit your keyword search terms? Please explain.

Q7 What types of the Library of Congress digital library features are not available but would have been helpful for you to create and submit keyword search terms? Please explain.

Q8 Please rate to what extent did the Library of Congress digital library features assist or not assist you to change your keyword search terms? Rate on a scale of 1=Not at all helpful to 7=Extremely helpful.

Q9 How did the Library of Congress digital library features assist or not assist you to change your keyword search terms? Please explain.

Q10 What types of Library of Congress digital library features are not available but would have been helpful for you to change your keyword search terms? Please explain.

Q11 Please rate to what extent did the Library of Congress digital library assist or not assist you to find relevant or useful items from search result pages? Rate on a scale of 1=Not at all helpful to 7=Extremely helpful.

Q12 How did the Library of Congress digital library features assist or not assist you to find relevant items from search result pages? Please explain.

Q13 What types of the Library of Congress digital library features are not available but would have been helpful for you to find relevant items from search result pages? Please explain.

Q14 Please rate to what extent did the Library of Congress digital library features assist or not assist you to find relevant topics or items from browsing collections or categories? Rate on a scale of 1=Not at all helpful to 7=Extremely helpful.

Q15 How did the Library of Congress digital library features assist or not assist you to find relevant topics or items from browsing collections or categories? Please explain.

Q16 What types of the Library of Congress digital library features are not available but would have been helpful for you to find relevant topics or items from browsing collections or categories? Please explain.

Q17   Please rate to what extent did the Library of Congress digital library features assist or not assist you to access the items in the DIGITAL LIBRARY, such as opening documents or accessing alternative text? Rate on a scale of 1=Not at all helpful to 7=Extremely helpful.

Q18 How did the Library of Congress digital library features assist or not assist you to access the items in the DIGITAL LIBRARY? Please explain.

Q19 What types of the Library of Congress digital library features are not available but would have been helpful for you to access the items in the DIGITAL LIBRARY? Please explain.

Q20 Please rate to what extent did the Library of Congress digital library features assist or not assist you to go back to previous pages? Rate on a scale of 1=Not at all helpful to 7=Extremely helpful.

Q21 How did the Library of Congress digital library features assist or not assist you to go back to previous pages? Please explain.

Q22 What types of Library of Congress digital library features are not available but would have been helpful for you to go back to previous pages? Please explain.

Q23 Please rate to what extent did the Library of Congress digital library assist or not assist you to know where you are in the DIGITAL LIBRARY? Rate on a scale of 1=Not at all helpful to 7=Extremely helpful.

Q24 How did the Library of Congress digital library features assist or not assist you to know where you are in the DIGITAL LIBRARY? Please explain.

Q25 What types of the Library of Congress digital library features are not available but would have been helpful for you to know where you are in the DIGITAL LIBRARY? Please explain.

Q26 Please rate to what extent did the Library of Congress digital library assist or not assist you to judge relevance or usefulness of individual items after you opened or accessed the items? Rate on a scale of 1=Not at all helpful to 7=Extremely helpful.

Q27 How did the Library of Congress digital library features assist or not assist you to judge relevance or usefulness of individual items? Please explain.

Q28 What types of the Library of Congress digital library features that are not available but would have been helpful for you to judge relevance or usefulness of individual items? Please explain.

Q29 Please rate to what extent did the Library of Congress digital library features assist or not assist you to judge relevance or usefulness of images? Rate on a scale of 1=Not at all helpful to 7=Extremely helpful.

Q30 How did the Library of Congress digital library features assist or not assist you to judge relevance or usefulness of images? Please explain.

Q31 What types of the Library of Congress digital library features are not available but would have been helpful for you to judge relevance or usefulness of images? Please explain.

Q32 Please rate to what extent did the Library of Congress digital library features assist or not assist you to judge relevance or usefulness of video or audio items? Rate on a scale of 1=Not at all helpful to 7=Extremely helpful.

Q33 How did the Library of Congress digital library features assist or not assist you to judge relevance or usefulness of video or audio items? Please explain.

Q34 What types of the Library of Congress digital library features are not available but would have been helpful for you to judge relevance or usefulness of video or audio items? Please explain.

Q35 How easy is the Library of Congress digital library to use? Please rate on a scale of 1=Not at all easy to 7=Extremely easy.

Q36 What made the Library of Congress digital library easy or difficult to use? Please explain.

Q37 How satisfied are you in using the Library of Congress digital library? Please rate on a scale of 1=Not at all satisfied to 7=Extremely satisfied.

Q38 Which of the Library of Congress digital library features are you satisfied with? Which the Library of Congress digital library features are you not satisfied with? Please explain.

Q39 How useful are the Library of Congress digital library features? Please rate on a scale of 1=Not at all useful to 7=Extremely useful.

Q40 Could you please describe briefly your overall reaction to the use of the Library of Congress digital library?

Q41 Do you have any final thoughts about the accessibility and usability of the Library of Congress digital library that you would like to share?

Please click the next button to submit the survey.