Ethical AI and Librarianship

A Resource Guide

Home
Overview
Resources
Bibliography

Humans-in-the-Loop RECOMMENDATIONS REPORT

Field
    		Description
    Title Humans-in-the-Loop RECOMMENDATIONS REPORT
    Type Reports
    Creator Library of Congress Labs, AVP (metadata solution provider)
    Link
    Creation Date 11/29/2021
    Last Updated Date --
    Summary This report documents the Humans-in-the-Loop (HITL) project (2020–2021), led by the Library of Congress (LC) Labs, which developed a framework for integrating human expertise with machine learning (ML) to improve metadata creation for digital collections. Using digitized U.S. Yellow Pages directories as a case study, the project established workflows that ethically and effectively combine crowdsourcing and ML to extract metadata such as business names and addresses. The report outlines a user-centered approach to designing human-in-the-loop systems across four stages: 1) collection selection, 2) design, 3) implementation, and 4) presentation and sharing. Each stage discusses objectives, goals, challenges, human involvement, feedback mechanisms, lessons learned, and future recommendations. Specifically, ethical considerations at each stage include:
    • Collection selection: 1) Respecting the privacy of collection subjects and creators, 2) Identifying and mitigating potential risks to users and collection contributors
    • Design: 1) Helping volunteers understand how their contributions support ML processes, 2) Identifying and mitigating risks to volunteers from exposure to potentially offensive content, 3) Identifying and documenting unintended consequences of ML outputs.
    • Implementation: 1) Tracking data provenance and accuracy of ML-generated data. 2) Clearly communicating how user-generated data supports ML pipelines
    • Presentation and sharing: 1) Communicating data provenance to library users. 2) Helping users understand potential biases and data incompleteness.
    The appendices provide reusable resources that support implementation of ethical HITL workflows in cultural heritage institutions, such as a “Workflow Database ER Diagram”, “Code Repository”, “Machine Learning and Crowdsourcing Data Flows”, “Collection Candidate Evaluation Sheet”, “Crowdsourcing Prototype User Testing Plan & Discussion Guide”. Some of these materials are also accessible through the project’s GitHub repository (see Link field).
    Topic Ethical AI. AI and Librarianship. Digital Collection.
    Source and Link Library of Congress Labs. https://labs.loc.gov/
    Access Open
    Accessibility --
    Audience Librarians – Academic (& Research). Scholars and Students. Information Professionals.
    Platform or Format Document (.pdf)
    Length 97 pages
    Geography USA
    Language ENG
    Description Date 06/18/2025

    Ethical AI and Librarianship: A Resource Guide

    Home
    Overview
    Resources
    Bibliography