A Framework for Designing Retrieval Effectiveness Studies of Library Information Systems Using Human Relevance Assessments

Behnert, C., & Lewandowski, D. (2017). A framework for designing retrieval effectiveness studies of library information systems using human relevance assessments. Journal of Documentation, 73(3), 509–527.

Download: Preprint / Publisher Version

  • Purpose
    • This paper demonstrates how to apply traditional information retrieval evaluation methods based on standards from the Text REtrieval Conference (TREC) and web search evaluation to all types of modern library information systems including online public access catalogues, discovery systems, and digital libraries that provide web search features to gather information from heterogeneous sources.
  • Design/methodology/approach
    • We apply conventional procedures from information retrieval evaluation to the library information system context considering the specific characteristics of modern library materials.
  • Findings
    • We introduce a framework consisting of five parts: (1) search queries, (2) search results, (3) assessors, (4) testing, and (5) data analysis. We show how to deal with comparability problems
      resulting from diverse document types, e.g., electronic articles vs. printed monographs and what issues need to be considered for retrieval tests in the library context.
      Practical implications
    • The framework can be used as a guideline for conducting retrieval effectiveness studies in the library context.
  • Originality/value
    • Although a considerable amount of research has been done on information retrieval evaluation, and standards for conducting retrieval effectiveness studies do exist, to our knowledge this is the first attempt to provide a systematic framework for evaluating the retrieval effectiveness of twenty-firstcentury library information systems. We demonstrate which issues must be considered and what decisions must be made by researchers prior to a retrieval test.