HealthQ: Unveiling Questioning Capabilities of LLM Chains in Healthcare Conversations

This project introduces HealthQ, a framework that evaluates how effectively large language model (LLM) chains ask questions in digital healthcare settings. Going beyond simple answering, HealthQ assesses the ability of advanced LLM chains — including Retrieval-Augmented Generation (RAG), Chain of Thought (CoT), and reflective chains — to elicit specific, relevant, and useful patient information. Using an LLM judge and metrics like ROUGE and NER-based comparisons, HealthQ is validated on medical dialogue datasets (ChatDoctor, MTS-Dialog) and tested across multiple judge models. The framework establishes a systematic, model-agnostic method for measuring questioning quality and demonstrates how stronger inquiry skills directly improve patient information gathering, leading to safer and more effective healthcare conversations.

Publications:


Full paper

Project information

  • Category: Future Healthcare, Large Language Models, and Translation and Practice
  • Contact Person: Amir M. Rahmani
  • HealthQ: Unveiling Questioning Capabilities of LLM Chains in Healthcare Conversations

info@futurehealth.uci.edu

© Copyright 2021 UCI Institute for Future Health - All Rights Reserved