Building an Evidence-Based Q&A System for Surgical Clinicians: Lessons from a Medical Q&A Pilot

How we built, tested, and learned from a RAG system processing 277,000+ medical document chunks

The Challenge: Instant, Accurate Answers for Operating Room Professionals

Reliable agent access to authoritative content at generation time is important both for the scholarly publishing ecosystem and for improving clinical outcomes—even when the "agent" is as simple as a Q&A RAG chatbot. This post documents our journey building one such system for a specialized medical domain.

A domain-specific embedding model outperformed OpenAI's then-best by ~2x on medical queries—but performed 4x worse on general benchmarks. That tension shaped every decision we made building a Q&A system for surgical clinicians.

A leading professional medical organization publishes comprehensive clinical guidelines, research articles, and best practices that serve as the gold standard for perioperative care. With tens of thousands of members worldwide relying on this content, the challenge is clear: thousands of documents containing critical medical information, but no easy way to quickly find specific answers when you need them most.

Enter our medical Q&A system: our attempt to transform this vast medical literature into an intelligent Q&A system using Retrieval-Augmented Generation (RAG). What started as a straightforward implementation quickly humbled us. Turns out, specialized medical domains break nearly every assumption you bring from general-purpose RAG.

This work was developed in collaboration with clinical domain experts and engineering team members who contributed to testing, evaluation, and system refinement.

The Technical Journey: Embeddings, Chunks, and the Art of Medical Information Retrieval

Building a Domain-Specific Benchmark: Why Generic Benchmarks Failed Us

Before we could even begin selecting embedding models, we faced a fundamental problem: how do you evaluate retrieval quality for specialized medical content? Standard benchmarks like MS MARCO or Natural Questions weren't particularly helpful. They don't contain questions about surgical site infections or sterile processing techniques.

Our solution came from the clinical journals themselves. Many issues contain clinical Q&A sections where experts answer common surgical questions. These were real queries from practicing clinicians, answered by subject matter experts, and peer-reviewed for accuracy. We had found our evaluation dataset hiding in plain sight.

Here's how we built our benchmark:

# Our benchmark creation process

Step 1: Extract Q&A pairs from clinical journals (2020-2024)
Step 2: Map each question to its source article(s)
Step 3: Add metadata (year, topic, question type)
Step 4: Validate with clinical experts

Result: 907 domain-specific question-answer pairs

The benchmark questions ranged from straightforward definitional queries to complex clinical scenarios:

Sample Benchmark Questions:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

"What is the recommended temperature for an operating room?"
→ Type: Factual / Expected precision: High

"How should surgical instruments be sterilized in resource-limited settings?"
→ Type: Context-dependent / Expected precision: Medium

"What are the current guidelines for preventing surgical site infections?"
→ Type: Comprehensive / Expected precision: High

"Can chlorhexidine gluconate be used on mucous membranes?"
→ Type: Clinical safety / Expected precision: Critical
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

This benchmark became our north star. Every technical decision would be evaluated against these real clinical questions.

Choosing the Right Embedding Model: When Domain Expertise Matters

Armed with our benchmark, we could now meaningfully compare embedding models. We tested six candidates:

The Contenders:
- OpenAI Models: text-embedding-3-small, text-embedding-3-large, ada-002
- Scientific Models: SciBERT, PubMedBERT
- General Model: ModernBERT

Here's where it got interesting:

Model Performance on Medical Benchmark:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Model            MRR      P@1      P@3      P@5      Dimensions
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
SciBERT          0.0129   0.0099   0.0143   0.0176   768
PubMedBERT       0.0105   0.0066   0.0110   0.0165   768
ModernBERT       0.0053   0.0000   0.0110   0.0143   768
OpenAI-3-sm      0.0065   0.0044   0.0077   0.0099   1536
OpenAI-3-lg      0.0071   0.0055   0.0077   0.0077   3072
OpenAI-ada       0.0080   0.0066   0.0077   0.0099   1536
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

SciBERT won decisively, with gains ranging from 23% over PubMedBERT to 143% over ModernBERT.

A note on absolute performance: These MRR scores are low in absolute terms—a reflection of the difficulty of retrieval in specialized medical domains. To be clear, pilot testers used this system for research and familiarization purposes, not for real-time clinical decisions in the operating room. Any production deployment for clinical use would require additional validation and appropriate safeguards.

But when we tested these same models on a general benchmark (LOCO), the results flipped:

Performance on General Benchmark (LOCO):
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Model            MRR      P@1      Medical MRR   Difference
─────────────────────────────────────────────────────────────
OpenAI-3-lg      0.5813   0.500    0.0071        -98.8%
OpenAI-3-sm      0.5585   0.500    0.0065        -98.8%
ModernBERT       0.2093   0.100    0.0053        -97.5%
SciBERT          0.1414   0.050    0.0129        -90.9%
PubMedBERT       0.1141   0.000    0.0105        -90.8%
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

The OpenAI models dominated on general content but performed poorly on medical queries.

Domain-specific models excel at medical retrieval but struggle with general content—and vice versa. This crossover pattern drove our model selection.

The reason lies in vocabulary coverage and training data. SciBERT was pre-trained on 1.14 million scientific papers from Semantic Scholar, meaning it had already learned representations for phrases like "perioperative hypothermia" and "surgical site infection prophylaxis." When these terms appear in queries, SciBERT maps them to meaningful vector space positions. OpenAI's models, trained primarily on web text, treat these as rare token sequences—encoding them less precisely. The result: SciBERT finds relevant chunks that OpenAI misses entirely.

From a practical deployment perspective, SciBERT also offered significant speed advantages. Running locally, it processed queries roughly 8x faster than OpenAI's API calls. This isn't a pure model comparison—local inference will always beat API round-trips—but for our deployment, the difference mattered: faster responses at a fraction of the cost.

The Chunking Strategy: Respecting Medical Document Structure

With our embedding model selected, we turned to chunking. Medical documents follow strict conventions, and we needed a strategy that respected this structure.

We tested multiple approaches:

# Strategies tested and results

1. Fixed-size chunks (512 words)     → 681 chunks total
2. Section-based chunks              → 1,205 chunks
3. Hybrid (section + fixed fallback) → 1,329 chunks
4. Original hierarchical             → 1,305 chunks
5. Hierarchical with paragraphs      → 3,056 chunks

The winning approach was hierarchical chunking that respected natural document boundaries. Our evaluation showed clear performance differences:

Chunking Strategy Evaluation Results:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Strategy                    Chunks    Avg Size    MRR      Context Loss
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Fixed-size (512 words)      681       512         0.0089   High (42%)
Section-based               1,205     847         0.0112   Medium (18%)
Hierarchical w/ paragraphs  3,056     326         0.0134   Low (4%)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Hierarchical chunking with paragraph boundaries achieved the highest MRR (0.0134) and lowest context loss—only 4% of chunks had semantic breaks mid-thought, compared to 42% for fixed-size chunking. Here's what it actually does:

# Our hierarchical chunking approach

Level 1: Document Overview (11.7% of chunks)
├── Contains: Title, abstract, authors, metadata
├── Purpose: Document discovery, author search, providing context
└── Size: Up to 2,000 words

Level 2: Sections with Paragraph Boundaries (88.3% of chunks)
├── Contains: Section content split at paragraph boundaries
├── Purpose: Primary source of answers
├── Key feature: Never cuts mid-paragraph
└── Size: Paragraphs within sections, with section title for context

The key insight: we don't arbitrarily split text. Each chunk represents either a complete section (if small enough) or individual paragraphs within that section. Each paragraph chunk includes the section title for context, but the actual content respects natural paragraph boundaries. No random cuts mid-sentence for overlap purposes.

This approach generated more chunks (3,056 vs. 681 for fixed-size) but preserved the semantic coherence of medical information. A clinical procedure described in a paragraph stays together. A research finding isn't split across chunks.

The Retrieval Challenge: When Metadata Defeats Content

We hit a frustrating problem. Our system would sometimes return document overview chunks (just metadata) instead of actual content chunks:

Query: "Are face shields sufficient protection from respiratory droplets?"

Bad Response (overview chunk):
"Title: COVID-19 Safety Guidelines
 Year: 2020
 Authors: Smith, J., Johnson, A."
→ LLM: "I don't have adequate information to answer."

Good Response (content chunk):
"The leaders also recommended that staff members wear a full face
 shield when wearing an N95 respirator to protect their eyes and
 the respirator from contamination. Face shields alone do not
 provide adequate respiratory protection..."
→ LLM: [Detailed answer with proper citations]

The same query would sometimes work perfectly and sometimes fail completely. After extensive debugging, we discovered Weaviate's hybrid search was inconsistently scoring overview chunks higher than content chunks.

Our solution was simple: database-level pre-filtering:

# Filter out overview chunks at query time

where: {
  operator: And,
  operands: [
    {
      path: ["hierarchy_level"],
      operator: NotEqual,
      valueString: "document"
    },
    {
      path: ["chunk_id"],
      operator: NotLike,
      valueString: "*_overview"
    }
  ]
}

The result: 100% consistent content chunk retrieval. We kept the document-level chunks for potential future use cases—author search, document discovery, and providing context for chunk-level results—but for direct Q&A, filtering them out eliminated an entire class of failures. Sometimes the simplest solution is the right one.

Retrieval Strategy: Balancing Precision and Recall

Medical queries require both semantic understanding and exact term matching:

Query A: "What's the policy on OR attire?"
→ Needs semantic understanding ("OR" = "operating room")

Query B: "chlorhexidine gluconate 2% aqueous solution"
→ Needs exact matching (concentration matters!)

We implemented dynamic weighting for our hybrid search:

def calculate_dynamic_alpha(query: str) -> float:
    """Balance vector vs keyword search based on query"""
    alpha = 0.4  # Start balanced
    
    clinical_terms = detect_clinical_terms(query)
    if clinical_terms > 2:
        alpha += 0.1  # More keyword weight
    
    if "guideline" in query.lower():
        alpha += 0.2  # Guidelines use specific terminology
    
    if query.startswith("what is"):
        alpha -= 0.1  # Definitional queries benefit from semantic
    
    return max(0.2, min(0.7, alpha))

For combining results, Reciprocal Rank Fusion (RRF) proved most robust. It sidesteps score normalization issues by focusing on ranks rather than raw scores.

The Pilot: Real Users, Real Feedback, Real Surprises

Setting the Stage

By June 27, 2025, we had:

• 1,629 documents processed
• 277,354 chunks generated
• SciBERT embeddings deployed
• Hybrid retrieval with RRF
• Pre-filtering for consistent results
• 199/199 tests passing

We opened the system to surgical clinicians for a two-week pilot.

The Results: A Tale of Two Systems

Pilot Statistics:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Total Conversations:        33
Feedback Responses:         26 (79% response rate)
Success Rate:               57.6%
Average Satisfaction:       0.53/1.0
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

User Satisfaction Distribution:
Score 1.0 (Excellent):   30.8%
Score 0.6 (Acceptable):  23.1%
Score 0.2 (Very Poor):   38.5%  ← Bimodal distribution
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

User satisfaction showed a bimodal distribution—responses clustered at "Excellent" and "Very Poor" with fewer in the middle. When the system worked, it worked well; when it failed, it failed completely.

The bimodal distribution told a clear story: when the system worked, it worked well. When it failed, it failed completely.

Success Stories

When the system had the right content, it delivered:

Query: "What is the role of the infection preventionist in the OR?"
Response Time: 2.8 seconds
User Score: 1.0/1.0
Result: Comprehensive answer with proper citations

Response times were consistently under 3.5 seconds, with 94% of queries answered in under 5 seconds.

Failure Patterns

The failures were instructive:

Query: "What allergies are considered hidden?"
Expected: Specific clinical article on hidden allergens
Result: "I don't have adequate information..."
User Score: 0.2/1.0

Query: "What was the perioperative staff vacancy rate in 2023?"
Expected: 2023 Salary Survey data
Result: No information available
User Score: 0.2/1.0

Analysis revealed that 43% of failures were due to missing content—the 2024 articles users expected simply weren't in our corpus. The remaining failures broke down into: retrieval mismatches (32%), where relevant content existed but wasn't surfaced; temporal query issues (15%), where users asked about trends or "since 2021" patterns; and edge cases (10%) involving highly specific clinical scenarios.

Note: The "I don't have adequate information" response appeared in both content-gap and retrieval-mismatch categories—but for different reasons. Pre-filtering, it occurred because overview chunks lacked actual content. Post-filtering, it correctly indicated genuine content gaps. Same message, very different root causes.

The Cost Surprise

At $0.00036 per query, the system was remarkably economical:

Cost Breakdown per Query:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
SciBERT (self-hosted):    $0.00001
Weaviate Search:          $0.00002
Redis Cache:              $0.00001
Azure OpenAI:             $0.00032
─────────────────────────────────────────
Total:                    $0.00036
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

We could answer 100,000 medical questions for $36.

Post-Pilot Analysis: Tracking Improvements

After the pilot ended, we added some missing content and continued monitoring the system. Without active users during this phase, we focused on technical performance metrics rather than satisfaction scores. The post-pilot data showed interesting improvements:

Performance Evolution:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Metric              Pilot        Post-Pilot    Change
───────────────────────────────────────────────────────
Success Rate        57.6%        68.4%         +10.8pp
Max Latency         11.06s       6.49s         -41.3%
Response Variance   σ=2.31s      σ=1.42s       -39%
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Specific queries that failed during the pilot now worked:
• "Can chlorhexidine gluconate be used on the vagina?"
• "Can patients be allergic to iodine?"

Others still failed, confirming our content gap hypothesis:
• "What allergies are considered hidden?" (still missing)
• "What was the perioperative staff vacancy rate in 2023?" (still missing)

The post-pilot period provided valuable data about system stability and the impact of content additions, though without active users providing feedback, we couldn't assess satisfaction improvements.

Lessons Learned: Building RAG for Specialized Domains

1. Create Domain-Specific Benchmarks

Generic benchmarks can be misleading. Our domain-specific benchmark, built from actual clinical Q&As, was invaluable. Without it, we might have chosen OpenAI's embeddings (4x better on general benchmarks) and wondered why medical retrieval was failing.

2. Content Completeness Beats Algorithms

Nearly half of pilot failures were due to missing content. No amount of algorithmic sophistication can retrieve documents that don't exist.

3. Domain Models Provide Significant Value

SciBERT's ~2x advantage over general models was transformative. It understood medical abbreviations, drug names, and clinical terminology that general models missed.

4. Respect Document Structure

Medical documents have structure for good reasons. Our hierarchical chunking that preserved paragraph boundaries maintained semantic coherence. A clinical procedure stays together. A research finding isn't split arbitrarily.

5. Simple Solutions Often Win

Our pre-filtering fix was just five lines of GraphQL, but it eliminated an entire class of failures. Always try the obvious solution first.

6. Feedback Loops Need Active Maintenance

During the pilot, we achieved a 79% feedback rate through active engagement. This feedback was invaluable for understanding both successes and failures. Building feedback collection into the core user experience from the start is crucial.

7. Temporal Queries Need Special Handling

"Since 2021" queries require date filtering, ranking by recency, and temporal understanding. We never fully solved this. Build it from the start.

8. Cost Efficiency Surprises

At $0.00036 per query, RAG systems can be incredibly economical. Self-hosted embeddings and aggressive caching are key.

Conclusion

Building this medical Q&A pilot taught us that specialized RAG systems require domain understanding, respect for conventions, and attention to what users actually need.

Our journey from 57.6% to 68.4% success represents dozens of targeted fixes—and crucially, this "success rate" was measured against uncontrolled, real-world queries from actual clinicians, not our internal benchmarks. That distinction matters: your own benchmarks are easy to score well against because you can be sure the right content is in the database somewhere. Real-world usage is messier. Users ask about content you don't have, phrase questions in unexpected ways, and expect the system to understand context you never anticipated. The gap between benchmark performance and real-world success is where most RAG systems struggle.

The system evolved from unpredictable responses to consistent sub-3.5 second performance. We learned that excellence often comes from unglamorous work: ensuring content completeness, respecting document structure, and implementing obvious fixes.

The pilot demonstrated both the promise and challenges of RAG systems for specialized medical content. Each correctly answered query has the potential to impact patient care, making the continued development of such systems worthwhile.

And sometimes, the most impactful improvement really is just five lines of GraphQL.

Acknowledgments: Thanks to the clinical experts who took time to test the system and provide detailed feedback during the pilot.