AI-Enabled Policy Research Platform

Designing trustworthy search and conversational research across 60,000+ DoD policy documents

Illustration of an AI agent as a magnifying glass over a stack of policy documents

Project Background

The Department of Defense’s Chief Digital and Artificial Intelligence Office (CDAO) leads and oversees the acceleration of the DoD’s adoption of data, analytics, and AI to generate decisions. The Advana platform is a modern cloud-hosted data platform that provides a managed data service and a shared analytic capability.

Unstructured Data Search Enhanced by AI/ML (UD-SEAM) is part of the Search Products Portfolio of CDAO. Advana’s flagship product includes an AI-enabled policy research platform. As an AI enabled solution utilizing semantic search API, this platform provides a single place for users across the DoD to search and analyze a corpus of more than 60,000 policy documents from across the DoD and Intelligence Community. This product has over 16,000 monthly users and is Advana’s most widely used application as of September 2024. The product can be used for semantic search, summary generation, FAQ generation, Retrieval Augmented Generation (RAG), and natural-language querying powered by generative AI.

User problems the platform is trying to address:

policies are distributed across many sources;
terminology and document hierarchy make lookup difficult;
users need both exact retrieval and cross-document synthesis;
incorrect or decontextualized answers can affect consequential decisions.

My Role: Lead UX Designer Focus: Product strategy, research, conversational UX, information architecture, trust and source visibility

Technical fluency: semantic retrieval, crawler behavior, and Databricks document ingestion

My Contribution to the production platform

Owned research and product design across search, conversational research, information architecture, source visibility, and uncertain-result handling.
Worked closely with product and engineering through implementation.
Inspected portions of the Python crawler code.
Ran the full document-ingestion pipeline in Databricks.

*The case study combines lessons from my work on a production DoD research platform with an independent design exploration of how agentic workflows could make complex policy research more transparent, controllable, and evidence-grounded. Due to the nature of the project, the design shown in this case study is not the actual UI design created for this project but a design concept based on similar project constraints.

Agentic AI Document Research

I’ve created a specialized AI document research dashboard that coordinates agents for policy repository search and analysis, incorporating RAI (Responsible Artificial Intelligence) principles and accessibility features. The dashboard features a three-panel layout with comprehensive search capabilities, document management, and agent control functionality.

The application includes a search panel with advanced filters for date ranges, document types, and classifications, a central document results area with relevance scoring and AI summarization capabilities, and an agent control panel for monitoring active AI agents, task queues, and system logs. Key RAI features include safety controls, audit logging, explainability toggles, and comprehensive traceability through document metadata and provenance tracking.

• Search & Discovery: Advanced filtering with semantic search capabilities and real-time agent coordination • Document Analysis: AI-powered summarization, classification confidence scores, and accessibility-compliant previews • Agent Governance: Full control over AI agent operations with pause/resume functionality and comprehensive audit trails

Design Principles:

Visible system activity — researchers can understand what the system is doing.
Controllability — users can pause, redirect, or stop agent activity.
Evidence provenance — generated output remains traceable to documents and passages.

Agentic document research dashboard: search filters, document results with relevance scores, and an agent control panel with task queue and logs

Design Decision 1: Conversation Complements, Rather Than Replaces, Search.

For known-document lookup and metadata filtering, classic search provides greater speed and precision.
Exploratory questions and cross-document synthesis are better suited to conversational research.
Forcing every research task into a chatbot is inefficient and reduces predictability.

Platform home offering two entry points: Classic Search for precise lookup and Conversational Research for deep reasoning

Design Decision 2: Make the Evidence More Inspectable Than the Answer

Provide exact passages that support the answer.
Explain why each document was selected based on recency, authority, and applicability.
Let researchers open the complete document to understand its context.
Support comparison across multiple sources rather than presenting the model answer as ground truth.
Communicate when available evidence is incomplete, conflicting, or insufficient.

Research response with a direct answer, key points, confidence score, and a Grounding & Evidence panel of ranked source documents

Visible Retrieval Steps

Rather than presenting model “reasoning” as authoritative, the interface exposes operationally meaningful stages—query interpretation, retrieval, evidence selection, and synthesis—so researchers can identify where an answer may have failed.

Expanded reasoning and retrieval steps — query analysis, document retrieval, evidence synthesis, answer generation — beside the cited relevant passage

Document viewer shows document structure, key topics, entities, related policies and document metadata

Document viewer with structure navigation, semantic highlighting of policy requirements, key topics, entities, related policies, and version history

Document Summary gives users the ability to select and filter documents to summarize and view source policy and different options to view the summary and extracted requirements.

Multi-document policy summary with per-requirement source links, selectable source documents, and an evidence viewer of extracted requirements

Design Decision 3: Make Insufficient Evidence Visible and Actionable

When the corpus cannot support a reliable answer, the system abstains rather than filling the gaps. It distinguishes what the evidence supports from what remains unknown, explains why the retrieved sources fall short—such as recency, relevance, or conflicting guidance—and offers concrete ways to continue the research. This calibrates trust without leaving the researcher at a dead end.

Refusal state: the system abstains, explains what the evidence supports and is missing, diagnoses why sources fell short, and offers recovery paths

Working Across the Research Pipeline

I researched through the research pipeline to understand the sequence from source documents, crawler, ingestion and metadata, retrieval, evidence synthesis and user experience. When a result was missing or unreliable, I could investigate whether the failure was in source coverage, crawling, parsing, metadata, retrieval, or presentation.

Outcomes and Reflection

As of September 2024, the platform served more than 16,000 monthly users and was the most widely used application within Advana at that time. Within that broader product outcome, my design contribution helped:

Create a shorter path to answers by giving researchers task-appropriate entry points for precise document search and exploratory, cross-document questions.
Improve trust calibration by connecting AI-generated answers to exact supporting passages, source metadata, selection rationale, and the complete documents.
Reduce cognitive load through progressive disclosure, clearer information hierarchy, and structured presentation of complex policy content.
Improve discoverability by combining semantic retrieval with familiar search, filtering, and document-navigation patterns.

This work reinforced that interaction quality depends heavily on data quality. Even a clear interface cannot compensate for documents that were not crawled, incorrectly parsed, poorly described by metadata, or missed during retrieval. Working across parts of the crawler and ingestion pipeline helped me understand trust as an end-to-end product property—not simply a UI feature.

Before introducing visible multi-agent controls into production, I would test whether researchers benefit from managing individual agents or primarily need visibility into meaningful system activity, evidence, and intervention points. The system’s internal complexity should be exposed only when it improves user understanding or control.

Last but not least, designing refusal state clarified that trust is set by the worst answer a system gives, not the best one. The screens where the platform performs well are easier to design and less diagnostic; this one determines whether a researcher can rely on the tool when the corpus is incomplete — which, in policy research, is often.

If I were taking this into production, the open question is threshold behavior: whether 0.60 is the right bar across document families, and whether researchers want a hard suppression or a clearly labeled low-confidence answer they can choose to inspect. That is an empirical question, and I would want it settled by testing rather than by my preference.