Building a trust scoring system for LLM-assembled healthcare data using Databricks
By: Praveen Sundaresan Ramesh, Sidhant Guliani, and Amine Jallouli.
The Challenge
The Databricks Hackathon for Good, in partnership with OpenAI, was a 36-hour challenge that kicked off on the first day of the Data & AI Summit 2026. The dataset was assembled by Databricks in collaboration with the Virtue Foundation, a nonprofit organization working to deliver quality healthcare to those in need.
Dataset and Brief
At the heart of the solution was a single central dataset: 10,000 healthcare facility records from across India, containing 51 columns of both structured data (facility name, state, location, specialties) and unstructured data (capabilities, procedures, and equipment).
Every participating team received this same raw dataset. The challenge was to transform it into a Databricks App that empowered non-technical users, such as healthcare planners, NGO coordinators, and analysts to ask questions and make confident, data-driven decisions.
Teams could choose one of the four tracks:
- Facility Trust Desk: Can the facility actually do what it claims?
- Medical Desert Planner: Where are the highest-risk gaps in care?
- Referral Copilot: Where should a patient actually go?
- Data Readiness Desk: What needs to be fixed before this dataset can be trusted?
Decision
A few hours spent exploring the data, combined with a conversation with the person who assembled it, revealed some inconsistencies. The most striking issue was the presence of conflated records: a single record often represented data from several distinct hospitals merged together, with contradictory locations and capabilities borrowed from unrelated facilities.
We quickly realized that all four tracks shared a common dependency, the data. Any errors in the foundation would propagate through everything built on top of it. We chose the Data Readiness track because data is the foundation for every other question that stands on. A trust desk, a gap map, and a referral shortlist are only as reliable as the records beneath them.
We built the Data Readiness Desk to answer one question: which records are wrong, how wrong are they, and which ones matter most to fix? This post walks through our approach, the Databricks tools we used, and why we chose them.
The Solution
A planner cares whether a hospital truly offers services like dialysis, intensive care, or oncology. To address this, we built a trust score for every facility-specialty pair, rather than just grading the facility as a whole. Thus, our unit of trust is a (facility, specialty) claim. For instance, if Aravind Eye Hospital claims to offer gynecology, our trust score estimates how reliable that specific claim is.
Our first instinct was to let an AI model judge each claim outright. But a verdict with no explanation behind it is exactly what a planner cannot trust or act on. Instead, we used AI to make sense of the unstructured text and a transparent rule-based scoring system to make the final call.
On average, each facility listed 11 specialties. We therefore exploded the original dataset of 10,000 facilities into roughly 118,000 individual claims. For each of these claims, we calculated a weighted score as shown below:
trust = C × (0.45·E + 0.15·A + 0.15·I + 0.15·S + 0.10·R) - C · Coherence: Checks whether a record truly describes one real facility using geographic indicators (zip code, latitude-longitude, address) and number of distinct sources.
- E · Evidence: Is the mentioned specialty also seen in other columns like procedures, capabilities, or equipment?
- A · Authority: Weighs how credible the underlying sources are, boosting official and government websites.
- I · Independent corroboration: Can the claims be confirmed by several independent sources?
- S · Spatial consistency: Whether the zip code, coordinates, and address agree on the location.
- R · Recency: How recently was the information updated?
All weights are defined in a single config file that can be tuned by working with the field experts. The LLM’s role is limited to interpreting unstructured text and extracting evidence. Everything downstream, from scoring to persistence and serving, runs entirely on Databricks, as shown in the architecture diagram.
Unity Catalog
The raw dataset was sourced from Databricks Marketplace as a Delta table in the bronze layer. A scoring pipeline then processes a cleaned version of the table and writes multiple tables back into the catalog: cleaned facilities, a column profile, a review queue, per-facility issues, scored claims, and a readiness scorecard. Keeping the raw input and curated analytics in one catalog creates a single boundary for governance, lineage, and access control. This also ensures that Genie (Databricks’ NL to SQL Engine) and Lakebase (Databricks’ OLTP service), always read from the same set of tables, eliminating any risk of inconsistency across the application.
AI Query
Calculating the Evidence score required more than a simple keyword search. The same specialty can be described in dozens of different ways across procedures, capabilities, and equipment fields. To capture this reliably, we turned to AI Query, a general-purpose AI Function on Databricks. With 118,000 claims to score, AI Query’s built-in parallelization, automatic retries, and scaling capabilities proved essential without the need for a model-serving harness. One particularly useful configuration for large workloads like this was the ability to disable failure on error. This ensured that the job completed successfully, with failed rows flagged for review rather than stopping the pipeline entirely.
Genie
A reviewer shouldn’t have to write SQL to ask “Which states have the most flagged facilities?” We set up Genie, Databricks’ natural language interface, across the five analytical tables and connected it to the app’s assistant. This enables reviewers to ask questions in plain English and gain insights directly from the data. Genie translates each question into a database query, runs it automatically on a SQL Warehouse, and returns the results. The underlying query is displayed alongside the results, so every answer is transparent and verifiable, rather than presenting numbers with no traceable source.
OpenAI Agent
Genie is one of two tools available to the AI Assistant built into the web application, powered by OpenAI’s GPT model. The assistant has access to a web-search tool to search for external reference material such as clinical guidelines or medical standards. For each question the planner asks, the assistant decides which tool to use, combining multiple lookups in sequence where needed, and displays its progress in real-time in the chat interface. Ask “Are the Oncology facilities in Maharashtra ready to be trusted for referrals?” and it pulls the relevant trust scores from the internal tables, retrieves applicable clinical standards from the web, and compose a single, cohesive, and cited answer.
Lakebase
Dashboard and queues need to load instantly, and reviewer decisions need to be saved reliably the moment they are made. Traditionally, this would be achieved by two separate systems. Lakebase, Databricks’ built-in transactional database solution, helps with both requirements. The six analytical tables are automatically kept in sync with the underlying data for fast reads, while reviewer decisions are written directly to a dedicated table on Lakebase. We ran on Lakebase Autoscaling, which is the default option today, with all data access flowing through a single secure connection using automatically expiring credentials generated fresh for each session.
Databricks Apps
What ties everything together and makes the platform intuitive to use for the planner is a simple, easy-to-use web application. This typically involves a backend server, a frontend application, authentication, and a web host, all maintained separately from the data. Databricks Apps removes this overhead completely. The entire product ships as a single application living directly alongside the data it reads, inheriting the same security and access controls automatically. Resources like the data warehouse, database, and API keys are declared once and injected securely at runtime. A single deployment command puts the application live for the planner to use.
The Conclusion
Data quality is rarely the most exciting problem to solve, but it is almost always the most consequential. In 36 hours, our team built a system that doesn’t just flag bad data, it explains why a record is untrustworthy, quantifies how much it matters, and puts the right information in front of the right person to act on it. Databricks made that possible at this pace. Having data storage, AI functions, governed analytics, natural language querying, and application hosting all on one platform meant we spent our time solving the problem rather than managing the infrastructure around it. Winning the Databricks Hackathon for Good was a reflection of that focus: fix the foundation first, and everything built on top of it becomes more reliable. For the Virtue Foundation and the healthcare planners, NGO coordinators, and analysts who will ultimately use this data, we hope the Data Readiness Desk is a step toward decisions they can trust.
References
Cavallo Technologies
Cavallo Technologies is a Databricks-exclusive firm that helps enterprise teams design and build production-grade AI systems on the Databricks platform. If your organization is exploring this, we would love to discuss your architecture and how we can help.