By: Sidhant Guliani, Praveen Sundaresan Ramesh, Ayush Sharma, Kulaphong (Yok) Jitareerat.
Summary
- Learn how we helped our client modernize access to vessel knowledge and operational data with a multi-agent AI assistant, delivered iteratively from a two-week proof of concept to production, with a clear roadmap for what’s next.
- Discover how Cavallo Technologies leveraged Databricks to deliver a modular agentic framework; combining RAG-based knowledge assistants and Genie spaces for text-to-SQL, with governed per-user data access surfaced directly inside Microsoft Teams.
- Explore the technical architecture: a supervisor agent coordinating specialized sub-agents, an automated evaluation pipeline, a CI/CD-driven deployment across dev, test, and production environments, and a complete monitoring process spanning usage, cost, and user feedback.
Fragmented knowledge and data slowing down vessel operations
For one of the world’s leading independent owners and operators of containerships, frontline teams needed fast, reliable answers about vessel demographics, operational performance, and technical documentation. Structured operational data was centralized in Databricks but effectively out of reach for non-technical staff, who depended on the BI team and analysts for nearly every routine query. Meanwhile, vessel manuals, standard operating procedures, and market news were scattered across unstructured PDFs and emails.
What the client needed was a unified way to let an authorized team member ask a question and get an accurate and sourced answer right away, in a tool that they already use: Microsoft Teams.
Building a modular agentic AI framework on Databricks
We partnered with the client, designed and delivered an agentic AI assistant built entirely on the Databricks Data Intelligence Platform. Rather than architecting a single purpose chatbot, we built a modular framework with a supervisor agent that routes each question to the right specialized sub-agent, so we can add new capabilities as new needs come up. Today the agent spans several specialized assistants, all coordinated by a single supervisor:
- Several RAG-based knowledge assistants, spanning documents that ranged from unstructured internal documents to structured reports – using ai_parse_document for ingestion and Databricks Vector Search for retrieval.
- Multiple Genie Spaces over structured fleet data for natural-language Text-to-SQL queries, covering vessel information, operational and IoT data, crewing data, navigation information, and more.
- A Visualization Assistant that turns data already retrieved in a conversation into publication-quality charts or interactive data tables.
- A supervisor agent powered by Claude Sonnet 4.6 and orchestrated with LangChain ReAct, which coordinates the sub-agents and returns a single, coherent answer.
- Governed, per-user access through Microsoft Entra ID On-Behalf-Of (OBO) authentication, so every response is scoped to what the signed-in user is permitted to see.
- A feedback loop in Lakebase that captures user sentiment on each answer – whether the response was helpful. So, we can see which replies land well and where the agent needs tuning.
- Full observability and cost control for every call. A Unity AI Gateway tracks every model call. Inference and billing data land in governed tables that feed an AI/BI dashboard, giving the client clear visibility into both cost and performance as usage grows.
The agent is available directly inside Microsoft Teams, where team already collaborates.
What Made This Implementation Special?
- Modular by design. The supervisor-plus-sub-agents architecture means we can add new Genie Spaces and knowledge domains without rebuilding the assistant. Each module evolves independently, and the framework sets the stage for continued expansion across the enterprise.
- Quality you can measure. We curated a “golden” evaluation dataset of 200+ business questions and expected answers to anchors agent’s quality from day one; during development, an LLM-as-a-Judge scores agent responses and tool calls with results logged to a Delta table and MLflow. The pipeline reruns after each new iteration of agent, whether there is a model change, an added subagent or a prompt refinement. With this, every change is measurable rather than anecdotal, and we can verify routing and answer accuracy before promoting to production.
- Governed access by default. A best practice is to ensure that data access respects existing entitlements. Through On-Behalf-Of (OBO) authentication, Agent queries Databricks as the signed-in user, so responses are automatically scoped to that user’s permissions. With Entra federated to Databricks, the same security groups consistently control access across both Databricks and the Teams app.
- Built for production from day one. We ship the agent through an established CI/CD process using GitHub Actions and DABs, with isolated dev, test, and production environments. We also delivered agent-level and infrastructure documentation alongside the code, so the client’s teams can operate and extend the platform confidently.
- Delivered iteratively, and fast. Because the whole stack sits on one platform, the development could be done quickly. Data, RAG, Genie spaces, evaluation, and governance, everything lives on a single platform, Databricks. This makes it easy to develop a proof of concept in week one, a v1 with CI/CD, Teams integration, and Entra OBO by week four, and adding more Genie Spaces, Lakebase-backed feedback, and AI/BI dashboards by week eight.
Architecture at a Glance
The architecture brings every component required for the agent – document ingestion and parsing, vector search, agent orchestration, model serving, evaluation, and cost monitoring – onto a single governed platform, with Microsoft Teams as the user-facing experience and Microsoft Entra ID controlling access.
Why Databricks Made the Difference
When we scoped the agent, we faced a familiar enterprise-AI tension: move fast but never compromise on data governance. Wiring together separate tools for ingestion, RAG, Genie Spaces, serving, and evaluation would have meant weeks of integration work, and more moving parts to secure and govern.
Databricks collapsed that complexity. The same platform hosts the Gold-layer operational data, the document parsing and vector search for RAG, the Genie Spaces for Text-to-SQL, the model serving endpoints, the AI Gateway, the Lakebase feedback store, and the evaluation framework. Unity Catalog manages permissions across all of it from a single place, we don’t have to set up access controls separately in each tool.
Having all the key components: data, AI, governance, and orchestration in one environment is what let us ship a production-grade agent fast – and keep building on it since, while the system stayed secure and easy to maintain.
Results So Far
The Agent’s releases have delivered measurable outcomes:
- Self-service access for 200+ employees.
- Most routine data requests diverted away from the BI team, freeing specialists to focus on their core work, saving thousands of hours annually.
- Query turnaround cut from multi-day waits to a few seconds for everyday requests.
Cavallo Technologies
At Cavallo, a Databricks-exclusive firm, we help enterprise teams design and build production AI systems on Databricks. If your team is working on this, we will be happy to dig into the architecture with you.
Ready to bring agentic AI to your operations? Discover how Databricks’ unified platform can help you move from concept to production and keep scaling – without compromising on governance.