AI response evaluation framework for multi-agent RAG
Designed an evaluation framework to systematically measure the accuracy, relevance, groundedness, and safety of AI responses generated by Aware Super's multi-agent RAG system over Azure AI Search, benchmarked against a curated golden dataset.
Role: AI Architect
- quality gates evaluated: routing, retrieval, response
- 3
- quality gates evaluated: routing, retrieval, response
The approach
Designed an evaluation framework to systematically measure the accuracy, relevance, groundedness, and safety of AI responses generated by Aware Super's multi-agent RAG system over Azure AI Search, benchmarked against a curated golden dataset. Defined evaluators mapped to the system's three quality gates — routing, retrieval, and response — with alerting to flag incorrect, hallucinated, or unsafe answers before deployment. Delivered supporting artifacts including implementation samples, presentations, evaluation-measure analyses, and a Jira-ready backlog for CI/CD-integrated, automated regression testing.
Related projects
- BHPMining2025
Agentic AI automation for hire-to-retire HR processes
Designed and delivered an Agentic AI automation framework in ServiceNow to streamline end-to-end HR request processing.
- manual steps eliminated
- 17
- manual steps eliminated
- HR workload removed
- 40 FTE
- HR workload removed
DatabricksServiceNowOpenAI (BYOM)Agentic AIAzure Foundry - Confidential clientMining2026
Independent audit of an enterprise Databricks Lakehouse
Led a 10-day independent audit of a global mining company's enterprise Databricks Lakehouse platform across three pillars (Ingestion, Data Governance, Technical Governance), delivering evidence-backed findings from production environment analysis of 16,000+ tables across two AWS regions and three stakeholder workshops.
- tables analysed across two AWS regions
- 16,000+
- tables analysed across two AWS regions
- remediation roadmap delivered
- 21-point
- remediation roadmap delivered
DatabricksUnity CatalogDelta LakeDelta SharingAWS (multi-region) - NSW Government (DCCEEW)Government2026
Secure, governed Google Cloud AI sandbox
Built a governed, security-hardened Google Cloud environment for AI/Vertex AI PoCs, fully as Terraform IaC — Shared VPC with default-deny egress + per-PoC isolation, least-privilege IAM (service-account impersonation), org-policy guardrails, IAP bastion, audit logging and cost budgets, with a reusable project factory to onboard new teams in minutes under security guardrails by default.
- to onboard a new team, guardrails by default
- Minutes
- to onboard a new team, guardrails by default
TerraformGCP Vertex AIGCP Shared VPCGCP Cloud NATGCP IAM