Independent AI Evaluation & Analytics
We help organisations understand whether their AI works, where it fails, what it costs, and how to make it better.
We don't sell AI platforms. We measure whether they're working, independently — and a score isn't evidence unless you can explain how it was produced.
Sound familiar?
"We've built a RAG application. How do we know it's any good?"
"We're paying for AI tools across the business and can't tell what we're getting back."
"We have masses of customer interaction data and can't see what's happening."
"Our AWS bill keeps climbing. Where is the money actually going?"
What we do
01
Independent evaluation of chatbots, RAG systems and agentic AI. Test sets, failure analysis, LLM-as-judge frameworks and model comparison. What you get: agreed success criteria, a versioned test set, reproducible results, failure analysis and practical recommendations. Where it's useful, we'll also leave you with a regression baseline for testing future changes. The evidence is yours to keep and reproduce independently of Tromp IT.
02
Measure usage, cost, output quality and business outcomes across your AI tooling — from APIs and agents to enterprise AI platforms.
03
Make sense of complex operational and conversational data so you can see what's happening and decide what to do about it. From ETL pipeline design and data cleansing through to dashboards and visualisation.
04
Identify where cloud spend isn't delivering value. Analyse usage and billing, quantify the savings, assess the effort and risk, and work out what is worth changing.
How we work
Start with the problem you're trying to solve and the evidence you need — not a generic evaluation template.
Agree what we're testing, what success looks like, and what you'll get at the end, before any work begins.
Test the system, analyse failures and investigate where and why it behaves unexpectedly — including edge cases and failure modes that simple benchmarks miss.
Give you what you need to decide what happens next — the evidence, not just a verdict.
Two engagement modes. Bespoke evaluation — custom design and execution for a specific system. Platform-based evaluation — where the need is ongoing, structured testing at scale, we can advise on the right platform for it and help you get set up.
Your data stays under your control. We minimise access to sensitive data and prefer synthetic, anonymised or representative test data wherever practical. We don't send client data to third-party AI services without agreement. Where external AI services are used, we use commercial API arrangements that do not use client inputs or outputs for model training by default. If your data can't leave your environment, we'll design the evaluation around that constraint. The method doesn't depend on the technology we use to run it. If you have requirements around data residency, approved providers or where the evaluation runs, we design around them.
Selected work
Tromp IT works directly with organisations and as a specialist subcontractor within consulting and delivery teams. Other than Ada, which is our own, the engagements below were delivered through Dirawong.
The assistant in the corner of this page is also a test subject. One 189-question set was run against seven model backends; every answer was marked independently by a four-model judge panel, and every tool call was checked in code against what the runtime actually did rather than what the reply claimed. The published comparison covers accuracy, groundedness, latency and cost per thousand conversations — and reports the questions the panel could not agree on instead of resolving them quietly.
Developed a medical triage chatbot for the Nurse on Call service using Cognigy, and built an automated testing platform for it, covering both flow testing and regression testing.
Worked as a data scientist tuning conversational language and intent models for enterprise banking assistants, including builds for Westpac, BT (Bankers Trust) and Standard Chartered Bank. Focus on intent classification, model performance and language coverage.
Engaged as an analytics consultant to build the data models measuring how Telstra's voice recognition systems performed in production — not how they scored in the lab, but what happened when real callers used them.
Played a key role in shaping Dirawong's AI testing and validation platform for LLM-powered chatbots, RAG pipelines and agentic AI systems. Contributions include multi-judge evaluation frameworks, LLM-as-judge approaches and automated test generation.
Analysed an enterprise AWS environment to identify and quantify cost optimisation opportunities across compute, storage, databases and service commitments, working within Araza's delivery team. Produced a prioritised plan based on potential savings, implementation effort and risk.
About
Founder & Consultant
Phil has worked in technology, analytics and customer systems for more than 30 years. His current focus is AI evaluation, conversational AI and analytics — helping organisations understand whether their AI systems are actually working, what they cost, and where they fail.
He's particularly interested in the gap between what technology is supposed to do and what happens when real people actually use it.
That question has shaped his work across IVR and speech recognition, enterprise conversational AI, data science and now LLM evaluation.
Ask Phil
Ask our AI assistant — built and evaluated using the same methods we apply for clients.
Claude Sonnet 5 on brief 56fa584cd172 (evaluation v3). Scores are adherence to that brief and are not comparable across brief versions.
Hi — I'm Phil's AI assistant. Tell me about your situation and I'll let you know whether Tromp IT could help.
Ada is an AI assistant — your messages are sent to Anthropic's API to generate a reply. Please don't include confidential or sensitive information.
Prefer email? phil@trompit.com.au · LinkedIn · Melbourne, working nationally