{"id":111604,"date":"2026-07-10T12:00:00","date_gmt":"2026-07-10T12:00:00","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=111604"},"modified":"2026-07-07T14:23:59","modified_gmt":"2026-07-07T14:23:59","slug":"how-to-automate-fabric-data-agent-testing-with-promptfoo-and-github-actions","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/data-analytics\/microsoft-fabric\/how-to-automate-fabric-data-agent-testing-with-promptfoo-and-github-actions\/","title":{"rendered":"How to automate Fabric data agent testing with promptfoo and GitHub Actions"},"content":{"rendered":"\n

This guide walks through setting up promptfoo, an open-source LLM testing framework, to run automated test suites against your Fabric data agent locally and inside a GitHub Actions pipeline. The result? Every prompt, assertion, and regression check will run on its own, every time you push a change<\/em>.<\/strong><\/p>\n\n\n\n

Why testing data agents is essential (and introducing a great way to automate the process)<\/h2>\n\n\n\n

If you read my previous article on why DataOps matters in the age of AI<\/a>, you already know I believe testing data agents is essential. But what I didn’t have a great answer for at the time was how to automate that testing in a repeatable, scalable way – especially one that could slot into a CI\/CD<\/a> pipeline.<\/p>\n\n\n\n

Now, with Microsoft’s recent introduction of service principal support with Fabric data agents<\/a>, we have the answer.<\/p>\n\n\n\n

This opens up a lot of possibilities, with my favorite being around continuous integration<\/a> and automated testing<\/a>. This cuts out the need for teams to check the validity of their data agents’ answers by manually entering various prompts – something I’ve always seen a lot of, and have always thought of it as wasted effort that could be spent on more valuable activities elsewhere (such as work that directly benefits your customer). <\/p>\n\n\n\n

After all, every hour a developer spends clicking through prompts is an hour not<\/em> spent building the next feature, or fixing a real defect. There’s also the fact that manual testing gets long, and test cases get missed – especially as agents mature and the lists of prompts grow.<\/p>\n\n\n\n

Plus, the underlying data and the model supporting these agents can change as the semantic model scheme evolves, data refreshes, or even if Microsoft updates (or retires) the LLM<\/a> (large language model) powering the agent. Manual testing will<\/em> miss regressions that happen between reviews.<\/p>\n\n\n\n

So, wouldn\u2019t it be nice if, with a single command, we could run multiple prompts and get confirmation that the agent responded appropriately? Even better, what if that command could run inside a GitHub Action<\/a> so that, whenever the agent is updated and synced to source control, the tests execute automatically?<\/p>\n\n\n\n

Well, now we can – and in this article, I’ll show you how.<\/p>\n\n\n\n

Introducing promptfoo (what it is, and how it works)<\/h2>\n\n\n\n

The testing library that makes this possible is promptfoo<\/a>. Recently acquired by OpenAI<\/a>, promptfoo is one of my favorite tools for automating the testing of interactions with LLMs.<\/strong> At its core, promptfoo revolves around a concept called providers<\/strong>. These are modules that serve as the interface between the testing framework and an AI tool – whether that\u2019s Claude<\/a>, Azure AI Foundry<\/a>,<\/strong> or another.<\/strong><\/p>\n\n\n\n

While Promptfoo ships with standard providers for the most popular services, it also lets you write your own custom provider<\/strong>. And this is exactly what we\u2019ll do to connect to the Fabric<\/a> data agent.<\/p>\n\n\n\n

\"An
Overview of the promptfoo architecture, showing how a custom provider bridges the testing framework and the Fabric Data Agent<\/em><\/figcaption><\/figure>\n\n\n\n

This flexibility is what makes promptfoo so powerful. We define our prompts and the assertions we want to validate, such as \u201cthe response must mention revenue\u201d, or \u201cthe question is not valid to this agent\u2019s purpose\u201d. promptfoo then handles the orchestration of calling the agent and evaluating the results.<\/p>\n\n\n\n

How to use promptfoo<\/h2>\n\n\n\n

I’ll now walk through exactly how to use promptfoo – from getting started and the basics, to running tests automatically. First, here’s exactly what you need to use promptfoo.<\/strong><\/p>\n\n\n\n

Everything you need in order to use promptfoo<\/h3>\n\n\n
\n