{"id":109725,"date":"2026-05-08T12:00:00","date_gmt":"2026-05-08T12:00:00","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=109725"},"modified":"2026-05-05T14:25:26","modified_gmt":"2026-05-05T14:25:26","slug":"how-dataops-helps-reduce-gen-ai-risk-improve-data-quality","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/ai\/how-dataops-helps-reduce-gen-ai-risk-improve-data-quality\/","title":{"rendered":"How DataOps principles help to reduce GenAI risk and improve data quality"},"content":{"rendered":"\n

John Kerski explains how DataOps principles improve trust, quality, and governance when using GenAI in data solutions. Includes John’s advice on how to apply these principles – plus tips for Git, testing, Microsoft Fabric, and more.<\/strong><\/p>\n\n\n\n

Over the past three years, the presence of Generative Artificial Intelligence (GenAI)<\/a> in the world of data has profoundly changed<\/a> how we build solutions. I’ve seen teams incorporate GenAI into their development processes as vendors continue introducing AI tools such as GitHub Copilot<\/a> and Copilot for Fabric<\/a>. The dependence on these tools to accelerate work is not much different from what IntelliSense<\/a> and integrated development environments (IDEs)<\/a> did years ago as the industry moved away from punch cards<\/a> and assembly.<\/p>\n\n\n\n

Yet, from what I have seen, AI only exacerbates existing problems with data and the processes we use to build solutions. AI is another tool we have, but productivity and trust in what our solutions produce can quickly be doused when it’s used incorrectly. Here are some of the issues I’ve seen with AI in data solutions in the past two years alone.<\/p>\n\n\n\n

AI updates code with no audit trail<\/h3>\n\n\n\n

I’ve seen teams use AI to update the likes of SQL<\/a>, Power Query<\/a> and Python<\/a> without<\/em> considering how they’d roll back those changes<\/a> if AI introduced a mistake. Whether it’s the model, poor constraints in prompting<\/a>, or context rot<\/a>, where the model no longer \u201csees\u201d prior information that would make it more effective, AI can make updates to code that introduce errors. How do you identify what changed and then roll those changes back?<\/p>\n\n\n\n

AI updates code with no safety net<\/h3>\n\n\n\n

Let\u2019s be clear – GenAI is built and trained by humans, and humans are fallible. So, when you ask AI to build a new set of data transformations<\/a> from an API source, should you trust it implicitly? How do you know the code it builds handles 400 errors gracefully, or backs off appropriately when it receives a 429 error? If I replaced the word AI with \u201cjunior data engineer,\u201d would you answer those questions differently? From my experience, the answers should not<\/em> be different.<\/p>\n\n\n\n

AI answers your client\u2019s questions with little oversight<\/h3>\n\n\n\n

I’ve encountered situations where agents in Copilot<\/a> or Data Agents in Microsoft Fabric<\/a> answer questions inconsistently. They may even answer questions they shouldn’t. Asking a Financial Data Agent for a good brownie recipe is not the desired outcome!<\/p>\n\n\n\n

How DataOps principles help with AI usage<\/h2>\n\n\n\n

These issues are exactly why DataOps<\/a> is more important than ever for your project teams. DataOps is a set of principles for reducing production errors while increasing the delivery of data solutions. The wonderful thing about principles is that they apply regardless of the tools or technologies involved. <\/p>\n\n\n\n

So, I’d like to offer a few principles you should make inherent to your teams\u2019 work. I’ll also include some tips for Fabric and Power BI<\/a> that you should be applying with GenAI today. I put them in order so you can focus on one principle at a time, each over a 1-2 month period.<\/p>\n\n\n\n

Principle #1: Make it reproducible<\/h3>\n\n\n\n

Reproducible results are required and therefore we version<\/strong> everything. That means data, low-level hardware and software configurations, and the code and configuration specific to each tool in the toolchain.<\/p>\n\n\n\n

Tip #1: Embrace Git<\/h4>\n\n\n\n

Notice that I bolded the words ‘we version’ above. In our industry, that means using Git<\/a>. Git is fundamental to giving your teams peace of mind that, whether AI updates a notebook or a Power BI report, you know exactly what<\/em> changed (and when.)<\/p>\n\n\n\n

There is<\/em> a learning curve to Git, yet features like Fabric Git Integration<\/a> make it easier than ever to save versions of your work. Fabric also provides support for version control<\/a> with both Azure DevOps and GitHub<\/a>.<\/p>\n\n\n\n

For many teams, Git represents the steepest part of the learning curve. However, once your team builds the habit of cloning, committing, syncing, and merging their changes – and treats the repository as the single source of truth rather than the workspace – you’ll have a solid foundation for the DataOps principles that follow.<\/p>\n\n\n\n

\n
\n
\n
\n

Enjoying this article? Subscribe to the Simple Talk newsletter<\/h3>\n
\n Get selected articles, event information, podcasts and other industry content delivered straight to your inbox. <\/div>\n <\/div>\n Subscribe now<\/a>\n <\/div>\n <\/div>\n<\/section>\n\n\n

Principle #2: Improve cycle times<\/h3>\n\n\n\n

We should strive to minimize the time and effort required to turn a customer need into an analytic idea. We should create it in development, release it as a repeatable production process and, lastly, refactor<\/a> and reuse that product.<\/p>\n\n\n\n

Tip #2 – Identify how you use AI to update your code<\/h4>\n\n\n\n

The companies providing these GenAI tools are of course interested in growing their customer base. That’s why they offer personal productivity plans. However, are you aware of the terms of service attached to these plans? They’re likely less strict on data residency and privacy<\/a>, especially the free ones (because free is not really free<\/em>.) Do you know how many people on your teams are using their own personal plans? Many of these GenAI tools have access to the data and code you’re working on, and then send that data to centers and logs around the world.<\/p>\n\n\n\n

I don’t intend to scare you away from using GenAI but you should consider these security aspects carefully. Fortunately, many enterprise-grade<\/a> tools have different terms of service that are more favorable to company data privacy. Tools like GitHub Copilot Enterprise, for example, lets you isolate GenAI models to ones deployed in your Azure tenant <\/a>with Foundry<\/a>. <\/p>\n\n\n\n

This isn’t a new concept that makes GenAI a security pariah, though. Many in our industry remember the security concerns around ‘Bring Your Own Device’<\/a> when mobile device usage dramatically increased. That was another tool that accelerated productivity and rankled IT security personnel.<\/p>\n\n\n\n

Ultimately, as they did with ‘BYOD’, teams just need to consider the security concerns and implement practices to mitigate the risks. Remember, also – the tools to manage these risks will<\/em> get better, so reassess the situation often.<\/p>\n\n\n\n

\n
\n
\n
\n

Protect your data. Demonstrate compliance.<\/h3>\n
\n With Redgate, stay ahead of threats with real-time monitoring and alerts, protect sensitive data with automated discovery & masking, and demonstrate compliance with traceability across every environment. <\/div>\n <\/div>\n Learn more<\/a>\n <\/div>\n <\/div>\n<\/section>\n\n\n

Tip #3 – Use Visual Studio Code to aid development with Fabric<\/h4>\n\n\n\n

Visual Studio Code<\/a> makes saving work to Azure DevOps<\/a> and GitHub much easier. There are also a cadre of extensions that make working with Fabric easier, including Fabric Data Engineering<\/a> and Microsoft Fabric MCP<\/a>.<\/p>\n\n\n\n

Plus, for Power BI Desktop<\/a> development, VS Code is becoming a complementary tool that lets AI make changes to your models and reports through the Power BI Modeling MCP<\/a>.<\/p>\n\n\n\n

Tip #4 – Implement workspace governance<\/h4>\n\n\n\n

With Git in place, you can ensure your development work is separate from what your customers see. At a minimum, you should keep two workspaces: one for development and one for production. If you can afford it, you should also have separate Fabric capacities for production and development. That way, production won’t be impacted if you make a mistake, such as a notebook mistakenly running a merge of large tables that consumes a lot of capacity. <\/p>\n\n\n\n

It also means you should have a capacity (albeit a smaller one like an F4) for Copilot for Fabric. AI assisting with building code or answering questions should not<\/em> come at the detriment of processing data. Analytics work is a volatile aspect of Fabric consumption – it’s hard to predict because it’s an exploratory endeavor, and asking AI to help explore is just as unpredictable. Keep your data engineering Fabric consumption separate from your analytics consumption.<\/p>\n\n\n\n

Principle #3: Quality is paramount<\/h3>\n\n\n\n

Analytic pipelines should be built with a foundation capable of automated detection of abnormalities (jidoka<\/a>), security issues in code, configuration, and data. It should also provide continuous feedback to operators for error avoidance (poka- yoke<\/a>).<\/p>\n\n\n\n

Tip #5 – See an error? Build a test<\/h4>\n\n\n\n

Testing is of utmost importance when AI is used in code generation. Without testing, how do we prove that AI didn’t introduce a mistake or fail to handle our requirements? <\/p>\n\n\n\n

Whether AI accidentally changed a data type that broke a relationship, or didn’t use the DAX<\/a> function TREATAS<\/code> on the right column for a DAX measure, the solution is the same: build a test. And it’s no different if one of your team makes a mistake. Generating a test is the first <\/em>thing you should do to help prevent it from happening again. Over time, you’ll build up a ‘safety net’ of tests to help prevent future mishaps.<\/p>\n\n\n\n

Tip #6 – Use Pytest, wheel files, and environments for notebook development<\/h4>\n\n\n\n

Python has long had the ability to test transformations. When you have AI build Python transformations, you should have defined tests in pytest<\/a> to validate those transformations. When those tests pass, the Python code should be compiled into wheel files<\/a> and then added to an environment. This encapsulates your code with sound testing practices, reduces AI-driven regression errors, and protects the version of that code in environments.<\/p>\n\n\n\n

Note: I have a sample project<\/a> you can use as an example to help you get started.<\/p>\n\n\n\n

Tip #7 – Test your semantic models<\/h4>\n\n\n\n

With the advent of the Power BI project file format<\/a> and the introduction of the Power BI Modeling MCP server (which gives GenAI tools the ability to directly update your model), I’ve seen teams let AI update DAX and Power Query without testing. But how do you know what AI updated is correct? <\/p>\n\n\n\n

Well, you can use the DAX Query View Testing Pattern<\/a> and User Defined Functions<\/a> such as PQL.Assert<\/a> to build tests against your model. You can validate that the content has the correct distinct columns, that DAX measures output consistent results under certain filters, and that the relationships in a model are preserved, all within the semantic model. This lays the foundation for automated testing.<\/p>\n\n\n\n

Tip #8 – Test your data agents<\/h4>\n\n\n\n

The Fabric Data Agent SDK<\/a>, built to be run in a notebook<\/a>, can be used to impersonate a customer asking questions so you can inspect the results and ensure they are consistent and appropriate. You should also be testing with inappropriate questions.<\/p>\n\n\n\n

Regarding this, your leadership should understand that the questions stored in the notebook may be unsuitable in pleasant conversation, but are necessary to make sure the agent responds appropriately. For example, by asking an agent to give the definition of a curse word, you can validate that the agent does not<\/em> return the word. It should instead simply say it cannot answer the question.<\/p>\n\n\n\n

\"\"
Testing a Data Agent with various questions<\/em><\/figcaption><\/figure>\n\n\n\n

Principle #4: Monitor for quality and performance<\/h3>\n\n\n\n

Our goal is to have performance, security, and quality measures that are monitored continuously to detect unexpected variation and generate operational statistics.<\/p>\n\n\n\n

Tip #9 – Testing does not stop once the solution has shipped!<\/h4>\n\n\n\n

To find issues before your customers do, it’s crucial to test and track your data’s journey to the customer during every step of the process. Eventhouse Monitoring<\/a> allows workspace admins to start getting real-time insights into refreshes, and I have a template<\/a> that can help.<\/p>\n\n\n\n

This best practice also includes Data Agents<\/a>, Fabric\u2019s AI implementation for chatting with data. Both the data the agent queries and<\/em> the model used to infer answers can, and will, change (including model deprecation<\/a>.) The notebook I referenced earlier<\/a> can also be used to test the agent and log results. Furthermore, these results can be logged to an eventhouse – giving you near real-time insights into the behavior of the Data Agent.<\/p>\n\n\n\n

As of March 2026, seeing the prompts and conversations used by users with Data Agents is not<\/em> built into the Fabric product. I’d instead suggest using Copilot Studio<\/a>. While there is an additional cost, it does have more robust options<\/a>. My hope is that we start seeing user activity sent to Eventhouse Monitoring in the near future.<\/p>\n\n\n\n

In summary: why you should use DataOps principles with GenAI<\/h2>\n\n\n\n

GenAI has the capability to accelerate the delivery of solutions, and DataOps provides the principles to keep teams from crashing (both technically and personally). GenAI will continue to improve, introducing new capabilities that disrupt the industry. At the same time, DataOps principles are still as useful as ever. I hope, as a result of this article, you consider how they can make your teams better in these early days of GenAI.<\/p>\n\n\n\n

\n

FAQs: How DataOps principles help to reduce GenAI risk and improve data quality<\/h2>\n\n

1. How does Generative AI (GenAI) impact data engineering workflows?<\/h3>\n
\n

Generative AI accelerates development but can introduce errors, inconsistent outputs, and governance challenges if not properly managed.<\/p>\n <\/div>\n

2. Why is DataOps important when using AI in data solutions?<\/h3>\n
\n

DataOps ensures reproducibility, quality, and monitoring, helping teams reduce errors and maintain trust in AI-assisted workflows.<\/p>\n <\/div>\n

3. What are the risks of using AI-generated code in data projects?<\/h3>\n
\n

Risks include lack of audit trails, missing error handling, security concerns, and untested code changes that may break pipelines.<\/p>\n <\/div>\n

4. How can teams safely use AI tools like GitHub Copilot?<\/h3>\n
\n

Teams should implement version control (Git), enforce testing, review AI-generated code, and use enterprise-grade tools with proper data governance.<\/p>\n <\/div>\n

5. What role does testing play in AI-driven data development?<\/h3>\n
\n

Testing validates AI-generated code, prevents regressions, and builds a safety net to ensure consistent and accurate data outputs.<\/p>\n <\/div>\n

6. How can Microsoft Fabric and Power BI teams manage AI risks?<\/h3>\n
\n

By using Git integration, workspace separation, automated testing, and monitoring tools to maintain performance and data quality.<\/p>\n <\/div>\n

7. What is the best way to track AI-generated code changes?<\/h3>\n
\n

Using version control systems like Git to maintain a clear audit trail and enable rollback of AI-generated updates.<\/p>\n <\/div>\n <\/section>\n","protected":false},"excerpt":{"rendered":"

Learn how DataOps principles improve trust, quality, and governance when using GenAI in data solutions. Tips for for Git, testing, and Microsoft Fabric.…<\/p>\n","protected":false},"author":344919,"featured_media":109899,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[159169,47,143526,147005],"tags":[159075,4483,158997],"coauthors":[159224],"class_list":["post-109725","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","category-data-science","category-development","category-web","tag-ai","tag-data","tag-microsoft-fabric"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/109725","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/users\/344919"}],"replies":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/comments?post=109725"}],"version-history":[{"count":6,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/109725\/revisions"}],"predecessor-version":[{"id":109983,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/109725\/revisions\/109983"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media\/109899"}],"wp:attachment":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media?parent=109725"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/categories?post=109725"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/tags?post=109725"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/coauthors?post=109725"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}