{"id":101338,"date":"2024-02-01T03:51:35","date_gmt":"2024-02-01T03:51:35","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=101338"},"modified":"2024-09-03T20:15:16","modified_gmt":"2024-09-03T20:15:16","slug":"feature-flags-in-data-projects","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/blogs\/feature-flags-in-data-projects\/","title":{"rendered":"Feature Flags in Data Projects"},"content":{"rendered":"<p>My motivation for writing this summary was an interaction with a project owner that didn\u2019t understand why we couldn\u2019t use feature flags directly in Power BI to control the user interface. This was different from our other deployments, so it took a few rounds of explanations to convince them that our use case didn\u2019t support feature flags. It\u2019s an oversimplification to say they can\u2019t be used in data projects. They can be used in Power BI and other reporting tools, but the implementation is different from coding languages and their usage is limited in comparison. Feature flags can also be used in ETL tools, data engines, ETL tools, and other data tools, but with some caveats. Sometimes those caveats are severe enough that you will want to carefully consider how you use feature flags in your data projects.<\/p>\n<h2>Description of feature flag and use cases<\/h2>\n<p>Feature flags are one of the primary strategies when implementing DevOps. Feature flags, also known as feature toggles or just flags, are used to branch code depending on the current value of the feature flag. This is no different from any other branching logic, except feature flags involve a lookup to a feature flag table, or configuration file, or a call to a feature flag API. The value returned by the feature flag system is used to determine which branch of code is executed. This allows code to be pushed to production using automated methods (CICD or near CICD) before it has been fully tested. The new, untested code does not execute until the feature flag is enabled.<\/p>\n<p>As mentioned, the primary use case of feature flags is to turn off or disable new features in code until they have been fully tested and related systems are ready for the changes. There are many ways to implement feature flags and many customizations possible. I won\u2019t be covering those options, rather just considering how feature flags can be used in data projects and their limitations.<\/p>\n<h2>Feature Flags in Data Projects<\/h2>\n<h3>Non-Breaking Changes<\/h3>\n<p>Before I get too deep into the technical aspects of feature flags in database engines and data products, the concept of non-breaking changes should be discussed. Non-breaking changes simply means that adding a new column to a table or view, adding output to a stored procedure, adding a new parameter to a stored procedure, or modifying the internal logic of a data object should not break existing usage of these objects. That includes ETL, reports, API calls, and anything else hitting that data object. Non-breaking changes are especially important for tables since they can\u2019t be feature flagged. When you add a new column, it is there and can\u2019t be disabled. It can be hidden behind a view, it can be empty, it can be ignored, but table changes happen immediately during deployment.<\/p>\n<p>Non-breaking changes are an important design consideration with database development, whether using DevOps or manually deploying database objects. In a database project, non-breaking changes means making sure that new columns are added to the end of tables, columns and tables are removed only after all references to those items are gone, and logic changes (in stored procedures or other objects) don\u2019t impact the output column definitions.<\/p>\n<p>Basic design considerations for non-breaking changes in databases is fairly straight forward. First, use ALTER TABLE to add columns rather than using the GUI to add them to the middle of your table. Next, specify columns in all SQL, SELECT, INSERT, UPDATE, MERGE, CTEs, view creation, temp tables, or any other reference to the objects rather than using the wildcard *. Some design patterns and data products gracefully handle new columns and some are very unfriendly to changes, even to views or tables when columns are added at the end. You will need to understand how the data products you use in your environment handle changes, and mange objects with that in mind.<\/p>\n<p>Changes are inevitable, but careful modeling at the beginning of a project and care when making changes helps immensely. You will need to make breaking changes at some point, but there are ways to minimize their impact and downtime.<\/p>\n<h4>Parallel table design<\/h4>\n<p>One option for extensive changes is parallel table design. If you are refactoring to a significant degree, you may have to keep two tables running and in synchronization while all related code and related products are updated. Feature flags will be used at an API \/ application level but you will likely not use feature flags in your database objects, or reports. At the database level, new database objects that reference the new parallel tables should be created. Creating new objects is also easier to manage when you clean up your code later. The new tables will likely need to be included in your ETL, depending on the design.<\/p>\n<p>Adding parallel tables can be a lot of work, but refactoring to this degree is always a lot of work. After the new tables are in place and data is being maintained in both copies, the real work begins of changing anything dependent on those tables. Hopefully you don\u2019t allow external applications or reports direct access to your transactional tables, which minimizes impact. When all code has been refactored, the old tables should be removed.<\/p>\n<h4>Changing existing objects<\/h4>\n<p>You won\u2019t normally need to create parallel tables. The existing table will be modified or data objects will be added and modified.<\/p>\n<h4>Change all code first<\/h4>\n<p>This is obvious, but all code that references the updated table should be added and feature flags put in place before you deploy your database changes. The new methods and feature flag code must be ready so the deployment with the modified data code doesn\u2019t break. Using DevOps, they could be part of the same release, but it needs to be ready when the data code is updated.<\/p>\n<h4>Update data objects<\/h4>\n<p>Database objects need to be updated as well. Update stored procedures, views, and functions. Don\u2019t forget items such as indexes, constraints and foreign keys too. If you are using a data project in Visual Studio, build your project before you check in the code to make sure everything compiles after your changes.<\/p>\n<p>It is sometimes necessary to manually update data with a pre-deployment script or items may need to be modified with a post-deployment script. This should be avoided when possible, but for breaking changes it isn\u2019t always possible. Also watch for issues with nullability of columns. It is sometimes necessary to deploy the column as nullable, populate it, then change it to be not nullable later or in your post-deploy.<\/p>\n<h4>Update data products<\/h4>\n<p>Reports and ETL flows also need to be updated. Data models, metadata, SQL scripts, and any other references to the modified database code needs to be fixed. The deployment timing of this is also critical. These deploys need to happen before the breaking change is implemented or in conjunction with the change.<\/p>\n<h4>Final steps and considerations<\/h4>\n<p>Removing columns can have the most impact. This will be the last change you would make, as it will break anything referencing that column.<\/p>\n<p>When everything is updated and the new code path is getting used, the deprecated objects can be removed. Monitor the database and connected systems for errors. You will need to work with external teams to be sure they modify their connected tools also, but this should happen early in the process.<\/p>\n<h4>Impact of adding columns<\/h4>\n<p>It can be tempting to add a column to the middle of a table. Developers and business users like data to be grouped together logically within tables. Adding a new column that logically fits with other existing columns, such as additional address information, might feel like it should be placed adjacent to those columns. This can cause huge performance problems in a production system. It can also break existing queries if they happen to reference columns by ordinal position. This can be done in SQL Server in the ORDER BY statement, but it isn\u2019t recommended.<\/p>\n<p>Using an ALTER TABLE statement can mean the difference between a sub-second deployment and many hours and system disruptions if the table is large. Adding a column anywhere but the end of the table causes a sequence of events that are very resource intensive.<\/p>\n<p>The following is a small portion of what happens when a column is added anywhere but the end of the table using ALTER TABLE. You can see that a temporary table is created, foreign keys are dropped and recreated on the new table, data is transferred, the identity value is set, extended properties are set and any other table properties. Finally, cleanup tasks are performed.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1136\" height=\"737\" class=\"wp-image-101419\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101338-1-1.png\" \/><\/p>\n<p>In contrast to the above, adding a column to the end of a table with the ALTER TABLE statement, either explicitly with the SQL statement or by using the GUI, is generally a sub-second operation and has very little impact on the table and database in general.<\/p>\n<pre class=\"lang:none theme:none\">ALTER TABLE Sales.Invoices\r\nADD NewColumn2 int\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0NULL\r\nGO<\/pre>\n<p>&nbsp;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"673\" height=\"38\" class=\"wp-image-101420\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101338-2-1.png\" \/><\/p>\n<p>&nbsp;<\/p>\n<p>Good coding practices at the application level are also needed to ensure database changes don\u2019t break the application. Avoid SELECT *, each column used by the application should be specified.<\/p>\n<h3>Test Base<\/h3>\n<p>The following examples presented here use a very simple feature flag table. A single table, a stored procedure and a function for lookups are the complete implementation. Your actual feature flag system would likely be more complicated and have additional features. The test database WideWorldImporters was used. The scripts are presented below.<\/p>\n<p>The schema and table definition:<\/p>\n<pre class=\"lang:none theme:none\">CREATE SCHEMA FF\r\nAUTHORIZATION dbo\r\nGO\r\nCREATE TABLE FF.FeatureFlags (\r\n\tFeatureFlagID\t\tint\t\t\t\tNOT NULL\t\tCONSTRAINT PK_FFFeatureFlags\t\t\t\t\tPRIMARY KEY CLUSTERED\t\tidentity\r\n\t,FeatureFlag\t\tvarchar(100)\tNOT NULL\t\tCONSTRAINT UNQ_FFFeatureFlags_FeatureFlag\t\tUNIQUE NONCLUSTERED\r\n\t,IsEnabled\t\t\tbit\t\t\t\tNOT NULL\r\n\t,CreatedDate\t\tdatetime\t\tNOT NULL\t\tCONSTRAINT DF_FFFeatureFlags_CreatedDate\t\tDEFAULT(GETDATE())\r\n)\r\nGO<\/pre>\n<p>The stored procedure, used in the SSIS and the ADF examples:<\/p>\n<pre class=\"lang:none theme:none\">CREATE OR ALTER PROCEDURE FF.prc_FeatureFlag_Lookup\r\n\t@FeatureFlag\t\tvarchar(100)\r\nAS\r\nSET NOCOUNT ON\r\nSELECT \r\n\tFF.FeatureFlag\r\n\t,FF.IsEnabled\r\nFROM FF.FeatureFlags FF\r\nWHERE FF.FeatureFlag\t\t= @FeatureFlag\r\nGO<\/pre>\n<p>The scalar function, used in the SQL example:<\/p>\n<pre class=\"lang:none theme:none\">CREATE OR ALTER FUNCTION FF.fn_FeatureFlag_Lookup (\r\n\t@FeatureFlag\t\tvarchar(100)\r\n)\r\nRETURNS bit\r\nAS\r\nBEGIN\r\n\tDECLARE @IsEnabled bit\r\n\tSELECT @IsEnabled = FF.IsEnabled\r\n\tFROM FF.FeatureFlags FF\r\n\tWHERE FF.FeatureFlag\t\t= @FeatureFlag\r\n\tSELECT @IsEnabled = ISNULL(@IsEnabled,0)\r\n\tRETURN @IsEnabled\r\nEND<\/pre>\n<h3>Feature Flags in SQL Server<\/h3>\n<p>The above section on non-breaking changes demonstrated that feature flags can\u2019t be used on tables. They could potentially be used with views but not to add or remove columns, only for logic in a view. The best location to put feature flags in the data engine is within stored procedures since they are generally where any database logic will reside. But it is usually easier to include that logic in your application. Given those limitations, this section describes how you can use feature flags in SQL.<\/p>\n<h4>Stored Procedures \/ Functions \/ Views<\/h4>\n<p>Stored procedures can easily use feature flags, especially if they are stored in the same database or the flags are passed in as a parameter. If an API is used, it can be called directly from SQL using sp_OACreate. This can also be done via a CLR stored procedure using C# and standard libraries. The first part of the code, or a predecessor to the current stored procedure needs to make this call, then CASE statements or IF statements can be used for the branching logic.<\/p>\n<p>Even though you can call APIs directly from SQL, I would not recommend this pattern. It has the potential to open security holes since it increases the surface area used by the SQL engine. It is also more fragile compared to calling the API from your regular coding environment. There are some additional steps needed to configure the engine to allow this and you have to allow the SQL Server to call the IP or range of IP addresses used by the API servers. It\u2019s not a great pattern. In addition to the security and configuration issues, it is also a more fragile design.<\/p>\n<p>Another method that can be used with stored procedures is to create a new version of the stored procedure and call the correct version from your API. This allows you to place all the logic in one location, it is less fragile, and cleanup is easier in SQL Server. Cleaning up feature flag code can be tedious and has potential to introduce new bugs. If there are versioned instances of stored procedures, they can simply be deleted after they are deprecated. Removing an entire stored procedure is easier to get right compared to surgically removing portions of code. Having multiple, versioned, SQL objects doesn\u2019t incur a performance penalty since it is just metadata storage. Be sure you have a clear naming standard and clean up code promptly or the duplicate objects can become very confusing.<\/p>\n<p>The following example shows using a feature flag inside of a view. The scalar function presented above is used to determine the current state of the feature flag. If it is set, the column is presented as-is. If it is not set, the hard-coded mask is presented. If there were issues, the flag could be disabled and the hard-coded masking would be presented to users again.<\/p>\n<p>This shows the view altered from the default installation of WideWorldImporters<\/p>\n<pre class=\"lang:none theme:none\">CREATE  OR ALTER VIEW [dbo].[vw_CustomerName]\r\nAS\r\nSELECT\r\n\tCustomerID\r\n\t,CASE FF.fn_FeatureFlag_Lookup('SQL Example') WHEN 1 THEN CustomerName ELSE LEFT(CustomerName,1) + N'xxx' END CustomerName\r\n\t,CityName\r\nFROM sales.Customers C\r\n\tINNER JOIN Application.Cities CTY\r\n\t\tON C.DeliveryCityID\t\t= CTY.CityID\r\nGO<\/pre>\n<p>Note that in the function, a default value of 0 is returned if a flag isn\u2019t setup. The feature flag is first inserted or updated and enabled. Results are shown from the view, then the flag is disabled with corresponding results.<\/p>\n<pre class=\"lang:none theme:none\">;\r\nWITH SQL_CTE AS (\r\n\tSELECT\r\n\t\t'SQL Example'\tFeatureFlag\r\n\t\t,1\t\t\t\tIsEnabled\r\n)\r\nMERGE INTO FF.FeatureFlags FF\r\nUSING SQL_CTE TF\r\n\tON FF.FeatureFlag\t\t= TF.FeatureFlag\r\nWHEN NOT MATCHED THEN INSERT (\r\n\tFeatureFlag\r\n\t,IsEnabled\r\n)\r\nVALUES (\r\n\tTF.FeatureFlag\r\n\t,TF.IsEnabled\r\n)\r\nWHEN MATCHED THEN UPDATE\r\nSET\r\n\tIsEnabled\t\t\t\t= TF.IsEnabled\r\n;\r\nSELECT TOP 10 *\r\nFROM dbo.vw_CustomerName\r\nORDER BY CustomerID\r\nUPDATE FF.FeatureFlags\r\nSET IsEnabled = 0\r\nWHERE FeatureFlag = 'SQL Example'\r\nSELECT TOP 10 *\r\nFROM dbo.vw_CustomerName\r\nORDER BY CustomerID<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"362\" height=\"379\" class=\"wp-image-101421\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101338-3-1.png\" \/><\/p>\n<p>A reminder about implementing feature flags in SQL via this method. It is more difficult to maintain feature flags within data objects than creating and versioning your data objects. I\u2019ll reiterate that deleting an entire object is much easier than correctly removing or modifying individual lines of code.<\/p>\n<h4>Entity Framework and Dynamic SQL<\/h4>\n<p>Using feature flags with Entity Framework (EF), or any other ORM, is very straight forward. Since the SQL is created by EF via code, feature flags are used in a normal fashion in the API using control flow logic. ORMs can also be used to call stored procedures. The same principles apply, use branching logic based on the feature flag to call the correct version of your stored procedure.<\/p>\n<h4>Table Options<\/h4>\n<p>Table options in database engines were discussed thoroughly in the non-breaking changes section. Table changes happen immediately and can\u2019t be put behind feature flags. Keep that in mind when adding columns, but especially when removing columns and tables. Follow the basic rules, don\u2019t add columns to the middle of your table, use a complete column list for all queries, both DDL and DML, and work with other teams to be sure changes are understood. This includes logic changes as well as column changes.<\/p>\n<h3>Feature Flags in Report Engines<\/h3>\n<p>This is where feature flags in data projects can get tricky. You have to be very specific when describing the capabilities of a reporting system in regards to feature flags. Can feature flags be used with reporting systems? Yes and no. It depends on what you are trying to do. You generally won\u2019t be able to control the user interface and you also probably can\u2019t dynamically add columns unless they are already defined and in your data model. Graphics are the core of modern reporting and it is what product owners generally want modified or added to a report. This is what you would put behind a feature flag in a regular application. SSRS and Power BI don\u2019t have feature flag support built into the user interface, but that doesn\u2019t mean they can\u2019t be used in other scenarios.<\/p>\n<h4>SQL Server Reporting Services (SSRS) Options<\/h4>\n<p>SSRS has not been deprecated yet, but most organizations are moving to Power BI or have it on their tech roadmap for replacement. It is still widely used and merits some consideration. You can\u2019t dynamically add columns to SSRS, but logic behind derived columns can be modified in several ways.<\/p>\n<p>The following shows the code that creates an SSRS report. It is a predefined, XML format. Trying to add a feature flag to this would not modify the report output, but would likely break the report. This is why feature flags can\u2019t be added directly to the GUI portion of report engines.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1232\" height=\"480\" class=\"wp-image-101422\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101338-4-1.png\" \/><\/p>\n<p>The methods mentioned previously for SQL Server can be used to control logic and versions for stored procedures and even dynamic SQL. This is no different from any application calling SQL. The feature flags can be looked up in the stored procedure itself, if they are stored in the database. The current feature flag can also be passed in as a parameter.<\/p>\n<p>VB.NET or a DLL can be used in SSRS. These methods can be used to look up active feature flags and call different versions of the code to manipulate data. Code is frequently used to create common blocks of code that operate on the data returned in the datasets. These blocks of code can use feature flags to determine and change the logic. If code is used to manipulate data in the report, feature flags can also be used in global variables to determine which path to run.<\/p>\n<p>Code can be added directly to the report using VB.NET in the Code section:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"436\" height=\"138\" class=\"wp-image-101423\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101338-5-1.png\" \/><\/p>\n<p>And assemblies can be added in the References section of the report properties:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"437\" height=\"182\" class=\"wp-image-101424\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101338-6-1.png\" \/><\/p>\n<p>The report variables can use expressions to set variables for your feature flags or as a result of feature flags.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"947\" height=\"254\" class=\"wp-image-101425\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101338-7-1.png\" \/><\/p>\n<p>An easier, more testable, and more maintainable method is simply to copy the existing report and have multiple versions on the server. Having two versions is added overhead, but would be my method if I needed to be ready with a new version with no delay. If reports are called via an application or API layer, the correct version can be called based on the feature flag system used by the application, rather than trying to control it in SSRS.<\/p>\n<h4>Power BI<\/h4>\n<p>Power BI (PBI) has many of the limitations that SSRS has, but it also has some advantages not available in SSRS. Several different languages are supported directly in PBI which expands some of the options available. The following shows the raw PBIX file. You can see that it is a binary file, and the header indicates it is a ZIP file. There is nothing to modify at this level and the GUI has no support for feature flags.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1242\" height=\"597\" class=\"wp-image-101426\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101338-8-1.png\" \/><\/p>\n<p>Even if you change the extension to a ZIP file and open the file, you can see it contains a set of files with a set format. Even if you were to modify it directly, the report engine would have to be looking for those files or modifications to implement feature flags directly in PBI.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"349\" height=\"204\" class=\"wp-image-101427\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101338-9-1.png\" \/><\/p>\n<p>The same methods used to modify SQL logic can be used when SQL Server is used with PBI. It is easier to control this logic on the SQL side rather than the PBI side. It also has the advantage of providing consistent output no matter what application is calling the procedure.<\/p>\n<p>The coding languages in PBI offer more flexibility. R is a statistical processing language, but it can be used to do many things, including calling databases or API services. R modules are available in PBI. You won\u2019t be able to change the base behavior or output in PBI via R, but graphics and other data created and manipulated via R can fully utilize feature flags. This is a potential method to change the graphical output in PBI.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1025\" height=\"214\" class=\"wp-image-101428\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101338-10-1.png\" \/><\/p>\n<p>Python is also integrated within PBI. It is a full-fledged programming language and can be used to manipulate data and check feature flag enablement. Python code can be controlled by feature flags, just like standard application logic. It can be used to change the logic used within Python. The data model can\u2019t be directly changed with this method, but logic can be manipulated. Graphics presented by Python can also be fully manipulated via feature flags.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"644\" height=\"192\" class=\"wp-image-101429\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101338-11-1.png\" \/><\/p>\n<p>As I mentioned, you can use feature flags with these programming languages, but that limits the development style in PBI. I have seen a large emphasis on end-user development, and this would greatly limit the pool of users able to modify the code.<\/p>\n<p>The other programming and data manipulation languages in PBI are more limited than R and Python. M, the query language for Power Query, MDX, used to manipulate data in PBI and SSAS, and DAX, also used in SSAS and PBI have conditional operators. IF, SWITCH, IIF, and variables can be created and used in the scripts. They can be used to evaluate feature flags and change output. This limited feature flag support depends on the data being available in the dataset or a lookup function. It is very similar to using feature flags within SQL objects and code.<\/p>\n<h3>Feature Flags in ETL<\/h3>\n<p>ETL packages have much more flexibility with the implementation of feature flags when compared to reporting engines. Control flow is standard. This makes implementing feature flags easy.<\/p>\n<h4>SQL Server Integration Services<\/h4>\n<p>SQL Server Integration Services (SSIS) can implement feature flags using any of the conditional logic operators available, as well as the SQL methods.<\/p>\n<p>Since SSIS can call web services via standard tasks, make connections to databases, run custom code in C# via a script task, and call virtually any application, your enterprise standards can be maintained while using feature flags with SSIS.<\/p>\n<p>The following is a simple example of using a feature flag to determine which Data Flow Task (DFT) to run. The precedence for each DFT is set to Expression and the appropriate feature flag return value.<\/p>\n<pre class=\"lang:none theme:none\">INSERT INTO FF.FeatureFlags (\r\n\tFeatureFlag\r\n\t,IsEnabled\r\n)\r\nVALUES ('SSIS Example',1)<\/pre>\n<p>The precedence for the \u201cDFT \u2013 Test Flow\u201d is set to execute when True<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"813\" height=\"678\" class=\"wp-image-101430\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101338-12-1.png\" \/> Debugging in the editor shows the intended path is being followed.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"689\" height=\"361\" class=\"wp-image-101431\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101338-13-1.png\" \/><\/p>\n<p>To be sure the regular flow isn\u2019t followed when the feature flag is set, it also has an expression precedence constraint. It is set to False.<\/p>\n<h4>Azure Data Factory<\/h4>\n<p>Azure Data Factory (ADF) can use the same methods as SSIS. Database connections can be made, connections to services and virtually anything else in the enterprise can be targeted.<\/p>\n<p>The following sets the feature flag for ADF to True.<\/p>\n<pre class=\"lang:none theme:none\">INSERT INTO FF.FeatureFlags (\r\n\tFeatureFlag\r\n\t,IsEnabled\r\n)\r\nVALUES ('ADF Example',1)<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"729\" height=\"355\" class=\"wp-image-101432\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101338-14-1.png\" \/> The debugger confirms that the flow followed the True path and activated the Metadata activity. <img loading=\"lazy\" decoding=\"async\" width=\"610\" height=\"172\" class=\"wp-image-101433\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101338-15-1.png\" \/><\/p>\n<p>To confirm that everything is working as expected, the False condition is tested next.<\/p>\n<pre class=\"lang:none theme:none\">UPDATE FF.FeatureFlags\r\nSET IsEnabled = 0\r\nWHERE FeatureFlag = 'ADF Example'<\/pre>\n<p>The debugger activities show that the Data Flow \u2013 Test path is executed during the false condition.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"632\" height=\"168\" class=\"wp-image-101434\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101338-16-1.png\" \/><\/p>\n<h4>ETL Product Summary<\/h4>\n<p>All modern ETL platforms, including those not mentioned such as Databricks and Informatica, include control flow logic. They can usually call a service or a database to access feature flag values. This will depend on your exact environment and tools, but there is a method to use feature flags in all of these platforms.<\/p>\n<p>The same reminders apply to any ETL tool. Consider what happens when the feature flag system can\u2019t be reached or the value is not present. Default behavior should be predictable and not cause an error. Implement feature flags in a way that makes cleanup easier. Creating a new data flow, even if most of it is copied, is easier and safer to remove later when the feature flag is deprecated. As with SQL, it is easier to remove a complete object instead of pieces of an object or a section of a data flow.<\/p>\n<h2>Summary<\/h2>\n<p>Feature flags can be used in data projects, but their scope is limited. Since report engines expect a specific file format and they use that format to create a report, flags can\u2019t be added to a report at the GUI level unless the vendor has specific support for them. The report files are proprietary and changes to them will be ignored at best, or more likely, cause a runtime error.<\/p>\n<p>Support increases at the SQL engine level, but it is still more limited than with full programming languages and custom code. Care must be taken when adding columns or modifying objects. Breaking changes should be avoided and only happen when necessary. Adding columns anywhere but the end of a table and removing columns are common breaking changes that should be avoided or carefully planned out first.<\/p>\n<p>Feature flags also need to be maintained. They shouldn\u2019t be in the system forever \u2013 they have a specific purpose and ideally will adhere to that single, time-limited purpose. Not everyone agrees with this scope for feature flags, but I firmly believe that a permanent feature flag should actually be feature of the software and not controlled by flags.<\/p>\n<p>Feature flag cleanup can be a time-consuming development task. Removing deprecated feature flags should be an integral part of development. This includes removing old stored procedures, methods, and any other code related to the deprecated feature flag.<\/p>\n<p>Depending on your method of feature flag deployment, there are limitations. SQL scripts, including post-deployment scripts, have a limit of 1000 items when used in an INSERT \/ VALUE statement. If you start hitting these limits or the performance of the system is impacted, feature flag cleanup needs to be a top priority.<\/p>\n<p>Creating and using feature flags at a method and data object level (stored procedures, views, functions) is easier than imbedding the code within single methods or stored procedures. Simple scripts can be run to look for the old code and removing them involves lower risk than surgically removing code. Rigor is needed for both activities, but it\u2019s better to keep things simple.<\/p>\n<p>Feature flags are not magic and don\u2019t solve all of your issues with deployments. Care must still be taken and best practices followed. Using feature flags doesn\u2019t eliminate the need for code reviews or testing. It is also possible to introduce errors based on the feature flag code itself. Use feature flags with care in data projects, know their limitations, be consistent, test both code paths, and remove flags when they are no longer in use.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>My motivation for writing this summary was an interaction with a project owner that didn\u2019t understand why we couldn\u2019t use feature flags directly in Power BI to control the user interface. This was different from our other deployments, so it took a few rounds of explanations to convince them that our use case didn\u2019t support&#8230;&hellip;<\/p>\n","protected":false},"author":19670,"featured_media":84613,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[2,159166],"tags":[159061,136299,159060,101611,4306,4400,159064],"coauthors":[98702],"class_list":["post-101338","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blogs","category-powerbi","tag-adf","tag-ci-cd","tag-feature-flags","tag-power-bi","tag-ssis","tag-ssrs","tag-summit-survey"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/101338","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/users\/19670"}],"replies":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/comments?post=101338"}],"version-history":[{"count":6,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/101338\/revisions"}],"predecessor-version":[{"id":101435,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/101338\/revisions\/101435"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media\/84613"}],"wp:attachment":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media?parent=101338"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/categories?post=101338"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/tags?post=101338"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/coauthors?post=101338"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}