{"id":100843,"date":"2023-12-13T17:00:46","date_gmt":"2023-12-13T17:00:46","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=100843"},"modified":"2024-09-03T20:15:17","modified_gmt":"2024-09-03T20:15:17","slug":"fabric-notebooks-and-deployment-pipelines","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/blogs\/fabric-notebooks-and-deployment-pipelines\/","title":{"rendered":"Fabric Notebooks and Deployment Pipelines"},"content":{"rendered":"

On my article about Fabric<\/strong> source control extended features, I explained how Microsoft included the notebooks on the source control.<\/p>\n

In this way we can include notebooks on a Software Development Lifecycle<\/strong> (SDLC<\/strong>) for Power BI<\/strong> objects.<\/p>\n

In this way, the notebooks need to flow from the development environment to test and production environments. However, what happens with the references the notebook may contain?<\/p>\n

The notebook may contain references to lakehouses and other configurations which may be different on each environment. We need to ensure these references are automatically changed when we promote the notebook to another environment.<\/p>\n

Two types of references<\/h2>\n

The notebooks in Fabric can contain two types of references and we need to handle each type in different ways:<\/p>\n

Configuration values:<\/strong> These are any value used in the notebook code.<\/p>\n

Default Lakehouse:<\/strong> The default lakehouse is essential for most of the notebook tasks. The notebook code doesn’t contain the default lakehouse code directly. For this reason, the default lakehouse requires a different method to be handled than other references, becoming a 2nd type of reference.<\/p>\n

Let\u2019s analyse how to manage each one of these reference types on our notebooks.<\/p>\n

Configuration Values<\/h2>\n

The rule we should use is simple: We should avoid using hard coded configuration values. We need to store the configuration externally to the notebook and in a way that we can change the values between environments.<\/p>\n

It doesn\u2019t matter where you choose to store the configuration values, you would need to repeat on every notebook the code to retrieve them.<\/p>\n

This creates another requirement: We need to centralize this code in one notebook, and we need to be able to call this notebook from any other.<\/p>\n

We can use the following statement:<\/p>\n

<\/p>\n

\n
%<\/span>run \u201c<\/span>notebook name\u201d<\/span>\r\n<\/pre>\n<\/div>\n
There are other methods to execute a different notebook, but this method makes the execution happen in the same session. In this way, the caller has access to the variables created on the centralized notebook.<\/p>\n
In this way, the code used on the centralized notebook can load the configuration in variables and these variables will be accessible by the caller.<\/p>\n
If we store the configurations in a JSON file in the lakehouse, the code of the centralized notebook will be like this:<\/p>\n
<\/p>\n
\n
df =<\/span> spark.<\/span>read.<\/span>json(\"Files\/configuration\/config.json\"<\/span>, multiLine=<\/span>True<\/span>)\r\nurlprefix=<\/span>df.<\/span>collect()[0<\/span>].<\/span>__getitem__(\"functionURL\"<\/span>)\r\ndatabase=<\/span>df.<\/span>collect()[0<\/span>].<\/span>__getitem__(\"database\"<\/span>)\r\nlakehouseName=<\/span>df.<\/span>collect()[0<\/span>].<\/span>__getitem__(\"lakehouseName\"<\/span>)\r\n<\/pre>\n<\/div>\n
On this way, we can move the notebook to different environments without the need to change any code.<\/p>\n
Default Lakehouse<\/h2>\n
The default lakehouse would be way more difficult to change without Power BI\u2019s<\/strong> help.<\/p>\n
As explained on the previous blog, the source control saves the notebook as a set of several files.\u00a0The most important one is a .PY<\/em> file containing the notebook code.<\/p>\n
The .PY<\/em> file also contains metadata which includes the default lakehouse for the notebook.<\/p>\n
Luckly, Power BI<\/strong> deployment pipelines support the creation of rules for notebooks, to change the default lakehouse of the notebooks.<\/p>\n
Deployment Pipelines<\/h2>\n
Deployment pipelines are used to move objects from one workspace to another and keep the workspaces synchronized.<\/p>\n
In this way, we can use different workspaces as development, test and production environment.<\/p>\n
We can create rules about the transfer of objects from one workspace to another. In this way, when we transfer the objects, the rules will ensure some configurations are changed, such as the default lakehouse of the notebooks.<\/p>\n
I have a published session about Deployment Pipelines in English<\/a> or in Portuguese<\/a><\/p>\n
Configuring Deployment Pipelines<\/h2>\n
Let\u2019s build a small demonstration.<\/p>\n
On the starting point, we will have 4 workspaces:<\/p>\n