Use source control with a single developer<\/li>\n<\/ul>\nBelow I explain these 3 scenarios. Could you imagine additional ones? Let\u2019s talk about it in the comments.<\/p>\n
Use source control together Deployment Pipelines<\/h3>\n
In this scenario, the deployment pipelines are the responsible to move the product between environments.<\/p>\n
The Development<\/strong> environment needs to be linked to a repository, but the Test<\/strong> and Production <\/strong>environment have no need of a repository. The movement from Dev to Test and then to Production will be responsibility of the deployment pipelines.<\/p>\nThis scenario ends up being like the Data Factory<\/strong> and Azure Synapse<\/strong> source control: only the Development environment requires a repository, the upper environments don\u2019t need one.<\/p>\nNext Steps<\/h3>\n
Let\u2019s understand the technical features Power BI<\/strong> offers for source control first, so we can analyse deeply the additional architectural options.<\/p>\nPower BI and Source Control: The current challenges<\/h2>\n
These are the current challenges to using Power BI in a source control scenario:<\/p>\n
\n- Power BI files have binary content
\n<\/strong>The binary content doesn\u2019t work so well with source control. The main purpose of the source control is to allow developers to work on the files an make it possible to merge the results. Using a binary file, this is not possible.<\/li>\n- Power BI files contain data<\/strong>
\nThe PBIX<\/code> file contains the data gathered during the dataset refresh. Usually, we don\u2019t want to send data to source control, we send only text files we can compare to find the differences implemented by the developer.<\/li>\n<\/ul>\nIn fact, the original PBIX<\/code> file is a .ZIP<\/code> file with a different extension. If you rename the file as .ZIP<\/code> and open it, you can see the individual files. However, they don\u2019t have extensions and are not built with source control in mind.<\/p>\n![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-computer-description-automatica-97.png\")
<\/p>\n
GIT Basic Concepts<\/h2>\n
The image below summarizes most GIT concepts you need to know to make a good use of source control in Power BI<\/p>\n
![](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/word-image-100615-3-1.png\")
<\/p>\n
The following list summarizes at a high level, the process to using GIT to source control your files.<\/p>\n
\n- The remote repository is CLONED<\/strong> to the local machine. The clone becomes a folder in the local machine. This folder is configured as a local repository.
\nAll the existing branches in the moment of the CLONE<\/strong> are created in the local repository. If the Branch B is created later, the local repository will not have the Branch B included.<\/li>\n- A FETCH<\/strong> operation can be triggered by the local repository. The fetch brings to the local repository the knowledge of existence of new branches on the server.<\/li>\n
- A CHECKOUT<\/strong> operation can be done to change the current branch to another one.
\nEach branch has different versions of the files. A CHECKOUT<\/strong> operation replaces the files in the repository folder by the version contained in the branch. In this way you can work in different versions at any moment. You only need to CHECKOUT<\/strong> the branch you would like to work with.
\nThe CHECKOUT<\/strong> process can generate different results:<\/p>\n\n- Change a local branch by another local branch<\/li>\n
- Retrieve a remote branch to the local repository<\/li>\n
- Create a new branch<\/li>\n<\/ol>\n<\/li>\n
- When you update the files in the local repository, you need to create a COMMIT<\/strong>. Basically, it means packing your changes together to be sent to the remote repository.<\/li>\n
- Once the local development is completed, the commits need to be PUSHED<\/strong> to the remote repository, updating the remote branch. On the image above, the Branch B is updated.<\/li>\n
- Branch B and Branch A have two different versions of the project in the remote repository. The versions need to be synchronized.\n
The synchronization process involves a request, which is called PULL REQUEST<\/code>. This happens because there are multiple possible scenarios. Here are some of the possible scenarios:<\/p>\n\n- The branches may be created by different developers. One developer may need to approve what the other developer produced<\/li>\n
- The target branch may be a test or production environment and the process involves publishing to a new environment. In some cases, may even include CI\/CD process.<\/li>\n
- The procedure may include a code review. The
PULL REQUEST<\/code> may be subject to approval by other developers or the team leader.<\/li>\n<\/ol>\n<\/li>\n- After the
PULL REQUEST<\/code>, the Branch A<\/code> on the server repository is more updated than the Branch A<\/code> in the local repository. The developer needs to execute a PULL<\/code> from the server to the local repository to update the local branch.<\/li>\n<\/ol>\nTODO: Save a Power BI Project<\/h2>\n
The Power BI Project is a preview feature. Before we can use it, we need to enable the feature. Once in Power BI, use File -> Options and Settings -> Options -> Preview Features<\/em> and enable the Power BI Project (.pbip) save option<\/em><\/p>\n![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-computer-description-automatica-99.png\")
<\/p>\n
\n- Create a folder to be your local repository<\/li>\n
- Open your sample Power BI file (
PBIX<\/code>)<\/li>\n- Save the project using File-> Save As<\/li>\n
- Click Browse this device to choose the correct folder<\/li>\n
- Change the file type to
PBIP<\/code><\/li>\n<\/ol>\n![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-computer-description-automatica-101.png\")
<\/p>\n
\n- Save the file<\/li>\n<\/ol>\n
The Power BI Project<\/h2>\n
The Power BI Project File, PBIP<\/code>, is not a single file by itself. When we save a PBIP<\/code>, the content of the Power BI file is broken down in many different files.<\/p>\nThe main points related to the files created by the Power BI project are the following:<\/p>\n
\n- The PBIP file is created in the root of the folder<\/li>\n
- Two subfolders are created, one for the reports and one for the dataset<\/li>\n<\/ul>\n
![](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/word-image-100615-6-1.png\")
<\/p>\n
\n- For each folder, dataset and report, a file
item.config.json<\/code> and a file item.metadata.config<\/code> are created. This file is used by Power BI to identify the content of the folder as a single object (a single dataset or a single report).<\/li>\n<\/ul>\n![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-computer-description-automatica-103.png\")
<\/p>\n
\n- Both folders contain a subfolder called \u201c<\/em>
.PBI<\/code>\u201d<\/em>. This folder is intended to contain files intended to be local, not included in the remote repository.<\/li>\n- The \u201c<\/em>
.PBI<\/code>\u201d<\/em> folder inside the dataset folder contains a file called cache.abf<\/code>. This file contains all the data generated by dataset refreshes. In this way, the data is never uploaded to the repository, only the object definitions.<\/li>\n<\/ul>\n![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-computer-description-automatica-104.png\")
<\/p>\n
\n- A
.gitignore<\/code> file is created in the root folder. It ignores the \u201c<\/em>.PBI<\/code>\u201d<\/em> folders, which makes them local files (that will not be version controlled).<\/li>\n- The report definition is saved in a file called
report.json<\/code>. As a JSON file, it becomes easier to make merges when two developers work on the same report at the same time.<\/li>\n- The dataset definition is stored in a file called
model.bim<\/code>, also in JSON format to help concurrent work.<\/li>\n<\/ul>\nIt\u2019s also possible to save multiple PBIP files in the same folder. The files keep a reference with each other and the folders, not allowing the objects to be mixed<\/p>\n
TODO: Initialize the local Repository and link to the remote one<\/h2>\n
This step needs to be taken after your remote repository is ready. In this example, I will use GIT BASH, a command line tool. This is GIT 101, you can do this step with different tools and skip this section if you already have this knowledge.<\/p>\n
This continues on the steps taken in the previous TODO section.<\/p>\n
\n- Open git bash\n
Change the current folder to the folder with the PBIP<\/code> files<\/li>\n- Use
CD C:\/repos\/Adventure<\/code> for this (the path is an example, of course).<\/li>\n- Use
GIT INIT<\/code>
\nThis statement initializes the folder as a local repository, creating the special git file on the folder \u201c.GIT\u201d<\/em><\/li>\n- Use
GIT ADD<\/code> .<\/li>\nGIT ADD<\/code> includes files to be tracked by GIT. The \u201c.\u201d Is telling to include everything, including subfolders. The files will be considered new files, not committed yet.<\/li>\n- Use
GIT COMMIT \u2013m<\/code> \u201cFirst Commit\u201d
\nWe are committing the files we just added to the local repository, including a commit message.
\nAt this point, the local repository is created and has its first commit, but it\u2019s not linked to the remote repository yet.<\/li>\n- Use
GIT REMOTE ADD ORIGIN <<your remote repository>><\/code>
\n\u201cORIGIN\u201d is not a statement, rather it is the name given to the remote repository. It\u2019s very common to use \u201corigin\u201d, because usually we have only one. But you can use any name you would like.<\/li>\n- Use
GIT FETCH<\/code>
\nThe local repository needs to discover what branches exist remotely before we link local branches with remote branches<\/li>\n- Use
GIT BRANCH \u2013SET-UPSTREAM-TO=ORIGIN\/MAIN<\/code>
\nThis links the local current branch (usually MAIN<\/code>) to the remote MAIN<\/code> branch.
\nThe remote repository usually already has at least a README<\/code> file. The local repository has the initial commit with our Power BI project. This means the repositories have a different history. This history needs to be synchronized. We will use the next three commands to synchronize the remote history with the local history.<\/li>\n- Use
GIT REBASE<\/code><\/li>\n- Use
GIT PUSH<\/code><\/li>\n<\/ol>\n![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-computer-screen-shot-of-a-program-description-a-2.png\")
<\/p>\n
The image below is the repository on Azure Devops after the last GIT PUSH<\/strong><\/p>\n![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-computer-description-automatica-105.png\")
<\/p>\n
TODO: Link the Power BI Workspace with the repository<\/h2>\n
Power BI will get the files from the workspace. We need to link a repository to the workspace. It\u2019s interesting how this new process will make the publish from the power bi desktop redundant.<\/p>\n
\n- Create a workspace, if you don\u2019t have one for this purpose yet<\/li>\n
- Click Workspace Settings<\/li>\n
- On WorkSpace Settings window, click GIT Integration<\/li>\n
- On Organization<\/em> dropdown, select your organization. Your Power BI<\/strong> account needs access to your repository<\/li>\n
- On Project<\/em> dropdown, select the project. Power BI<\/strong> will show the projects you have access to.<\/li>\n
- On GIT Repository<\/em> dropdown, select the name of the repository<\/li>\n
- On Branch<\/em> dropdown, select the name of the branch. The branch is the only setting you can change later without disconnecting from the repository.<\/li>\n
- On GIT Folder<\/em> textbox, type \u201c\/\u201d <\/em>. We will link the workspace with the root folder. Different folders in the repository, including subfolders, will be ignored.<\/li>\n<\/ol>\n
![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-computer-description-automatica-106.png\")
<\/p>\n
\n- Click the button Connect & Sync\n
<\/em>After syncing for a while, you will be able to see the files in the repository. Note that the Power BI Publish<\/em> command was not used. There is a good potential for the Publish command to become redundant.
\n![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-computer-description-automatica-107.png\")
<\/li>\n<\/ol>\n![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-computer-description-automatica-108.png\")
<\/p>\n
\n- Open the report\n
There is no data on the report. This illustrates how the data from the Power BI Desktop<\/strong> is never uploaded to the repository and doesn\u2019t reach the portal.
\n![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-close-up-of-a-logo-description-automatically-ge-2.png\")
<\/li>\n- On the Dataset, click the Expand button => Settings<\/li>\n<\/ol>\n
![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-computer-description-automatica-109.png\")
<\/em><\/p>\n\n- Under Data Source Credential<\/em>, click Edit Credentials<\/em> in the image below to fix the dataset credentials. You may notice the error message complaining about the credentials.<\/li>\n<\/ol>\n
![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-computer-description-automatica-110.png\")
<\/p>\n
\n- Back to the workspace, click the Refresh<\/em> Now button.<\/li>\n
- When the refresh is complete, open the report again.<\/li>\n<\/ol>\n
![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-number-with-black-text-description-automaticall-4.png\")
<\/p>\n
Considerations about the link between the Workspace and the Repository<\/h2>\n
The folder is one of the key values we need to choose when linking the workspace with the source control. Choosing a folder is a difference between linking the workspace to the entire repository or with one specific folder of the repository. This brings many different process options to be chosen on the correct situation:<\/p>\n
\n- We can have a repository containing a bigger data engineering project, where the power bi workspace is only a piece of the project, contained in one of the folders.<\/li>\n
- We can have a single repository containing multiple workspaces, each workspace in a different folder.<\/li>\n
- We can have one repository for each workspace, linking the workspace with the root folder<\/li>\n<\/ul>\n
Can you think about more possibilities? Tell me in the comments.<\/em><\/p>\nOnce we define a process to work with source control, the Publish<\/strong> from Power BI Desktop<\/strong> will not need happen anymore. We will only need to push to a repository and the repository linked to the workspace. The portal will synchronize with the repository.<\/p>\nWhen you open the report, you will notice it\u2019s completely empty. The data was never published to the repository. In this way, the reports appear empty until we make the first dataset refresh.<\/p>\n
Even so, the data will not be published to the repository. You may remind the PBIP generation created a .GitIgnore<\/em> file which excludes the data. In this way, the dataset refreshes are independent of the source control and branch checkouts. We will see more demonstrations of this further in this article.<\/p>\nWorkspace and Source Control: Technical Features<\/h2>\n
The user can see, use and change the integration with the source control on the following ways:<\/p>\n
![\"Everything](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/everything-you-must-know-before-getting-started-wi-2.png\")
<\/p>\n![](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/word-image-100615-18-1.png\")
<\/p>\n![](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/word-image-100615-19-1.png\")
<\/li>\n![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-computer-description-automatica-111.png\")
<\/li>\n![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-checkout-branch-description-aut-3.png\")
<\/li>\n![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-computer-description-automatica-112.png\")
<\/p>\n![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-computer-description-automatica-113.png\")
<\/li>\n![](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/word-image-100615-24-1.png\")
![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-computer-description-automatica-114.png\")
<\/li>\n![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-black-rectangular-object-with-a-red-border-desc-2.png\")
<\/li>\n![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-computer-description-automatica-115.png\")
<\/li>\n![\"A](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2023\/11\/a-screenshot-of-a-computer-description-automatica-116.png\")
<\/li>\n