Dataflows Gen 2 are not supported by source control and deployment pipelines yet.
When we need to promote a dataflow from Dev to Test and Prod, we need to move the dataflow manually.
However, the dataflows don’t have a Save As feature. It’s possible to edit the JSON from the queries, but it would need to be one by one, so it’s not the best option.
Let’s see the steps needed to move a dataflow from one workspace to another. For a matter of example, let’s consider the dataflow access a lakehouse on the Dev environment. When copied to the Test environment, the new copy needs to be linked to a new dataflow in the Test environment.
Check the steps to copy a Dataflow Gen 2 to another workspace:
- Open the lakehouse on the Test workspace and copy its URL. You will need it later. The URL contains the Workspace Id (after groups) and the lakehouse Id (after lakehouses)
- Export the dataflow Gen 2 as a template
A file with a PQT extension will be downloaded
- On the Test workspace, create a new Dataflow Gen 2
- Click on the link in the middle of the window to Import the template – the PQT file
Repeat the steps below for each query:
- Select the Navigation1 step
- Change the workspace Id on this step using the workspace Id from the captured lakehouse URL
- Select the Navigation 2 step
- Change the lakehouse Id using the Id captured from the Test lakehouse URL
- Adjust the dataflow name and publish
Summary
The process is a bit complex, or at least, more complex than we would expect for a simple promotion from one environment to another.
However, we hope to only need this process during a short period, while we wait the dataflow support in the source control and deployment pipelines.
Load comments