Fabric Copy Job: What’s new in it

Comments 0

Share to social media

At first sight, the new Copy Job may seem only one more redundancy: Why do we need this?

We have Data Pipelines for the Copy Activity and also a Copy Assistant to help us configure it. What’s new about the Copy Job?

Spoiler: It’s much more than a new UI

The new Copy Job Object and UI

The Copy Job is exactly what the name says: a “dedicated data pipeline” intended only to execute a copy activity. We schedule it exactly as a data pipeline.

The new UI is cute, you finish the configuration with the job already scheduled for every 15 minutes by default.

A screenshot of a computer

Description automatically generated

These are only the small details.

Incremental Load with the Copy Job

In my opinion, this is the great advantage of this new feature: It builds an incremental load for you.

The configuration for incremental load requires an incremental column. A date/time column which will allow the job to identify changed records.

The source model is the responsible for updating the date/time column every time a record is updated. After that, the job manages all the work.

A screenshot of a computer

Description automatically generated

The job will make a first execution with a full load, and continue with incremental load every 15 minutes, by default.

Copy Job and Other Options

How this new feature compare with the other options?

We have the CDC Ingestion from Azure SQL and Mirroring from Azure SQL, how the new job compares with them?

Let’s enumerate some differences:

  • It is not real time, it works by schedule
  • It requires a special field in the source for the incremental work. CDC doesn’t
  • The Copy Job is not limited to Azure SQL, the two previous options are

You can also take a look on Fabric Monday 58: Copy Job and Incremental Load

Summary

Everyday Microsoft makes it easier to put data into Fabric

Article tags

Load comments

About the author

Dennes Torres

See Profile

Dennes Torres is a Data Platform MVP and Software Architect living in Malta who loves SQL Server and software development and has more than 20 years of experience. Dennes can improve Data Platform Architectures and transform data in knowledge. He moved to Malta after more than 10 years leading devSQL PASS Chapter in Rio de Janeiro and now is a member of the leadership team of MMDPUG PASS Chapter in Malta organizing meetings, events, and webcasts about SQL Server. He is an MCT, MCSE in Data Platforms and BI, with more titles in software development. You can get in touch on his blog https://dennestorres.com or at his work https://dtowersoftware.com