How Shortcuts affect Lakehouse’s Maintenance

Comments 0

Share to social media

I wrote about lakehouse maintenance before, about multiple lakehouse maintenances, published videos about this subject and provided sample code about it.

However, there is one problem: All the maintenance execution should be avoided over shortcuts.

The tables require maintenance in their original place. According to our solution advances, we start using shortcuts, lots of them. Our maintenance code should always skip shortcuts and make the maintenance only on the tables.

In this way, the maintenance in each lakehouse manages the tables in that lakehouse and delegates the maintenance of the shortcuts to their source location.

The Problem: how to identify which objects in a lakehouse are shortcuts?

This is not a straightforward task. It’s not something like a boolean value nor a check we can make on the object.

We need to use the Fabric API to list all shortcuts from the lakehouse. Once we have a list of all shortcuts, we can change the maintenance code to skip the shortcuts.

This is the original code for lakehouse maintenance:

We need an additional function to retrieve the list of shortcuts from the lakehouse:

The shortcuts are loaded as a collection because this makes it easier to filter the table names.

We need to change the main code to retrieve the shortcuts and skip them during the maintenance.

Conclusion

We need to be careful with shortcuts in our maintenance code

Article tags

Load comments

About the author

Dennes Torres

See Profile

Dennes Torres is a Data Platform MVP and Software Architect living in Malta who loves SQL Server and software development and has more than 20 years of experience. Dennes can improve Data Platform Architectures and transform data in knowledge. He moved to Malta after more than 10 years leading devSQL PASS Chapter in Rio de Janeiro and now is a member of the leadership team of MMDPUG PASS Chapter in Malta organizing meetings, events, and webcasts about SQL Server. He is an MCT, MCSE in Data Platforms and BI, with more titles in software development. You can get in touch on his blog https://dennestorres.com or at his work https://dtowersoftware.com