Let’s consider a simple statement for partitioning and save a table in a lakehouse: df.write.mode("overwrite").format("delta").partitionBy("Year","Month","Day").save("Tables/" + table_name) Let’s consider we load the data daily, with all the transactions from the day. The table will save the transactions for each day in different partitions. We can expect the table to keep the partitions from previous day, … Read more
On the blog Fabric Notebook and Deployment Pipelines I explained a technique to keep notebooks configuration values in JSON files on lakehouses, a good solution from many different points of views. What if we need to provide maintenance to the JSON configuration file using notebooks? The first problem is the fact the typical statement to … Read more
Dataflows Gen 2 are the new version of Power BI dataflows. There are so many changes in relation to the previous version they are considered a new feature. The main difference is the possibility to set a target for the result of each query in the dataflow. In this way, it can be used as … Read more
The animation on the top of this article tries to track the evolution of the enterprise architecture since SQL Server 7.0 introduced tools for ETL, Semantic model and much more. Some of you probably remember these tools as SSIS and SSAS. At that time they had even older names, but no one wants to confess remembering … Read more
The possibility to use Visual Studio Code (or VS Code) to develop your Microsoft Fabric notebooks seems very interesting. It may bring many benefits for usability and for the SDLC (Software Development lifecycle): You may prefer developing on your own machine than on the interface of a portal It’s possible to develop and test before … Read more
There are some differences and secrets between the UI of a SQL Endpoint and the UI of a Lakehouse. I believe the lakehouse UI was changed recently, otherwise you can blame me for being distracted to this level. Let’s analyze the differences and pending points between these UI’s. SQL Endpoints When using a SQL Endpoint, … Read more
Microsoft Fabric was first announced during Microsoft BUILD. Since then, I’m publishing content about Microsoft Fabric, creating an interesting sequence of content. In this blog, I’m summarizing the content I published about Fabric, helping you to navigate on this content. Main Articles As you may have noticed, there is one article about concepts, one about … Read more
Microsoft Fabric storage uses OneLake and Delta Tables, the core storage of all Fabric objects, as explained in my introduction to Microsoft Fabric. Either if you use lakehouses, data warehouses, or even datasets, all the data is stored in the OneLake. OneLake uses Delta Tables as storage. In this way, we need to be aware … Read more
Download the notebook used on this blog I explained in a previous article how the Tables in a lakehouse are V-Order optimized. We noticed this configuration depends on our settings, which can be enabled or not. One question remains: How could we check if the tables are V-Order optimized or not? The tables we will … Read more
When Microsoft Fabric was born, the only method to convert files to tables was using notebooks. Nowadays we have an easy-to-use UI feature for the conversion. As I explained on the article about lakehouse and ETL, there are some scenarios where we still need to use notebooks for the conversion. One of these scenarios is … Read more
Loading On-Premises and external data is a fundamental need for a Data Platform such as Microsoft Fabric. However, during this preview stage, Microsoft Fabric brings many surprises (and, should I say, deceptions?) in this area. This blog may explain a lot, but I’m not bringing conclusions, but lot of questions. It will be a pleasure … Read more
I have been working as a no-code data engineer: Focused on Data Factory ETL and visual tools. In fact, I prefer to use visual resources when possible. On my first contact with Fabric Lakehouse I discovered to convert Files into Tables I need to use a notebook. I was waiting a lot of time for … Read more
The Data Wrangler is as interesting as hidden inside Microsoft Fabric. It’s not easy to find and activate it, but it is worth the trouble. Before digging into the Data Wrangler, let’s analyze the data features in the query environment. Data Features in Query Environment The new query environment, which allow us to make data … Read more
After the great announcement of Microsoft Fabric this week, it’s time to get a trial and start playing with this great product. Microsoft Fabric is a very broad product, so it requires some additional configuration for the start of a trial. Configuring the Tenant By default, the Power BI tenants are disable for Microsoft Fabric. … Read more
