15 May 2024
15 May 2024

Data and SaaS : Who owns your company’s Data?

On the era of SaaS, I see many companies hiring a lot of SaaS services. They create a dependency on them for their production operations. It may seems the company is saving work and being modern when joining e-commerce portals with SaaS ERP’s and other online providers. Event-driven architectures are modern, aren’t them? They can … Read more
08 May 2024
08 May 2024

Edit The JSON of a Fabric Pipeline

A Fabric Pipeline uses JSON as source code. They are also saved in repositories as JSON. We first idea we get is editing the pipeline in JSON format. We can copy the JSON and create new pipelines with small variations, making changes directly on the JSON. However, at first sight we get disappointed, because the … Read more
30 April 2024
30 April 2024

PySpark Secrets to use with Fabric

PySpark is a powerful language for data manipulation and it’s full of tricks. Let’s discover some of them. Control the Type of a NULL column If you are creating a pysspark dataframe, but one of the columns contains only null values (None), how could you control the type of the column? There is an interesting … Read more
23 April 2024
23 April 2024

4 Reasons to use KustoDB for Real-Time

When implementing real-time ingestion, we usually implement an architecture called lambda. Using the lambda architecture, KustoDB in Microsoft Fabric is always recommended for the speed layer. Do you know why? Let’s analyze in detail. 1 – KustoDB uses SSD KustoDB uses an internal SSD storage. Lakehouses use ADLS as their backend. In this way, Kusto … Read more
10 April 2024
10 April 2024

Azure OpenAI and RAG Security

The most watched videos on my youtube channel are related to security. Azure SQL Networking Secrets and Cloud Security: Secure Access from your Applications to Azure SQL explain private endpoints from different points of views: Azure SQL and Function Apps. The concepts of Private Endpoints apply to most Azure features and even the relation of … Read more
04 April 2024
04 April 2024

Building a Culture of Data

One of the major trends in enterprise computing, and really in enterprises themselves is an increased emphasis on data. My career has always revolved around data, but this is a new focus for many parts of the organization. Even business units that traditionally don’t care about data realize that access to more, and better, data … Read more
03 April 2024
03 April 2024

Dynamic Partitioning and a Simple Incremental Load

Let’s consider a simple statement for partitioning and save a table in a lakehouse: df.write.mode("overwrite").format("delta").partitionBy("Year","Month","Day").save("Tables/" + table_name) Let’s consider we load the data daily, with all the transactions from the day. The table will save the transactions for each day in different partitions. We can expect the table to keep the partitions from previous day, … Read more
13 March 2024
13 March 2024

5 Secrets about Dataflows Gen 2

Dataflows Gen 2 are the new version of Power BI dataflows. There are so many changes in relation to the previous version they are considered a new feature. The main difference is the possibility to set a target for the result of each query in the dataflow. In this way, it can be used as … Read more
06 March 2024
06 March 2024

Reverse Creating the PBIP file from Source Control

On my article about Source Control with GIT, Power BI and Microsoft Fabric, I illustrate how to use the PBIP file format to include Power BI reports and semantic models in a source control process and stablish a SDLC (Software Development Lifecycle) for Power BI. However, the complete explanation is based on saving the development using … Read more
01 March 2024
01 March 2024

Friendly Deletes in SQL Server

One of the first things you learn when working with SQL Server, and other transactional based SQL systems, is that set based operations perform best. If you are querying data, a cursor pulling individual rows doesn’t perform as well as a single query. It doesn’t matter if that cursor is on the client side or … Read more