Eventstream and the new KQL Processing

Comments 0

Share to social media

Eventstream has many differences in relation to the technologies it proposes to replace. Event Hub, Stream Analytics, Streaming Dataflows and more. We can compare these technologies, but EventStream in Microsoft Fabric has some specific differences from all of them.

One of the differences is how the transformation of the input data is linked to the target of the data.

On previous technologies, we could receive the data stream, define transformations for it, and send the result of the transformations to multiple targets. The features for the transformation were the same.

On EventStream, this works differently. The transformations are linked directly to the target of the data. Lakehouse, as a target, has a specific set of transformations possible.

On November, Fabric announced the transformation features in the KQL target. This is a considerable advance. KQL target is usually the Speed Layer in a lambda architecture, while the lakehouse is the Batch Layer.

A diagram of a process

Description automatically generated

 

The possibility of making transformations on the Batch Layer and not on the Speed Layer was a problem. Now we can make the transformations on both.

KQL Event Processing in EventStream

When configuring the target, we can choose between two options:

  • Direct Ingestion
  • Event Processing before ingestion

A screenshot of a computer

Description automatically generated

In my opinion, the most common scenario for a Speed Layer is to transform and summarize the data, while the Batch Layer may ingest the data as is.

A screenshot of a computer

Description automatically generated

However, you may face many different scenarios and challenges.

Transformations

We have some simple transformations:

  • Manage Fields
  • Filter
  • Expand

These are some transformations for grouping data:

  • Group By
  • Aggregate

This is a transformation using multiple sources:

  • Union

A screenshot of a computer

Description automatically generated

Summary

This feature is absolutely needed. In my opinion, it’s strange to have the transformations linked to the target. However, we have features to make transformations for the Speed Layer and for the Batch layer, allowing us to build a complete lambda architecture starting with a single EventStream.

The EventStream is different, but it seems more powerful than all previous features it replaces.

 

Load comments

About the author

Dennes Torres

See Profile

Dennes Torres is a Data Platform MVP and Software Architect living in Malta who loves SQL Server and software development and has more than 20 years of experience. Dennes can improve Data Platform Architectures and transform data in knowledge. He moved to Malta after more than 10 years leading devSQL PASS Chapter in Rio de Janeiro and now is a member of the leadership team of MMDPUG PASS Chapter in Malta organizing meetings, events, and webcasts about SQL Server. He is an MCT, MCSE in Data Platforms and BI, with more titles in software development. You can get in touch on his blog https://dennestorres.com or at his work https://dtowersoftware.com