Real-Time Intelligence: More than a name change

MS Fabric_ real time analytics
Comments 0

Share to social media

When everyone starts to announce Real-Time Intelligence in Microsoft Fabric as something new, I need to double check what’s happening: Am I crazy or is everyone else? Wasn’t this already there?

Finally, I realize that Real-Time Intelligence is a new name for Real-Time Analytics, and they are doing this so fast we don’t even have time to notice the difference.

What’s Real-Time Intelligence and what’s the difference from Real-Time Analytics?

What’s Real-Time Intelligence

The difference is very simple. Real-Time Intelligence is the Real-Time Analytics but:

  • It has additional features
  • It has features we were long waiting for
  • It has a new UI which levels up our work with real-time data

Microsoft is making a habit of announcing new features as “available” while we can’t see them in our tenants yet.

These are the promised features for Real-Time Intelligence:

 

These are the features we can see now in our tenants:

 

Real-Time Hub

The new UI comes in the form of the Real-Time Hub. It appears on the left-side bar, and it brings a completely new UI to work with real-time data.

 

Real-Time Intelligence Features

It seems like everything is getting a new name. It’s important to map the new names of the features with our current knowledge:

Event Ingestion

This is the eventstream. What’s new?

The evolved UI in Real-Time Hub allow us to start the process in an easier way using the Get Event button.

We also can visualize all our eventstreams in the environment, check if they are endorsed and choose to use them.

 

 

One of the very interesting promises for the EventStream is the capability to process the data independent of the destination of the data.

Today, we need to choose a destination and the processing depends on the destination. However, we will be able to process first and sending the processed data to multiple destinations.

 

´This is a new preview UI for the EventStream. We have to methods to reach it: Using the Get Event in the Real-Time Hub or creating a new event stream and selecting to use the preview UI

 

 

I’m proud that last year, when delivering a great session about real-time in Fabric during the Data Saturday Sofia, I pointed out that the fact the transformations were dependent on the target was a flaw.

That’s it: They aren’t any more. Should I say, “I told you!”?

 

 

You may would like to watch Fabric Monday 39: Enhanced Eventstream

 

Real Time Analytics

This feature mainly involves our Kusto database, the place for our data and analysis. Of course, we don’t call it a Kusto database anymore.

We create an EventHouse. An EventHouse is made to contain multiple Kusto databases. When the EventHouse create it already contains one.

The existing Kusto databases are being converted to EventHouses, we can’t create a Kusto Database anymore.

Real-Time AI

Let’s simplify and summarize this to what it really is: It’s a co-pilot to generate Kusto queries from your natural language questions about your data in the EventHouse.

It’s great, but are we really expecting managers and other people who need to query with natural language to reach this point?

Last year I would say no. Today, I would say maybe. Everything is evolving so fast that maybe we will need this kind of tool as well.

 

Real-Time Dashboards

This was another critical pending point in real-time analytics: The regular Power BI dashboards were not capable to support this data and refresh the data in real-time. A new kind of dashboard was needed.

KQL Querysets got a new button “Pin to Dashboard”. Like what we can do in a Power BI report, but in this case, it can be done directly from the queryset.

 

 

Real-Time Triggers

Real-Time Triggers is like a subset of what was Data Activator. Is Data Activator dying?

I wouldn’t say it’s dying, I would say it’s evolving in such a way it will be integrated to the UI in many different ways. In a not so distant future, we may not recognize anymore when we are using Data Activator. 

We should be able to generate triggers from many different places, but at the moment I could only locate one: In the Real-Time Hub, we open an EvenStream and we use the button Set Alert

 

One important new feature related to the triggers is the capability to run a Fabric Item, which can be a pipeline or a notebook.

You may would like to watch Fabric Monday 42: Taking Actions from your data

Event Driven Actions

Last, but not least, we have the Event Driven Actions.

In other words, we can trigger actions based on events, such as a file created in a storage.

In the way it is now, I think it’s quite incomplete. The most important item we would like to see is not there yet: Trigger an action based on a file added to the Files area of a lakehouse.

This would be great, because the lakehouse could be the drop zone and automatically trigger transformations on the data.

However, at the moment, there are only two types of events available:

  • Blob Storage Items
  • Fabric Workspace Items

Both options trigger our imagination, but don’t reach the main goal.

The blob storage events are only for Azure blob storage, not for files in Fabric. In this way, our drop zone would still need to be in Azure.

The Fabric workspace item doesn’t include Files in a lakehouse. In this way, it has mostly an administrative purpose. Do you see other purposes, besides the administrative one? Let’s talk on the chat.

The UI and the response for these events is quite the same as the triggers.

Summary

The news is great, but it’s in preview. Only to give us the taste of what’s coming.

About the author

Dennes Torres

See Profile

Dennes Torres is a Data Platform MVP and Software Architect living in Malta who loves SQL Server and software development and has more than 20 years of experience. Dennes can improve Data Platform Architectures and transform data in knowledge. He moved to Malta after more than 10 years leading devSQL PASS Chapter in Rio de Janeiro and now is a member of the leadership team of MMDPUG PASS Chapter in Malta organizing meetings, events, and webcasts about SQL Server. He is an MCT, MCSE in Data Platforms and BI, with more titles in software development. You can get in touch on his blog https://dennestorres.com or at his work https://dtowersoftware.com

Dennes's contributions