{"id":101029,"date":"2024-02-21T17:00:21","date_gmt":"2024-02-21T17:00:21","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=101029"},"modified":"2024-09-03T20:04:47","modified_gmt":"2024-09-03T20:04:47","slug":"eventstream-and-the-new-kql-processing","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/blogs\/eventstream-and-the-new-kql-processing\/","title":{"rendered":"Eventstream and the new KQL Processing"},"content":{"rendered":"

Eventstream<\/strong> has many differences in relation to the technologies it proposes to replace. Event Hub<\/strong>, Stream Analytics<\/strong>, Streaming Dataflows<\/strong> and more. We can compare these technologies, but EventStream<\/strong> in Microsoft Fabric<\/strong> has some specific differences from all of them.<\/p>\n

One of the differences is how the transformation of the input data is linked to the target of the data.<\/p>\n

On previous technologies, we could receive the data stream, define transformations for it, and send the result of the transformations to multiple targets. The features for the transformation were the same.<\/p>\n

On EventStream<\/strong>, this works differently. The transformations are linked directly to the target of the data. Lakehouse, as a target, has a specific set of transformations possible.<\/p>\n

On November, Fabric announced the transformation features in the KQL<\/strong> target. This is a considerable advance. KQL<\/strong> target is usually the Speed Layer<\/strong> in a lambda architecture, while the lakehouse is the Batch Layer<\/strong>.<\/p>\n

\"A<\/p>\n

 <\/p>\n

The possibility of making transformations on the Batch Layer<\/strong> and not on the Speed Layer<\/strong> was a problem. Now we can make the transformations on both.<\/p>\n

KQL Event Processing in EventStream<\/strong><\/h2>\n

When configuring the target, we can choose between two options:<\/p>\n

    \n
  • Direct Ingestion<\/li>\n
  • Event Processing before ingestion<\/li>\n<\/ul>\n

    \"A<\/p>\n

    In my opinion, the most common scenario for a Speed Layer<\/strong> is to transform and summarize the data, while the Batch Layer<\/strong> may ingest the data as is.<\/p>\n

    \"A<\/p>\n

    However, you may face many different scenarios and challenges.<\/p>\n

    Transformations<\/strong><\/h2>\n

    We have some simple transformations:<\/p>\n

      \n
    • Manage Fields<\/li>\n
    • Filter<\/li>\n
    • Expand<\/li>\n<\/ul>\n

      These are some transformations for grouping data:<\/p>\n

        \n
      • Group By<\/li>\n
      • Aggregate<\/li>\n<\/ul>\n

        This is a transformation using multiple sources:<\/p>\n

          \n
        • Union<\/li>\n<\/ul>\n

          \"A<\/p>\n

          Summary<\/strong><\/h2>\n

          This feature is absolutely needed. In my opinion, it’s strange to have the transformations linked to the target. However, we have features to make transformations for the Speed Layer<\/strong> and for the Batch layer<\/strong>, allowing us to build a complete lambda architecture starting with a single EventStream.<\/p>\n

          The EventStream<\/strong> is different, but it seems more powerful than all previous features it replaces.<\/p>\n

           <\/p>\n","protected":false},"excerpt":{"rendered":"

          Eventstream has many differences in relation to the technologies it proposes to replace. Event Hub, Stream Analytics, Streaming Dataflows and more. We can compare these technologies, but EventStream in Microsoft Fabric has some specific differences from all of them. One of the differences is how the transformation of the input data is linked to the……<\/p>\n","protected":false},"author":50808,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[2,159164],"tags":[159056,126250,158997,159057],"coauthors":[6810],"class_list":["post-101029","post","type-post","status-publish","format-standard","hentry","category-blogs","category-microsoft-fabric","tag-eventstream","tag-kql","tag-microsoft-fabric","tag-real-time"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/101029","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/users\/50808"}],"replies":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/comments?post=101029"}],"version-history":[{"count":3,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/101029\/revisions"}],"predecessor-version":[{"id":101036,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/101029\/revisions\/101036"}],"wp:attachment":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media?parent=101029"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/categories?post=101029"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/tags?post=101029"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/coauthors?post=101029"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}