Testing the StreamInsight Service for Windows Azure

Getting 'up to speed' with StreamInsight is easier if you take the time to run it and test it out. Roger Jennings expains how to be up and running in nine simple steps, once you've sorted out the prerequisites

In my previous article StreamInsight Service for Windows Azure about the “Project Austin” StreamInsight service, I explained how to provision it on Windows Azure with Visual Studio 2010 or 2012 and the downloadable AustinCtpSample solution. StreamInsight is a Complex event processing (CEP) application that tracks streams of information that combines data from several sources in order to identify meaningful events and respond to them rapidly.

In this article, I’ll be showing you how to test the StreamInsight Service for Windows Azure CTP with the SampleApplication and EventSourceSimulator projects and use the graphical Event Flow Debugger. To do this, you will need the following: (see StreamInsight Service for Windows Azure).

  • Visual Web Developer 2010 Express or Visual Studio Express 2012 for Web or higher version
  • Windows Azure SDK v1.7 (June 2012), which includes the Visual Studio Tools for Windows Azure
  • A Windows Azure trial or paid subscription, preferably dedicated to StreamInsight
  • An invitation to the StreamInsight Service for Windows Azure CTP, the source code and binaries for which you must download and install
  • Completion of all sections of Part 1 except “Deleting a Service Instance Deployment”

Run the AustinCtpSample Solution’s SampleApplication Project

The SampleApplication project’s Program.cs file includes code to connect to and manage a named Project “Austin” instance, and define, deploy and run queries in that instance. There’s also code to deploy (and, optionally, redeploy) the following .NET assemblies created by the SampleApplication project to that instance:

  • StreamInsight.Samples.Austin.EventTypes.dll
  • StreamInsight.Samples.Austin.AzureTableSink.dll

The AzureTableSink class represents an Observer that writes result events into a Windows Azure table in the storage account you created in Part 1. The next section provides more information about event streams and sinks.

To configure and start the SampleApplication program, do the following:

  1. In Visual Studio’s Solution Explorer, right-click the SampleApplication node and choose Set as Startup Project.
  2. Expand the SampleApplication project node, and double-click the App.config node to open the document in the editor window. Replace the AccountName placeholder with the storage account name, oakaustin for this example, the AccountKey placeholder with the value copied from the portal, and the austinDnsPrefix placeholder in three places (see Figure 1).


Figure 1. Replace Accountname, AccountKey and austinDnsPrefix in the three places shown here.

  1. Double-click the Program.cs item to open the C# code in the editor window, expand the first region and replace austinDnsPrefix in lines 118 and 125 as shown in Figure 2.


Figure 2. Replace austinDnsPrefix on lines 118 and 125 with the Cloud Service name, oakaustin for this example.

  1. Press F5 to deploy input and output adapters and a query, and start the query, which displays progress in a console window (see Figure 3):


Figure 3. Type y when asked if you want to (re)deploy the user assemblies by the SampleApplication‘s console.

At this point, the SampleApplication is running in the StreamInsight service and waiting for a stream of events (Observable sequences) to arrive at the ingress endpoint. .NET 4.0’s Reactive Extensions (Rx) library enables the composing of asynchronous and event-based programs using Observable sequences and LINQ-style query operators. To learn more about the .NET Reactive Extensions library, see MSDN’s Reactive Extensions topic and subtopics.

Processing Emulated Events withStream Insight Queries

C# code in the SampleApplication project’s Program.cs file deploys an event sink, AzureTableSink, that passes httpSource events through to a Windows Azure HttpMessages Table. Queries generate average values that populate HighDeviceAverages, DeviceAverages and LowDeviceAverages tables (see Figure 4.)


Figure 4. An event flow diagram for the SampleApplication project. Processing events creates the four Windows Azure tables if they don’t already exist; queries add entities to them depending on average values.

The Main program in the SampleApplication project’s Program.cs file defines the LINQ averages and highaverages temporal aggregate queries with the following code:

The EventSourceSimulator project generates an external event stream to validate the service deployment and its interaction with the SampleApplication project. To send one event by the HTTP GET and 60 by HTTP POST methods, follow these steps:

  1. Reprovision the StreamInsight instance and set up the SampleApplication, if you deleted the instance in the preceding section.

Note: Consider changing the IngressProtocol from https to http in line 46 of the ServiceProvisioningproject’s app.config file to minimize potential certificate issues.

  1. If you chose http in Part 1’s step 1, replace https with http in lines 36 and 37 of the EventSourceSimulator project’s Program.cs file (see Figure 5.)


Figure 5. The EventSourceSimulator‘s protocol must match that of the provisioned StreamInsight instance.

  1. Press F5 to build and run the EventSourceSimulator project, which sends a total of 61 events to the SampleApplication in about one minute (see Figure 6.)


Figure 6. The EventSourceSimulator project sends a single event by the HTTP POST method followed by 30 events each with sequentially increasing values from two simulated devices (X and Y) by the HTTP POST method. The POST payload is in JSON format.

  1. Launch a Windows Azure Storage utility, such as Cerebrata Storage Studio for this example, connect to the storage account and expand the tables list to expose the four tables created and populated by the simulated events (see Figure 7.)


Figure 7. The entities of theHighDeviceAverage table, shown here in Cerebrata Storage Explorer, alternate between devices X and Y.

Instance-specific tables prefixed siXoakaustinX provide detailed logs of activities involved with deploying and dismantling application objects, such as theHttpEventPipeline (see Figure 8).


Figure 8. Entities in this siXoakaustinXIngressLog table display events for deletion and creation of the HttpEventPipeline for processing events.

Enabling Verbose Windows Azure Diagnostics and Monitoring Service Activity

The Austin August CTP enabled Windows Azure Diagnostics (WAD) for tracing StreamInsight Service execution. By default, only the Minimal diagnostics level is enabled by default in the Management Portal’s Cloud Service Dashboard’s Configure page for the instance. To increase diagnostics granularity, click the Verbose and Save buttons (see Figure 9).


Figure 9. Increase tracing granularity by changing the Monitoring Level from Minimal to Verbose. The warning flag advises that you might incur storage charges. The August CTP enables Windows Azure Diagnostics and adds default Diagnostics Connection Strings.

Verbose diagnostics add Windows Azure tables named WADGUIDPT##XRTable to the storage utility’s display (see Figure 10.)


Figure 10. Enabling Verbose tracing adds detailed WAD tables to your storage account. Table lifetime is fixed at 10 days to prevent excessive storage costs.

Configure and Run the EventFlowDebugger Program

The Austin CTP binaries include a graphical StreamInsight Event Flow Debugger program (C:\Program Files\Microsoft StreamInsight 2.1\Tools\EventFlowDebugger.exe). MSDN’s Using the StreamInsight Event Flow Debugger topic provides details for using the program.

The Debugger has an EventFlowDebugger.exe.configconfiguration file in the same folder for on-premises StreamInsight instances. To reconfigure for Windows Azure deployment and run the debugger, follow these steps:

  1. Rename EventFlowDebugger.exe.configto EventFlowDebugger.exe.orig.
  2. In Solution Explorer, expand the Other resources node and double click the EventFlowDebugger.exe.configitem to open it in the editor window.
  3. Replace two instances of austinDnsPrefix with your service name, oakaustin for this example (see Figure 11.)


Figure 11. Configure the EventFlowDebugger for use with instances deployed to Windows Azure.

  1. Chose File, Save EventFlowDebugger.exe.configAs to open the Save File As dialog, navigate to the C:\Program Files\Microsoft StreamInsight 2.1\Tools\ folder and click Save.
  2. Choose Start, All Programs, Microsoft StreamInsight v2.1 x64 (en), StreamInsightEventFlowDebugger to start the debugger.
  3. Choose File, Connect to Server to open the eponymous dialog, select Options to expand it, select the Use Predefined Endpoint option and accept the default MyAustinEndpoint in the list (see Figure 12.)


Figure 12. Specify the connection to the Windows Azure StreamInsight instance.

  1. Click OK to open the debugger window and expand the Applications and Processes nodes to display running queries. Right-click a StreamableBinding_1 query item and choose Display Query to open a query graph (see Figure 13.)


Figure 13. Monitor activity, as well as debug applications and instances in the Stream Insight Event Flow Debugger configured for hosting by Windows Azure Cloud Services. Selecting a query enables the Start Recording Events button; however, you can’t record events of a remote StreamInsight server. Clicking it creates a local *.etf file that contains code to define the graph but no event data.

  1. Right-click the Http Messages Passthrough Process node’s StreamableBinding_1 query item and choose ‘Show Diagnostics to display details of the first emulated event, which used the HTTP GET method (see Figure 14.)


Figure 14. The initial event streamed with the HTTP GET method generates two outgoing events.

  1. Right-click a ‘High Averages Process node’s StreamableBinding_1 query item and choosehow Diagnosticsto display details of the next 60 emulated events, which used the HTTP POST method (see Figure 15.)


Figure 15. The 60 events streamed with the HTTP POST method generate 164 outgoing events.

If you want to take advantage of the Event Flow Debugger’s event recording, replay, single-step and breakpoint features, you must install the local server bits with StreamInsight_AustinAugustCTP_x64.msi or StreamInsight_AustinAugustCTP_x86.msi from the August CTP distribution, which is compatible with StreamInsightClient_AustinAugustCTP_x64 or StreamInsightClient_AustinAugustCTP_x86, respectively.


StreamInsight is rapidly maturing with closely spaced CTPs and the latest update for the StreamInsight Service for Windows Azure promises a larger potential customer base. There’s a serious shortage of developers with on-premises or cloud-based StreamInsight skills, as well as for the newly christened HDInsight (Hadoop) Service for Windows Azure and Windows Server. Leverage your .NET chops by learning to write and test LINQ queries for processing fast-moving streams of financial or sensor data to give your career a healthy boost.