In my previous article StreamInsight Service for Windows Azure about the “Project Austin” StreamInsight service, I explained how to provision it on Windows Azure with Visual Studio 2010 or 2012 and the downloadable AustinCtpSample solution. StreamInsight is a Complex event processing (CEP) application that tracks streams of information that combines data from several sources in order to identify meaningful events and respond to them rapidly.
In this article, I’ll be showing you how to test the StreamInsight Service for Windows Azure CTP with the SampleApplication and EventSourceSimulator projects and use the graphical Event Flow Debugger. To do this, you will need the following: (see StreamInsight Service for Windows Azure).
- Visual Web Developer 2010 Express or Visual Studio Express 2012 for Web or higher version
- Windows Azure SDK v1.7 (June 2012), which includes the Visual Studio Tools for Windows Azure
- A Windows Azure trial or paid subscription, preferably dedicated to StreamInsight
- An invitation to the StreamInsight Service for Windows Azure CTP, the source code and binaries for which you must download and install
- Completion of all sections of Part 1 except “Deleting a Service Instance Deployment”
Run the AustinCtpSample Solution’s SampleApplication Project
The SampleApplication project’s Program.cs file includes code to connect to and manage a named Project “Austin” instance, and define, deploy and run queries in that instance. There’s also code to deploy (and, optionally, redeploy) the following .NET assemblies created by the SampleApplication project to that instance:
AzureTableSink class represents an Observer that writes result events into a Windows Azure table in the storage account you created in Part 1. The next section provides more information about event streams and sinks.
To configure and start the SampleApplication program, do the following:
- In Visual Studio’s Solution Explorer, right-click the SampleApplication node and choose ‘Set as Startup Project‘.
- Expand the SampleApplication project node, and double-click the App.config node to open the document in the editor window. Replace the AccountName placeholder with the storage account name, oakaustin for this example, the AccountKey placeholder with the value copied from the portal, and the austinDnsPrefix placeholder in three places (see Figure 1).
- Double-click the Program.cs item to open the C# code in the editor window, expand the first region and replace austinDnsPrefix in lines 118 and 125 as shown in Figure 2.
- Press F5 to deploy input and output adapters and a query, and start the query, which displays progress in a console window (see Figure 3):
At this point, the SampleApplication is running in the StreamInsight service and waiting for a stream of events (
Observable sequences) to arrive at the ingress endpoint. .NET 4.0’s Reactive Extensions (Rx) library enables the composing of asynchronous and event-based programs using
Observable sequences and LINQ-style query operators. To learn more about the .NET Reactive Extensions library, see MSDN’s Reactive Extensions topic and subtopics.
Processing Emulated Events withStream Insight Queries
C# code in the SampleApplication project’s Program.cs file deploys an event sink, AzureTableSink, that passes httpSource events through to a Windows Azure
HttpMessages Table. Queries generate average values that populate HighDeviceAverages, DeviceAverages and LowDeviceAverages tables (see Figure 4.)
The Main program in the SampleApplication project’s Program.cs file defines the LINQ
highaverages temporal aggregate queries with the following code:
var averages = from e in deviceStream
group e by e.DeviceId into gs
from win in gs.TumblingWindow(TimeSpan.FromSeconds(5))
select new DevicePayload
DeviceId = gs.Key,
Value = win.Avg(e => e.Value),
var highAverages = averages
.Where(e => e.Value > 10);
The EventSourceSimulator project generates an external event stream to validate the service deployment and its interaction with the SampleApplication project. To send one event by the HTTP GET and 60 by HTTP POST methods, follow these steps:
- Reprovision the StreamInsight instance and set up the SampleApplication, if you deleted the instance in the preceding section.
Note: Consider changing the IngressProtocol from https to http in line 46 of the ServiceProvisioningproject’s app.config file to minimize potential certificate issues.
- If you chose http in Part 1’s step 1, replace https with http in lines 36 and 37 of the EventSourceSimulator project’s Program.cs file (see Figure 5.)
- Press F5 to build and run the EventSourceSimulator project, which sends a total of 61 events to the SampleApplication in about one minute (see Figure 6.)
- Launch a Windows Azure Storage utility, such as Cerebrata Storage Studio for this example, connect to the storage account and expand the tables list to expose the four tables created and populated by the simulated events (see Figure 7.)
Instance-specific tables prefixed siXoakaustinX provide detailed logs of activities involved with deploying and dismantling application objects, such as theHttpEventPipeline (see Figure 8).
Enabling Verbose Windows Azure Diagnostics and Monitoring Service Activity
The Austin August CTP enabled Windows Azure Diagnostics (WAD) for tracing StreamInsight Service execution. By default, only the Minimal diagnostics level is enabled by default in the Management Portal’s Cloud Service Dashboard’s Configure page for the instance. To increase diagnostics granularity, click the Verbose and Save buttons (see Figure 9).
Verbose diagnostics add Windows Azure tables named WADGUIDPT##XRTable to the storage utility’s display (see Figure 10.)
Configure and Run the EventFlowDebugger Program
The Austin CTP binaries include a graphical StreamInsight Event Flow Debugger program (C:\Program Files\Microsoft StreamInsight 2.1\Tools\EventFlowDebugger.exe). MSDN’s Using the StreamInsight Event Flow Debugger topic provides details for using the program.
The Debugger has an EventFlowDebugger.exe.configconfiguration file in the same folder for on-premises StreamInsight instances. To reconfigure for Windows Azure deployment and run the debugger, follow these steps:
- Rename EventFlowDebugger.exe.configto EventFlowDebugger.exe.orig.
- In Solution Explorer, expand the Other resources node and double click the EventFlowDebugger.exe.configitem to open it in the editor window.
- Replace two instances of austinDnsPrefix with your service name, oakaustin for this example (see Figure 11.)
- Chose File, Save EventFlowDebugger.exe.configAs to open the Save File As dialog, navigate to the C:\Program Files\Microsoft StreamInsight 2.1\Tools\ folder and click Save.
- Choose Start, All Programs, Microsoft StreamInsight v2.1 x64 (en), StreamInsightEventFlowDebugger to start the debugger.
- Choose File, Connect to Server to open the eponymous dialog, select Options to expand it, select the Use Predefined Endpoint option and accept the default MyAustinEndpoint in the list (see Figure 12.)
- Click OK to open the debugger window and expand the Applications and Processes nodes to display running queries. Right-click a StreamableBinding_1 query item and choose Display Query to open a query graph (see Figure 13.)
- Right-click the Http Messages Passthrough Process node’s StreamableBinding_1 query item and choose ‘Show Diagnostics‘ to display details of the first emulated event, which used the HTTP GET method (see Figure 14.)
- Right-click a ‘High Averages Process‘ node’s StreamableBinding_1 query item and choose‘how Diagnostics‘to display details of the next 60 emulated events, which used the HTTP POST method (see Figure 15.)
If you want to take advantage of the Event Flow Debugger’s event recording, replay, single-step and breakpoint features, you must install the local server bits with StreamInsight_AustinAugustCTP_x64.msi or StreamInsight_AustinAugustCTP_x86.msi from the August CTP distribution, which is compatible with StreamInsightClient_AustinAugustCTP_x64 or StreamInsightClient_AustinAugustCTP_x86, respectively.
StreamInsight is rapidly maturing with closely spaced CTPs and the latest update for the StreamInsight Service for Windows Azure promises a larger potential customer base. There’s a serious shortage of developers with on-premises or cloud-based StreamInsight skills, as well as for the newly christened HDInsight (Hadoop) Service for Windows Azure and Windows Server. Leverage your .NET chops by learning to write and test LINQ queries for processing fast-moving streams of financial or sensor data to give your career a healthy boost.