2187-just_azure.svg

In my last article I’ve explained the main idea and concepts behind Azure Queues and tried to cover some aspects from an architectural, engineering and commercial point of view. Further, I’ve explained some of the difference between Service Bus Queues and Azure Queues, as well as spotlighted major milestones in the life of an Azure Queue message. What I didn’t show you was source code.

REST API: Behind the curtain

The most basic way of accessing Cloud Services can be done via REST calls, although it’s not the easiest one. Nevertheless let’s take a quick look at how things work behind the curtain before we move on to more high-level APIs.

REST request for listing all existing queues in a storage account:

Life could be so easy without security, right? The above GET request calls the queue endpoint of your storage account (http://<STORAGE_ACCOUNT_NAME>.queue.core.windows.net) and asks for all existing queues (parameter comp=list). Because you don’t want to allow everyone to access your queue you have to authenticate with a signed signature string – and that’s the most time-consuming part of the story. It would take us too long to explain the details, but if you need to work with the Queues Service REST API directly, further information about authentication can be found at Authentication for the Azure Storage Services.

Variety of APIs: Choose which one fits best

Azure is not just a .NET thing. As long as the services run on Azure, Microsoft is fine with many programming languages on the client – and that’s not only true for Queues. You can find official wrapper APIs for .NET, Java, PHP, Node.js, Python and Ruby (September 2014) and they will generate the appropriate REST requests for you easily.

Samples in this series are mostly written in C#, but below you can see the exemplary creation of a queue in Python or Node.js. Feel free to use your favorite programming language when accessing Microsoft Azure.

Create a queue with Python

Sample based on How to Use the Queue Storage Service from Python, Windows Azure SDK for Python 2.7

Creating a queue with Python

Create a queue with Node.js

Note: some environment variables that specify the account and the access key have to be set up additionally (sample based on Microsoft Azure Storage SDK for Node.js, Microsoft Azure SDK for Node.js 0.9.16)

Creating a queue with Node.js

Scenario: Ticketing as a Service (TaaS)

I’d like to introduce my sample scenario that will be used for the rest of this JustAzure series: TaaS – Ticketing as a Service. When a new event gets public, many people around the world can’t wait for the registration to open. They will hit F5 twice a second just to notice when the time has come to purchase a ticket. For every ticket ordered you have to check the availability first and create an invoice that is sent through email to the user.

You have a typical solution where scalability is a major requirement: for those few hours you really need performance whilst after a week no one cares about your registration site. Hence you should be very fast and resource-saving in the hot phase and it’s not a good idea to spend 5-10 seconds per ticket purchase on the webserver.

Queues are a great way to separate the intention to buy a ticket from the time-consuming processing part. A queue message representing the ticket is created immediately, but the processing is done afterwards by as many workers as you currently need in an asynchronous way.

Note: While Queues are of course not the only way to solve this problem, features such as try/retry or easy parallelization come in very handy in this scenario.

Set up your Azure subscription

It should be clear that you need an Azure subscription to try out the following samples. If you don’t have one you can sign-up for a free one month trial here. If you have an MSDN Subscription you can also take advantage of some Azure benefits from MSDN to give you Azure credit each month.

Queues, table storage and blob storage are unified under a Storage Account and you probably have to create one now. From the Azure Management portal click the New button in the lower left hand corner and select Data Services, Storage, Quick Create. Just specify a name, the location of your storage and your preferred replication strategy.

2215-Queues-2-Fig-01-storageaccount-620.

Figure 1: Creating a storage account

Set up your project

For the first API tests I’ll create a simple Console application to demo a few things. To access your queues from .NET you will need the Storage API which can be referenced easily as a NuGet package named Windows Azure Storage (the sample code uses Storage API 4.3.0).

2215-Queues-2-Fig-02-nuget-620.png

Figure 2: Adding storage reference via Nuget

Create a queue

Let’s create a queue. Of course you can do that with your favorite tool too (e.g. Visual Studio or Azure Management Studio), but as you’ve seen already it’s also possible through the API. The main class for your communication is called CloudQueueClient and an instance is provided through a factory method from the CloudStorageAccount class.

To access your storage account you will need StorageCredentials: take the previously specified name of your storage account and one of the access keys that you can find for your storage account in the Azure portal (How to: View, copy, and regenerate storage access keys). The second parameter of the CloudStorageAccount constructor states if you want to use HTTPS for the communication.

With the CloudQueueClient.GetQueueReference method you can get a reference to your queue. Before working with the queue, you have to create it by using CreateAsync or Think carefully if you want to spend this additional call every time you’re accessing your queue, usually you can setup your queue once and there’s no need to ensure the existence again and again.

Creating a queue with .NET SDK

Inserting messages

Next we want to insert a TaaS ticket request in our newly created queue. Queue messages can be stored as string or as byte array but be aware of the size limits and the base64 encoding of the .NET Storage API (see my previous article, “Adding a message: Don’t be chatty”).

I use Newtonsoft.Json to serialize my simple TicketRequest object and pass that JSON formatted string to the constructor of the CloudQueueMessage class. Just call AddMessageAsync on your queue instance and you’re done!

Inserting a message into the queue

There are some overloads for the AddMessageAsync method that enable you to specify additional options for the message, e.g. the maximum time to allow the message to be in the queue (time-to-live) or the initial invisibility delay. See MSDN for further information.

Processing and deleting messages

Worker roles are probably the best place to process queue messages. Worker roles can be run in parallel, you have auto-scaling features that can depend on queue sizes and they are quite easy to implement. I’ll show a realistic architecture in my last article about queues, but for now you can put the code in your Console application as well.

Processing a message

There are several options to get messages from a queue: you can peek messages (PeekMessagesAsync) just to see what’s in the queue, but you won’t own those returned messages exclusively. Using GetMessagesAsync you get this exclusive access, but only for a specified amount of time. This time span is limited to a maximum of 7 days and can be provided as a parameter in the GetMessagesAsync method. The above used overload of the method optionally takes QueueRequestOptions (e.g. server timeout or retry policy) and an OperationContext (for additional tracing information) as third and fourth parameter.Finally, you can also read more than one message per request, but you cannot read more than 32 messages at once. Reading more messages with one request has a positive performance effect, but don’t be too greedy: you must accomplish to process all the messages during the specified timeout period.

To deserialize the JSON string I use Newtonsoft.Json and pass the reconstructed object to my long-running processing operation. If something goes wrong, the message automatically becomes visible for other workers after the above mentioned amount of time – and if the processing of the ticket completes in time, you have to delete the message manually to ensure the opposite.

One thing I’d like to mention is the DequeueCount property. If you don’t take care, erroneous messages will live forever in your queue: as long as no one deletes them, they will become visible again and again (that’s why they are also called “poisonous messages”). To prevent this behavior you can check the DequeueCount of each message: the counter automatically increments with every exclusive read. A good pattern is the creation of a so-called deadletter queue and to move all the poisonous messages into that queue for further investigation.

Summary

In this article I showed some basic samples for the communication with Azure Storage Queues. I used the Storage API from .NET but there are many more APIs for different programming languages available. Next time we’ll go deeper into this topic and talk about security and performance optimization. I’d truly appreciate any comments or suggestions to help make this series more useful, so feel free to contact me!