Building a Customised ALM Platform with TFS and VSO

The latest versions of Team Foundation Server are not only sophisticated, but extensible. Continue their course of openness, Microsoft have opened up TFS, and are making it possible to build custom systems on top of it. Karsten Kempe points out three new features that make it possible to build an ALM system to exactly suit you needs, using TFS as the underlying platform, and demonstrates what's possible with an open source dashboard and work-tracking system.

Today’s software development is becoming increasingly complex and varied. The time when companies could concentrate on a specific technology stack are over, and there is a new challenge for software development companies: Customers want to purchase and use products, regardless of the platform or device. In this process, there is a permanent demand for new features with no negative effects on quality. If the product is unable to cope with the demands, it is rigorously replaced by another one. As a result, software manufacturers use a host of tools to master these challenges and support their development processes in the best way possible.

Unfortunately, using several systems during coordination and implementation of software projects often leads to unwanted costs for everyone involved. A feedback or bug tracking system, a requirements list and a test catalogue – all these systems must be synchronised and coordinated to be able to achieve the best possible result. The good news is that Microsoft’s ALM platform for Team Foundation Server now offers an easy way to integrate external systems and applications and make it the customised ALM solution for any company.

The features enabling this degree of customisation are planned for the on-prem installations of Team Foundation Server 2015, and are already available to several millions of Visual Studio Online users (whether or not they’ve used them is a different question!) Essentially, Microsoft has implemented three key elements in the revamped platform to achieve such a high degree of flexibility:

  • The new REST API, the future basis interface for Visual Studio Online and Team Foundation Server,
  • The OAuth 2.0 authentication mechanism, and
  • The so-called Service Hooks.

The following sections will describe the options these three components enable.

Treading new paths

It goes without saying that you can just leave everything as it is and continue to use your existing systems to create and evolve your products. However, you should at least consider your options and evaluate the potential benefits you could uncover to sustainably improve your development process and tool environment.

Significant benefits can result from being able to integrate external systems into your ALM platform, regardless of whether this is your in-house issue tracking system or a commercial system. You not only do away with having to tediously transfer customer demands or errors to your Team Foundation Server, but you also never have to synchronise between different platforms again. For instance, your service centre staff will be delighted about the fact they can provide upset customers with information more quickly, and also that the development team has already identified an error which will be eliminated from the product by the next update.

The UserVoice platform is a prime example of how to provide two-way synchronisation from as early as the beta integration stage.

2191-UserVoiceInt-a3a99125-13f0-4c21-889

Figure 1: UserVoice – creating work items in Visual Studio Online

An additional benefit of the new integration features for Visual Studio Online and TFS 2015 is that (depending on the specific situation) you can obtain information at any time, from anywhere in the world. For example:

  • Being in a daily scrum meeting, and pulling up story sizing and burn-down data to see if the team is going to able to fulfil all tasks
  • Being in a management meeting, and having access to the current error rate in the product(s)
  • Being able to obtain targeted information, while on-the-go or on-site, about the customers’ level of satisfaction with the product

Of course, these ideas are just scratching the surface. If you think through what your team needs from your ALM platform (or experiment to see what works well), and then make targeted calls to your Team Foundation Server, you can achieve almost anything involving your software development data. The new REST API is exactly the right tool for this – it allows a targeted collection of data to be accessed from and by the on-prem TFS server and Visual Studio Online.

Processing data with the REST API

The REST API in question has already been successfully in use for several months in the TFS cloud variant (Visual Studio Online), and it will also be available with the next version of TFS on-premises.

Rest APIs are a type of web application interface and, true to the culture of the web, are not restricted by any technological boundaries, and are accessible with almost any programming language. As a result, the new REST API is an interface for any platform, and it significantly enhances the range of TFS options compared with conventional interfaces based on the .NET or Java SDKs.

Before we see this in action, let’s quickly recap – A so-called RESTful service is generally characterised by the following features:

  • It references all data objects with a unique URL
  • It uses standard http protocol methods (GET, PUT, POST and DELETE)
  • It can also return requested object as JSON serials.

These three features make Team Foundation Server’s REST API relatively easy to use, not least because the rest calls and responses themselves are well-structured. In this process, the URL is based on the following patterns:

Visual Studio Online (VSO):

https://{account}.VisualStudio.com/DefaultCollection/_apis[/{area}]/{resource}?api-version=1.0

Team Foundation Server (on-prem):

http://{servername}:8080/tfs/{collectionname}/_apis[/{area}]/{resource}?api-version=1.0

The {account} wildcard represents the actual Visual Studio Account and {servername} represents the actual Team Foundation Server (Version 2015) name. Whereas Visual Studio Online always requires the DefaultCollection, Team Foundation Server requires a certain {collectionname}. {area} is used to reference the data objects and {resource} describes the data object to be requested.

REST API units

The TFS REST API consists of the following sections:

2191-RestAPI-d2055f78-7d2c-405e-a485-09a

Figure 2: Visual Studio Rest API overview

Naturally, a handy reference guide to the API is available on the Visual Studio website.

As a rule, the Visual Studio Online REST API is in version 1.0 at the time of writing. However, because things are never quite that simple, there are a few units with more recent API versions and, for this reason, the API version must be listed in each HTTP request. Thankfully, Microsoft are doing a pretty good job of keeping track of the valid version, and you can find the details on the Visual Studio Online “Versions and History” page.

Requesting project information

We’ve recapped and I’ve outlined the basics, so now for an example: it is possible to request information on a certain project by entering the following request:

https://almsports.visualstudio.com/DefaultCollection/_apis/projects/f2c5c767-36ba-485a-b478-f6de0f8f84b3?api-version=1.0

The request is directed at the Visual Studio Online account almsportsand the project with the ID f2c5c767-36ba-485a-b478-f6de0f8f84b3. A JSON object is returned as the response, as standard for REST requests:

The structure of the return object is identical in almost every case. Looking at this example, the object features the basic data of the team project and below it lists “gitStart” as the project name in the property name. The object also always returns the id and url of the request for unique identification purposes.

A further object is included in the returned data, described by the defaultTeam property. It describes the project’s standard team, and also features the id, name and urlproperties.

In .NET console applications it is relatively easy to request and output data from a Visual Studio Online account. All it needs are a few lines of codes. The first step is to create an HttpClient with a using statement to be able to send and receive http requests and responses, respectively. As the REST API in Visual Studio Online returns JSON objects, the Accept header of the request must be “application/json”.

In the above example, authentication is taken care of by the alternative credentials of a VSO user (notice that the authentication header is set to “Basic”.) Please note: this type of authentication is definitely not intended for production applications, as the user name and password must be transferred in each http request. We will look at a better authentication type below, namely the OAuth 2.0 I mentioned at the start of the article.

Rather that have me run through examples, I recommend you visit GitHub at http://github.com/almsports/VSORestSample for a selection of presented code samples for your convenience. All you need to do is enter your account name and you’re all set.

Getting Work Item Information

Once you have familiarised yourself with the basics of the API, you will also be able to tackle more complex components, such as Work Item Tracking. I want to mention this in particular because, whereas simple Get requests are sufficient to request projection information, Work Item Tracking requires the entire bandwidth of the API to allow you to create or edit work items, read out attachments and histories, and assign tags.

2191-RestAPI_WorkItems-597d5aad-7576-496

Figure 3: Work item units within Rest API

Work item information can be requested in two different ways: Either you use a predefined query from a team project, or you create your own query using the Work Item Query Language (WIQL). Regardless of the method you ultimately opt for, both query types initially only return a list of work item IDs. Send a second request to obtain more detailed work item information, using one of the work item IDs as an additional identifier:

In this query, the returned work item IDs are listed with the requested data fields. In this process, it is important to make sure not to state a project name in the URL otherwise the service is unable to find the requested resource and it returns a “404 not found” error message.

Secure authentication and authorisation with OAuth 2.0

As mentioned above, the authentication type selected for VSORestSample is not secure and must not be used in productive applications. Thankfully, REST API supports the OAuth 2.0 authentication protocol, which means the requesting application must no longer provide the user credentials and transfer them upon opening interfaces. OAuth 2.0 is not a standard used exclusively by Visual Studio Online, but is rather a well-adopted protocol relied upon by many companies (e.g. Dropbox, Foursquare, GitHub and Google). In terms of general characteristics, OAuth 2.0:

  • is a standard for secure data transfers
  • requires user registration for authorisation
  • provides users with unique authentication

The underlying mechanism of OAuth is actually relatively simple:

  • First, an application must register for the requested service. This registration process usually requests data, such as the application name, website, logo, etc.
  • You then receive a Client ID and a Client Secret. The Client ID features public information and is used within the application to open the service. In contrast, the Client Secret must be treated with absolute confidentiality to safeguard secure authentication.

OAuth 2.0 for Visual Studio Online

The authentication process for Visual Studio Online follows a similar pattern, although Figure 4 (below) illustrates that Visual Studio Online must also register the requesting application to open the API to it. During registration it is particularly important to list the correct API units, called “scopes”, to which the application is granted access. If incorrect scopes are selected, the application will later be unable to query these units and an error message is returned instead of data.

At this point it is also important to note that the scope cannot be changed after registration (as at April 2015). Consequently, selecting an incorrect API unit means the application must re-register to change its permission scope.

2191-OAuth-4d1f7e41-ddcc-4cee-9578-c08ff

Figure 4: Visual Studio Online OAuth mechanism

When a user wants to connect to their Visual Studio Online system with an external application, an authorisation request is sent to Visual Studio Online upon starting the application. Visual Studio Online then prompts the user to enter their credentials into the conventional Microsoft authentication dialogue, and then to authorise the application (i.e. enable the application to access and edit data on behalf of the user.) Visual Studio Online then transfers an authorisation code to the application which then uses said code to request an access token. This token enables the application to access data for the rest of the session without users having to authenticate themselves again.

Securely requesting data

To keep everything secure, each query the application makes must feature the access token to safeguard the functionality of the authentication mechanism. For this purpose, the http request from the VSORestSample is changed and amended by the token information:

Compared with the first example, notice that the authentication header now no longer features “Basic”, but rather “Authorization” as the scheme, and that the parameter has changed from “Username/Password” to the “bearer {Access Token}” token.

Please note: The current OAuth 2.0 workflow is not really suitable for native apps (desktop or mobile) as the aforementioned Client Secret must be stored in the app. As a result, there is the risk that the secret may be accessed by other applications and misused to read out data. Microsoft is currently working on a workflow for “public clients” that do not need secrets.

Always up to date with Service Hooks

We’ve had a whirlwind tour through the new REST API and updated authentication system. So far, so good. An additional, important element in your journey towards a customised ALM platform is the so-called Service Hooks feature. Service Hooks enable notification of external services and applications for specific events within a team project. This allows the establishment of a workflow between Visual Studio Online or Team Foundation Server 2015 and external services, and completes the process started with the new API and Authentication. Here is the Service Hooks mechanism in an overview:

2191-ServiceHooks-5690ba7f-6724-47d2-949

Figure 5: Event model for Service Hooks

The Service Hook publisher defines events, which can be consumed by a Service Hook Consumer via subscriptions. These consumers, in turn, trigger actions. Nowadays there are more than a dozen services that can be integrated using Service Hooks, including Jenkins, HipChat, Slack, UserVoice, Trello, and so on. Let’s see an example of that.

Integrating Trello and TFS 2015 CTP

I’ll demonstrate the Service Hook principle with an example that is already working very smoothly – the Trello online collaboration tool. The Service Hook for Trello can be configured so that events from the Build, Version control, Team room or Work items units trigger an action in Trello. In the following example a new card is created in Trello for each newly created work item in a certain TFS team project.

2191-On-Prem_ServiceHooks-39626193-e6ad-

Figure 6: Service Hooks for TFS 2015 CTP

I was positively surprised by how easy the integration with Trello was – I was able to integrate the tool into TFS 2015 in seconds. I merely created a new Service Hook in the TFS administrator user interface in the Service Hooks section, and select Trello as the service. I needed to grant TFS access to Trello data and boards to be able to create new cards in Trello, as you can see below.

2191-ServiceHookTrello-997516b2-1efb-476

Figure 7: Creating a Service Hook for Trello

It looks like, at time of writing, Trello engineers are not quite prepared for the fact that the TFS on-prem can now also create a service hook because the dialogue continues to read “Let Visual Studio Online use your account?”. However, this is no reason for concern. After completing configuration of the Service Hook, newly created product backlog items are seamlessly transferred from the TFS 2015 to a Trello board and shown there as new cards.

2191-TrelloBoard-952509c1-6fc7-471a-a76c

Figure 8: Trello card created by TFS

As I mentioned briefly earlier on, although Service Hooks have been in use in Visual Studio Online for a while, they will only be introduced to on-premise environments with the most recent TFS version. An initial test of the TFS 2015 CTP version demonstrated that the number of available Service Hooks is slightly lower than in Visual Studio Online. Unfortunately, I am unable to say whether there will be more selection at the launch of the TFS 2015.

TFS Evolution – an open source demo project

Finally, to give you a sense of how this all fits together, I’ll introduce an open source project which I’ve worked on, together with my colleagues Marco Richardson and Ulli Stirnweiß (ullibility.net). The TFS Evolution Windows app uses the three improvements and integration options we’ve discussed, (Rest API, OAuth 2.0 and Service Hooks) to visualise data from Visual Studio Online.

TFS Evolution shows Team Foundation Server data in a previously unseen way. The app was designed for scrum teams who demand a concise visualisation of the most important data as part of daily meetings, so TFS Evolution provides a detailed overview of the team performance and current tasks.

The app is split into two sections: a dashboard, providing teams with a burndown chart, sprint progress chart and an event ticker as well as a touch-optimised task board:

2191-vsoimage9.png

Figure 9: “Team Performance & Health State” dashboard

And a task board, allowing for fast and simple adaptations to the information with touch gestures.

2191-vsoimage10.png

Figure 10: Task board

TFS Evolution is open source and available to download from GitHub, and we invite you to modify and enhance it, or even just take a look to get some inspiration for your own projects!

Adding Value and Fostering Creativity

Thanks to an integration of external services, applications and tools, the Visual Studio ALM platform can now be customised to suit any company’s processes. System restrictions are either removed or easy to work around, and manual tasks can be automated more than ever before. The added value to users now not only lies in the use of individual tools, but on the information these tools share amongst each other.

The REST API and Service Hooks also mean that the platform is becoming increasingly accessible for other technologies. Coupled with the existing Git version control and support for cross-platform development tools (Cordova and Xamarin), Microsoft’s ALM platform has the potential to be the product development foundation for any company.

Finally, the TFS Evolution showcase I mentioned should help you see what’s possible on this new platform, and let you think outside the usual process constraints you might traditionally associate with a Microsoft tool. These new integration options enable countless possibilities, and will hopefully lead to some more streamlined ways to support everything from daily stand-ups to actual coding. Be creative, and craft an ALM platform to suit your process.