Basic Defensive Database Programming Techniques

Comments 0

Share to social media

DevOps, Continuous Delivery & Database Lifecycle Management
Continuous Integration

The goal of defensive database programming is to produce resilient database code; in other words code that does not contain bugs and is not susceptible to being broken by unexpected use cases, small modifications to the underlying database schema, changes in SQL Server settings, and so on.

If you fail to program defensively, then code that runs as expected on a given standalone server, with a specific configuration, may run very differently in a different environment, under different SQL Server settings, against different data, or under conditions of concurrent access. When this happens, you will be susceptible to erratic behavior in your applications, performance problems, data integrity issues and unhappy users.

The process of reducing the number of vulnerabilities in your code, and so increasing its resilience, is one of constantly questioning the assumptions on which your implementation depends, ensuring they are always enforced if they are valid, and removing them if not. It is a process of constantly testing your code, breaking it, and then refining it based on what you have learned.

The best way to get a feel for this process, and for how to expose vulnerabilities in your code and fix them using defensive programming techniques, is to take a look at a few common areas where I see that code is routinely broken by unintended use cases or erroneous assumptions:

  • Unreliable search patterns
  • Reliance on specific SQL Server environment settings
  • Mistakes and ambiguity during data modifications

In each case, we’ll identify the assumptions that lead to code vulnerability, and show how to fix them. All the examples in this article are as simple as possible in that there is no concurrency and the underlying database schema is fixed.

My forthcoming book on this subject introduces many of the additional dangers that can arise when exposing the code to changes in the database schema, running it under high concurrency, and so on.

Reducing Code Vulnerability

There are four key elements to defensive database programming that, when applied, will allow you to eliminate bugs and make your code less vulnerable to be being subsequently broken by cases of unintended use:

  1. Define and understand your assumptions
  2. Test as many use cases as possible
  3. Lay out your code in short, fully testable, and fully tested modules
  4. Reuse your code whenever feasible, so that the code to solve a given problem is implemented in one place only

While I will occasionally make brief mention of the sort of checks and tests that ought to be included in your unit tests (steps 2 and 3), this article is focused on defensive programming, and so on the rigorous application of the first two principles.

Define your Assumptions

One of the most damaging mistakes made during the development of SQL code, and any other code, is a failure to explicitly define the assumptions that have been made regarding how the code should operate, and how it should respond to various inputs. Specifically, we must:

  • Explicitly list the assumptions that have been made
  • Ensure that the these assumptions always hold
  • Systematically remove assumptions that are not essential, or are incorrect

When identifying these assumptions, there can be one of three possible outcomes. Firstly, if an assumption is deemed essential, it must be documented, and then tested rigorously to ensure it always holds; I prefer to use unit tests to document such assumptions. Failure to do so will mean that when the code makes it into production it will inevitably be broken as a result of usage that conflicts with the assumption.

Secondly, if the assumption is deemed non-essential, it should, if possible, be removed. Finally, in the worst case, the code may contain assumptions that are simply wrong, and can threaten the integrity of any data that the code modifies. Such assumptions must be eliminated from the code.

Rigorous Testing

As we develop code, we must use all our imagination to come up with cases of unintended use, trying to break our modules. We should incorporate these cases into our testing suites.

As we test, we will find out how different changes affect code execution and learn how to develop code that does not break when “something”, for example a language setting or the value of ROWCOUNT, changes

Having identified a setting that breaks one of our code modules, we should fix it and then identify and fix all other similar problems in our code. We should not stop at that. The defensive programmer must investigate all other database settings that may affect the way the code runs and then review and amend the code again and again, fixing potential problems before they occur. This process usually takes a lot of iterations, but every time we end up with better, more robust code and we will save a lot of potential wasted time in troubleshooting problems, as well as expensive retesting and redeployment, when the code is deployed to production.

Throughout the rest of this article, we’ll discuss how this basic defensive coding philosophy is applied in practice, by way of some simple practical examples.

Defending Against Cases of Unintended Use

All-too-often, we consider our code to be finished as soon as it passes a few simple tests. We do not take enough time to identify and test all possible, reasonable use cases for our code. When the inevitable happens, and our code is used in a way we failed to consider, it does not work as expected.

To demonstrate these points, we’ll consider an example that shows how (and how not) to use string patterns in searching. We’ll analyze a seemingly working stored procedure that searches a Messages table, construct cases of unintended use, and identify an implicit assumption on which the implementation of this procedure relies. We will then need to decide whether to eliminate the assumption or to guarantee that it always holds. Either way, we will end up with a more robust procedure.

Listing 1 contains the code needed to create a sample Messages table, which holds the subject and body of various text messages, and load it with two sample messages. It then creates the stored procedure, SelectMessagesBySubjectBeginning, which will search the messages using a search pattern based on the LIKE keyword. The stored procedure takes one parameter, SubjectBeginning, and is supposed to return every message whose subject starts with the specified text.

Listing 1: Creating and populating the Messages table along with the stored procedure to search the messages

Some preliminary testing against this small set of test data, as shown in Listing 2, does not reveal any problems.

Listing 2: A few simple tests against the provided test data demonstrate that results match expectations

Handling Special Characters in Searching

In defensive database programming, it is essential to construct cases of unintended use with which to break our code. The test data in Listing 1 and the stored procedure calls in Listing 2 demonstrate the cases of intended use, and clearly the procedure works, when it is used as intended.

However, have we considered all the possible cases? Will the procedure continue to work as expected in cases of unintended use? Can we find any hidden bugs in this procedure? In fact, it is embarrassingly easy to break this stored procedure, simply by adding a few “off topic” messages to our table, as shown in Listing 3.

Listing 3: Our procedure fails to return “off topic” messages

Our procedure fails to return the expected messages. In fact, by loading one more message, as shown in Listing 4, we can demonstrate that this procedure can also return incorrect data.

Listing 4: Our procedure returns the wrong messages when the search pattern contains [OT]

When using the LIKE keyword, square brackets (“[” and “]“), are treated as wildcard characters, denoting a single character within a given range or set. As a result, while the search was intended to be one for off-topic posts, it in fact searched for “any messages whose subject starts with O or T“. Therefore Listing 3 returns no rows, since no such messages existed at that point, whereas Listing 4 “unexpectedly” returns the message starting with “O”, rather than the off-topic messages.

In a similar vein, we can also prove that the procedure fails for messages with the % sign in subject lines, as shown in Listing 5.

Listing 5: Our stored procedure returns the wrong messages, along with the correct ones, if the pattern contains %

The problem is basically the same: the % sign is a wildcard character denoting “any string of zero or more characters”. Therefore, the search returns the “500 new customers…” row in addition to the desired “50% bugs fixed…” row.

Our testing has revealed an implicit assumption that underpins the implementation of the SelectMessagesBySubjectBeginning stored procedure: the author of this stored procedure did not anticipate or expect that message subject lines could contain special characters, such as square brackets and percent signs. As a result, the search only works if the specified SubjectBeginning does not contain special characters.

Having identified this assumption, we have a choice: we can either change our stored procedure so that it does not rely on this assumption, or we can enforce it.

Enforcing or Eliminating the Special Characters Assumption

Our first option is to fix our data by enforcing the assumption that messages will not contain special characters in their subject line. We can delete all the rows with special characters in their subject line, and then add a CHECK constraint that forbids their future use, as shown in Listing 6. The patterns used in the DELETE command and in the CHECK constraint are advanced and need some explanation. The first pattern, %[[]%, means the following:

  • Both percent signs denote “any string of zero or more characters”
  • [[] in this case denotes “opening square bracket, [“
  • The whole pattern means  “any string of zero or more characters, followed by an opening square bracket, followed by another string of zero or more characters”, which is equivalent to “any string containing at least one opening square bracket”

Similarly, the second pattern, %[%]%, means “any string containing at least one percent sign”.

Listing 6: Enforcing the “no special characters” assumption

Although enforcing the assumption is easy, does it make practical sense? It depends. I would say that under most circumstances special characters in subject lines should be allowed, so let’s consider a second, better option: eliminating the assumption. Note that Listing 6 rolls back the transaction, so that our changes are not persisted in the database.

Listing 7 shows how to alter the stored procedure so that it can handle special characters. To better demonstrate how the procedure escapes special characters, I included some debugging output. Always remember to remove such debugging code before handing over the code for QA and deployment!

Listing 7: Eliminating the “no special characters” assumption

Listing 8 demonstrates that our stored procedure now correctly handles special characters. Of course, in a real world situation, all previous test cases have to be rerun to check that we didn’t break them in the process of fixing the bug.

Listing 8: Our search now correctly handles [ ] and %

Whether we ultimately decide to enforce or eliminate the assumption, we have created a more robust search procedure as a result.

Defending Against Changes in SQL Server Settings

A common mistake made by developers is to develop SQL code on a given SQL Server, with a defined set of properties and settings, and then fail to consider how their code will respond when executed on instances with different settings, or when users change settings at the session level.

Let’s examine a few simple cases of how hidden assumptions with regard to server settings can result in vulnerable code.

How SET ROWCOUNT can break a Trigger

Traditionally, developers have relied on the SET ROWCOUNT command to limit the number of rows returned to a client for a given query, or to limit the number of rows on which a data modification statement (UPDATE, DELETE, MERGE or INSERT) acts. In either case, SET ROWCOUNT works by instructing SQL Server to stop processing after a specified number of rows.

However, use of SET ROWCOUNT can have some unexpected consequences for the unwary developer. Consider a very simple table, Objects, which stores basic size and weight information about objects, as shown in Listing 9.

Listing 9: Creating and populating the Objects table

We are required to start logging all updates of existing rows in this table, so we create a second table, ObjectsChangeLog, in which to record the changes made, and a trigger that will fire whenever data in the Objects table is updated, record details of the changes made, and insert them into ObjectsChangeLog.

Listing 10: Logging updates to the Objects table

Please note that my approach to all example in this book is to keep them as simple as they can be while still providing a realistic demonstration of the point, which here is the effect of SET ROWCOUNT. So, in this case, I have omitted:

  • A “real” key on the ObjectsChangeLog table, enforced by a UNIQUE constraint (ObjectID, ChangedColumnName, ChangedAt), in addition to the surrogate key on ObjectsChangeLogID
  • The equivalent insert and delete triggers to log INSERT and DELETE modifications, as well as UPDATEs

Likewise, there are several ways of logging changes and the one I chose here may not be the best approach; again my goal was to keep the example focused and simple. Listing 11 shows the code that tests how our trigger logs changes against the Objects table.

Listing 11: Testing the trigger

Apparently, our trigger works as expected! However, with a little further testing, we can prove that the trigger will sometimes fail to log UPDATEs made to the Objects table, due to an underlying assumption in the trigger code, of which the developer may not even have been aware!

The ROWCOUNT Assumption

Let’s consider what might happen if, within a given session, a user changed the default value for ROWCOUNT and then updated the Objects table, without resetting ROWCOUNT, as shown in Listing 12.

Listing 12: Breaking the trigger by changing the value of ROWCOUNT

As a result of the change to the ROWCOUNT value, our trigger processes the query that logs changes to the SizeInInches column, returns one row, and then ceases processing. This means that it fails to log the change to WeightInPounds column. Of course, there is no guarantee that the trigger will log the change to the SizeInInches column. On your server, the trigger may log only the change of WeightInPounds but fail to log the change in SizeInInches. Which column will be logged depends on the execution plan chosen by the optimizer, and we cannot assume that the optimizer will always choose one and the same plan for a query.

Although the developer of the trigger may not have realized it, the implied assumption regarding its implementation is that ROWCOUNT is set to its default value. Listing 12 proves that that when this assumption is not true, the trigger will not work as expected.

Enforcing and Eliminating the ROWCOUNT Assumption

Once we understand the problem, we can fix the trigger very easily, by resetting ROWCOUNT to its default value at the very beginning of the body of the trigger, as shown in Listing 13.

Listing 13: Resetting ROWCOUNT at the start of the trigger

We can rerun the test from Listing 12 and this time the trigger will work as required, logging both changes. Note that the scope of our SET ROWCOUNT is the trigger, so our change will not affect the setting valid at the time when the trigger was fired.

SET ROWCOUNT is deprecated in SQL Server 2008…

…and eventually, in some future version, will have no effect on INSERT, UPDATE or DELETE statements. Microsoft advises rewriting any such statements that rely on ROWCOUNT to use TOP instead. As such, this example may be somewhat less relevant for future versions of SQL Server; the trigger might be less vulnerable to being broken, although still not immune. However, at the time of writing, this example is very relevant.

In this case, one simple step both enforces the underlying assumption, by ensuring that it is always valid, and eliminates it, by ensuring that the code continues to work in cases where ROWCOUNT is not at its default value.

Proactively Fixing SET ROWCOUNT Vulnerabilities

We have fixed the ROWCOUNT vulnerability in our trigger, but our job is not done. What about other modules in our system? Might they not have the same vulnerability?

Having learned of the potential side effects of SET ROWCOUNT, we can now analyze all the other modules in our system, determine if they have the same problem, and fix them if they do. For example, our stored procedure SelectMessagesBySubjectBeginning (Listing 1) has the same vulnerability, as demonstrated by the test in Listing 14.

Listing 14: SET ROWCOUNT can break a stored procedure just as easily as it can break a trigger

We can apply the same fix, adding SET ROWCOUNT 0; to the very beginning of this stored procedure. Similarly, we should apply this fix to all other modules that need it.

If your code is supposed to exist for a considerable time, then it makes perfect sense to fix problems proactively. It is usually faster and easier to do so than to wait until the problem occurs, spend considerable time troubleshooting, and then eventually implement the same fix.

How SET LANGUAGE can break a Query

Just as the value of ROWCOUNT can be changed at the session level, so can other settings, such as the default language. Many developers test their code only under the default language setting of their server and do not test how their code will respond if executed on a server with a different language setting, or to a change in the setting at the session level.

This practice is perfectly correct as long as our code always runs under the same settings as those under which we develop and test it. However, if or when the code runs under different settings, this practice will often result in code that is vulnerable to errors, especially when dealing with dates.

Consider the case of a stored procedure that is supposed to retrieve from our ObjectsChangeLog table (Listing 10) a listing of all changes made to the Objects table over a given date range. According to the requirements, only the beginning of the range is required; the end of the range is an optional parameter. If an upper bound for the date range is not provided, we are required to use a date far in the future, December 31st, 2099, as the end of our range.

Listing 15: Creating the SelectObjectsChangeLogForDateRange stored procedure

Note that this stored procedure uses a string literal, 12/31/2099, to denote December 31st, 2099. Although 12/31/2099 does represent December 31st, 2099 in many languages, such as US English, in many other cultures, such as Norwegian, this string does not represent a valid date. This means that the author of this stored procedure has made an implicit assumption: the code will always run under language settings where 12/31/2099 represents December 31st, 2099.

When we convert string literals to DATETIME values, we do not have to make assumptions about language settings. Instead, we can explicitly specify the DATETIME format from which we are converting.

The following scripts demonstrate both the safe way to convert character strings to DATETIME values, and the vulnerability of our stored procedure to changes in language settings. The script shown in Listing1.18 populates the ObjectsChangeLog table and calls the SelectObjectsChangeLogForDateRange stored procedure under two different language settings, US English and Norwegian.

Listing 16: Our stored procedure breaks under Norwegian language settings

Under the Norwegian language settings we receive an error at the point where it attempts to convert 12/31/2099 into a DATETIME string.

Note that we are, in fact, quite fortunate to receive an error message right away. Should we, in some other script or procedure, convert ’10/12/2008′ to DATETIME, SQL Server would silently convert this constant to a wrong value and we’d get incorrect results. Listing 17 shows how our stored procedure can return unexpected results without raising errors; such silent bugs may be very different to troubleshoot.

Listing 17: Our stored procedure call returns different results, depending on language settings

To fix the stored procedure, as shown in Listing 18, we need to explicitly specify the format from which we convert the VARCHAR values provided when the stored procedure is executed.

Listing 18: Fixing the stored procedure

The stored procedure will now run correctly, regardless of the language settings. In this case, we chose to fix the problem by eliminating the assumption. Alternatively, in some cases, we might choose to enforce it by setting the language at the beginning of the stored procedure, just as we did with the ROWCOUNT setting.

Of course, there are situations when our code will always run under one and the same settings, in which case there is no need to do anything. For example, if a module implements business rules specific to the state of Minnesota, it is reasonable to assume that it will always run under the same language settings.

Defensive Data Modification

Data modification is, in general, an area in which I see developers getting into trouble time and again. We’ll start with a case that demonstrates how data can be erroneously updated as a result of a false assumption in the stored procedure that modifies it. It is a simple example, but the underlying problem is a very common one: using search criteria that affect more rows than intended.

We’ll then discuss a second, somewhat more complex case, where an UPDATE can go wrong because it fails to unambiguously identify the row(s) to be modified, perhaps falsely assuming that the underlying data structures will ensure that no such ambiguity exists.

Updating more rows than intended

Listing 19 creates a simple Employee table, and a SetEmployeeManager stored procedure that assigns a manager to a given employee.

Listing 19: The Employee table and SetEmployeeManager stored procedure

Clearly, the person who developed the stored procedure assumed that, at most, one employee may have the provided first and last name. If there happens to be two people in the organization with the same name then this stored procedure will assign them both to the same manager.

Again, having uncovered the assumption, we need to decide whether to enforce it or eliminate it. We could enforce it simply by placing a UNIQUE constraint on the FirstName and LastName columns. However, in this case, it seems much more reasonable to assume that there may well be more than one employee with the same first and last name, and that these namesake employees may report to different managers. Therefore, we need to eliminate the incorrect assumption. There are many ways to do this, the simplest being to ensure that the parameter supplied to the stored procedure, and used in the search criteria, identifies a unique row, as shown in Listing 20.

Listing 20: Using unambiguous search criteria

As long as EmployeeID is the primary key on the dbo.Employee table, this procedure will work correctly.

The Problem of Ambiguous Updates

The results of data modifications may be unpredictable in the hands of the careless programmer. Let’s consider a very common requirement: populating a permanent table from a staging table. First of all, let’s create our permanent table, Codes, and a staging table, CodesStaging, as shown in Listing 21. Note that CodesStaging does not have a primary key. This is very common for staging tables, because data is often loaded into such tables before detecting duplicates and other data integrity violations.

Listing 21: Creating the Codes and CodesStaging tables

Now, let’s populate each table with some sample data, as shown in Listing 22.

Listing 22: Populating the Codes and CodesStaging tables

Now, we’ll examine two different ways of updating data in the permanent table, based on data in the staging table, both of which are subject to ambiguities if care is not taken:

  • Using UPDATE…FROM
  • Updating an inline view

We’ll then discuss strategies for avoiding such ambiguities.

Using UPDATE…FROM

Notice in Listing 22 that the incoming data in our staging table has a duplicate: the code AR occurs twice, with different descriptions. Suppose that we have not detected or resolved this duplicate, and that we are updating our Codes table from the staging table.

Listing 23: An ambiguous UPDATE…FROM, when loading data from a staging table (CodesStaging) into a target table (Codes)

Although two descriptions were provided for the AR code, the UPDATE…FROM command did not raise an error; it just silently updated the corresponding row in Codes table with one of the two provided values. In this case, the ‘old description for Arkansas’ has been overwritten with the ‘description for Argentina’.

Updating Inline Views

When we update inline views, we may encounter exactly the same problem. First, repopulate each of the tables with the original data, using the code from Listing 22. Next, create an inline view, and then use it to implement exactly the same functionality as the previous UPDATE…FROM commands, as shown in Listing 24.

Listing 24: An ambiguous update of an inline view

Note that neither in this example nor the previous UPDATE…FROM example, can we predict which of these two values will end up in the target table – that, as usual, depends on the execution plan and as such is completely unpredictable. It is by pure chance that, in my examples, Argentina was chosen over Arkansas in both cases. I was able to get different results, with the description of Arkansas rather than Argentina inserted into Codes, just by changing the order in which the rows are inserted into CodesStaging. However, again, there is no guarantee that you will get the same results on your box. Also, bear in mind that if we ever did add an index to the staging table, this would almost certainly affect the result as well.

How to Avoid Ambiguous Updates

In both previous examples, the developer has written the UPDATE command apparently under the assumption that there can be no duplicate data in the CodesStaging – which cannot be guaranteed in the absence of a UNIQUE or PRIMARY KEY constraint on the Code column – or that any duplicate data should have been removed before updating the permanent table.

Generally, performing this sort of ambiguous update is unacceptable. In some cases, we might want to refine the query to make sure it never yields ambiguous results. However, typically we want either to raise an error when an ambiguity is detected, or to update only what is unambiguous.

In SQL Server 2008, we can circumvent such problems with UPDATE…FROM or CTE-based updates, by use of the MERGE command. However, prior SQL Server 2008, we have to detect these ambiguities.

Using MERGE to Detect Ambiguity (SQL Server 2008 only)

If you are working with SQL Server 2008, then easily the best option is to use the MERGE command. In Listing 25, we use the MERGE command to update our primary table from our staging table and immediately encounter the expected error.

Listing 25: MERGE detects an ambiguity in incoming data

An ANSI-standard Method

Pre-SQL Server 2008, we are forced to seek alternative ways to raise an error whenever there is an ambiguity. The code in Listing 26 is ANSI-standard SQL and accomplishes that goal.

Listing 26: An ANSI Standard UPDATE command, which raises an error when there is an ambiguity

Note that in order to update just one column we had to use two almost identical subqueries in this command. This is definitely not a good practice. Should we need to update ten columns, we would have to repeat almost the same code eleven times! If, at some later time, we need to modify the subquery, we will have to make one and the same change in eleven places, which is very prone to errors.

Defensive Inline View Updates

Fortunately, there are several ways to improve the robustness of inline view updates, as well as UPDATE…FROM updates (covered in the next section), which work with SQL 2005.

In the previous two examples, an error was raised when ambiguity was detected. This is usually preferable but, if your business rules allow you to ignore ambiguities, and only update that which is unambiguous, then the solution shown in Listing 27 will work.

Listing 27: Using a subquery to ignore ambiguities when updating an inline view

This time, only the description of Indiana is updated. In a similar fashion, we could filter out (i.e. ignore) ambiguities with the help of an analytical function, as shown in Listing 28.

Listing 28: Using PARTITION BY to ignore ambiguities when updating an inline view

In some cases, the approach of only performing unambiguous updates, and silently ignoring ambiguous ones, is unacceptable. In the absence of built in methods, we can use tricky workarounds to reuse the code as much as possible and still raise an error if there is an ambiguity. Consider the example shown in Listing 29, in which a divide by zero occurs if there is an ambiguity.

Listing 29: An UPDATE command using an inline view and raising a divide by zero error when there is an ambiguity

Of course, the error message raised by this code (divide by zero) is misleading, so we should only use this approach when any of the previous alternatives are not viable.

Defensive UPDATE…FROM

Some of the approaches just outlined for improving the robustness of inline view updates, apply equally as well to improving the UPDATE…FROM command.

For example, we can use a sub-query to ignore ambiguities, as shown in Listing 30.

Listing 30: Using a subquery to ignore ambiguities when using UPDATE…FROM

Likewise, we can use an analytical function for detecting and ignoring ambiguities, as shown in Listing 31.

Listing 31: Using an analytical function to detect and ignore ambiguities when using UPDATE…FROM

Summary

The goal of this article was to introduce, by way of some simple examples, some of the basic ideas that underpin defensive database programming. It is vital that you understand and document the assumptions that underpin your implementation, test them to ensure their validity, and eliminate them if they are not. It is also vital that you consider as many use cases as possible for your code, and ensure it behaves consistently in each case. Where inconsistencies or incorrect behavior are found, the defensive programmer will not only fix the offending module, but also test all other modules that might suffer from a similar problem and proactively safeguard against it.

Along the way, I hope you’ve learned the following specific lessons in defensive programming:

  • How to use complex patterns to improve the robustness of LIKE searches
  • How to avoid potential difficulties with SET ROWCOUNT
  • The importance of safe date formats and of explicitly specifying the required format when converting dates
  • How to avoid dangerous ambiguity when performing updates by, for example:
    • Using MERGE, in SQL Server 2008
    • Using subqueries, pre-SQL Server 2008
  • How to use subqueries or the COUNT(*) OVER analytic function to improve the robustness of modifications when using UPDATE…FROM, or updating inline views, so that ambiguous updates are ignored.

DevOps, Continuous Delivery & Database Lifecycle Management
Go to the Simple Talk library to find more articles, or visit www.red-gate.com/solutions for more information on the benefits of extending DevOps practices to SQL Server databases.

Load comments

About the author

Alex Kuznetsov

See Profile

Alex Kuznetsov has been working with object oriented languages and databases for more than a decade. He has worked with Sybase, SQL Server, Oracle and DB2. Alex has published multiple articles on simple-talk.com and sqlblog.com and wrote a book entitled Defensive Database Programming with SQL Server. Currently he works in an agile team in Chicago. In his leisure time, Alex prepares for and runs ultramarathons.

Alex Kuznetsov's contributions