"Beyond sensible"
 |
A discussion of |
 |
James Moore, |
David Atkinson, |
Testing at Red Gate
Question: So, what's the role of testers?
JM: I think, when you look at the really big software companies – Microsoft, IBM, Oracle – they all have very large test departments, with very stringent test aspects built into their software development life cycle. They are basically there to ensure the quality of the final product.
DA: Microsoft has recently changed its test strategy and, I think, has moved in favor of dev/test pairings, so their testers have greater technical ability and have to work quite closely with the developers. That's similar to what we do here.
Question: I thought "Testing" simply meant "Trying to break stuff."
DA: Yes, but to know how to break it, you need quite an in-depth knowledge of how to build the data that would break it. So you have to understand SQL Server, and how databases work. And that's a lower level than most testers will work at. Most testers will run at a higher, "black-box" level. They're system testing from the outside. They'll get an application, put some values in, click something, and there will be an expected outcome which they will see on a screen. They don't necessarily have to look at the database, they don't have to construct fairly obscure datasets.
JM: In typical software companies, after unit testing there's usually nothing at all until system testing begins. At Red Gate, we have the opportunity to test earlier, because all our products have an API that allows us to access and test the functionality without the UI being in place...and the earlier you find issues, the better the payback. It's well known that, if you can fix an issue at design level or requirements level, it will cost a lot less to fix. It will save you a lot more money than if you find something later on.
DA: We're filling a gap that a lot of companies don't look at at all. We can test earlier, because testers here know something about the internals of the product. The box is your product and the opposite of black-box is white-box testing, sometimes called glass-box – i.e. you know what's in it before you test it.
Question: Are testers essentially proxies for all the potential customers of the product?
JM: That's the idea. You try to replicate the data that each potential customer might have...
DA: ...and also to go beyond that. Customers are unlikely to find something even more obscure than you've already found.
JM: And to cover every crazy thing that customers might do! In database design you get a lot of people doing it, and amongst them are a lot of people who perhaps shouldn't be doing it.
DA: Testers have always tried to find the extreme cases. For example, say you get a field that takes a number. A tester will create an input case by just keeping his finger pressed on nine for half an hour and then using this input within tests. It might break. It might not. If it doesn't break, you can feel more confident that no user will produce an input that your application can't take. So you go beyond what's sensible. If you can support beyond what's sensible, then the chances are that customers are going to have a better experience and not run into problems.
Relentless testing
Question: David, how many bugs have you found in, say, SQL Compare?
DA: On average, there's one bug per 10 lines of code. If not more.
JM: SQL Compare has about 140,000 lines of code.
Question: So that's more than 10,000 bugs over the lifetime of a product?
DA: Yes, but this is quite a mature product. [First released seven years ago.]
JM: If I was to sit down and write 100,000 lines of code, you could easily expect there to be 10,000 bugs or more – unless I decided to write very defensive code, but then it would probably take 3 or 4 times as long to develop the product. I mean, there's a real balance, and I think these tester guys keep us in check. They try and make sure we're not too sloppy. It's a difficult balance to achieve, but I think Red Gate is getting there, as we get more and more experienced people.
Question: Do you get a buzz out of finding problems in the product?
DA: Of course. I mean, with a product where you never found a bug, you wouldn't feel fulfilled as a tester, so you'd feel as if your job was pointless, superfluous. So you're justifying your existence to some extent, by finding these problems.
JM: We classify all our bugs in terms of priority and severity.
Question: So, do you find that you get a bigger buzz out of finding a "high priority – high severity" bug than a "low – low" one?
DA: Yeah, I suppose so. If it's something like a missing full stop it's not quite so impressive...
JM: ...as a full-scale crash.
DA: Yeah, but if the product falls over and deletes your hard disk at the same time, then you know that you've saved thousands of pounds by finding this one issue. Whereas finding a full stop missing just isn't worth as much.
JM: But, as developers we prefer it when the testers find low – low bugs...
DA: No you don't! Well, only because you can dismiss them as not worth looking at.
JM: [laughs] Well, there are some bugs you just know you're never ever going to fix, and some we know we're going to postpone fixing…but anything high priority and high severity (HP/HS) will get looked at and fixed, if at all possible. Again, it's a balance. When you're approaching a product release, you've got to realize that every bug fix has the probability of introducing new bugs. And you have diminishing time to actually check whether it's caused regression in the rest of the product. So you've got to be very careful when you've got, say, two weeks to go before a product release.
DA: It's quite funny. As you get closer to deadlines, testers feel a lot under pressure because everyone says "Come on, we need to release it!" and the marketing people are saying they've got all these email campaigns scheduled. So we're pushing the other way to a certain extent. And I suppose the idea is that you meet somewhere in the middle and that's optimal. Because, if you didn't have testers, then the developers would say "Oh, it's good enough!" and the marketing people are shouting to release. So you need someone to say "Come on, just give us an extra day or so, because there are tests we'd still like to run."
Question: So it's ultimately a consensus decision to release?
JM: No! It's all about bullying!! [laughs]
DA: No! No one's happy. A lot of developers or marketing people wanted it a few days ago. Testers want it released next week. So nobody's happy!!
Question: Maybe the customers are happy?
JM: If it works! If it works…
DA: Yeah, just so long as it works.
The developer-tester relationship
Question: So what's the developer-tester relationship really like?
DA: Personally, I've found that if there is a problem with the relationship, it's usually a problem with the developer [laughs].
JM: [laughing] I think it's very important to keep a good sense of humor between the tester and the developer. I mean, if you're a developer and you've been working on a project for a long time, and you've been working really hard on it, and someone comes along and...
 |
DA: And tells you it's rubbish... JM: ...and they really start whacking it, finding real problems with it... Which, of course, they're paid to do... but they're really sort of... it can become quite personal. DA: I'm sure it sometimes feels like a personal attack. I mean you're proud of your work and somebody's basically saying "Here's a list of problems with it." Testers don't list the good things about a product. That's assumed, you see. If testers actually came and said "Here's a list of a hundred things that are great about the product, and here's a smaller list of problems with it," I suppose that might be better, but we don't do that! |
 |
JM: As a developer, I think we all appreciate that testing and bug fixing have to be done. But for the last month or two before this release [SQL Bundle 5], all I've done is fix bugs. The stuff I love to do is to create really cool new bits of software. And, generally, you find that that turns out to be about half the project. You spend an afternoon really racking your brains trying to get something to work. Then David tests it, finds a bug and sends it to me to fix. So I'll fix it, generally break something else, and give it back to him. Then he retests it and comes back with another half a dozen issues!! And you know things like that can get really...
Question: It seems that the egos that get dented are more often the developers' egos rather than the testers' egos? Is there no payback?
DA: Well, there's a lot of payback, because it's usually the testers who get it in the neck at first if it goes into the field and there's a problem. Because the Test Department are going to be the last people to have looked at the product before it's gone out there.
JM: I guess the testers do a really important job. Luckily, at Red Gate, we have a pretty good relationship between developers and testers.
DA: Developers sometimes get defensive. But what you have to remember is that we're all working to the same goal: we all want to release high-quality products on time, which is the motto we work to. On the way, if it means your ego gets dented because somebody says your product needs a bit of work, then that's good. It's a good thing, not a bad thing because, provided it's done diplomatically, you can resolve the issue and collectively get closer to that ultimate goal.