Writing a System Test Plan

Using a test plan to ensure your software really works.

"Extraordinary quality doesn't make good short-term economic sense."
- Peopleware, Tom Demarco

Imagine for a moment you’ve just finished a web application for a client. Swimming around within that application is 100 bugs. What proportion of those bugs would you say is acceptable for the client to find (e.g. 50, 30, 10)? The point is, those bugs are going to be found by someone, and its either going to be you or the client. You can think of it as a sliding scale, the more bugs the client finds, the more your creditability suffers and the more damage you do to the business relationship.

In the martial arts there’s a saying; “expect the best, but prepare for the worst”. The US Navy Seals have a slightly more gun-ho version which goes; “The more you sweat in training, the less you bleed in battle.” Notice how it says the ‘less’ you bleed in battle, implying that if you enter into combat, you are going to get bloody. Software development is like this too, if you’re going to create software, you are going to get bugs.

As a rough guide, you want to try catch at least 80% of all bugs. Ideally, you would go for 100%, a worthwhile but lofty goal. Your best bet for catching as many bugs as possible is to create a system test plan. Creating a decent system test plan is easier said then done. A good place to start is your functional specification (assuming you have one). In effect, your specification gets converted into the system test plan. What you are doing is checking to see if the things you said you were going to do have actually been done.

A test plan is nothing new, in fact, the structure I use was developed by Microsoft years ago. It’s a simple format, a test case has a title, some steps, and an expected result.



I write my test plans with the intent of having an independent QA tester run through it, so no prior knowledge of the system should be assumed. You want someone who has never used the system before because they will take a fresh approach. You’re counting on them to use it in a manner which was never intended (e.g. on the ‘My Profile’ page, they type in a really long name for the ‘Company’ field and crash the system because the database only has room for a 32 character string).



You should keep your test cases short, to the point, and self contained (i.e. they generally don’t link to other test cases). If you have a test case that goes over a page, consider breaking it into smaller units. The document itself should be self contained as well, that is; it shouldn’t require the QA tester to get up and ask programmers questions like “what’s the login password?”. Instruct your QA tester to log any bugs they find in your bug tracking system and also to write a proper description rather then just saying “Test Case 16 failed.” You may want to read my article on Logging Bugs Like a Pro for some suggestions on good bug logging practices.

Article reviewed by Alex Barclay.

Logging Bugs Like a Pro

Tips and hints for reporting bugs more effectively.

This article serves as a brief guide on best practices when it comes to reporting bugs. Encouraging QA testers, clients, and other project stake-holders to follow these habits makes life easier for the development team when it comes to tracking down and fixing a bug. The end result is less time wasted in going back-and-forth finding out the details of a bug. The act of submitting a bug in a formalised way is very important – having a record of the bug means it wont get lost or go away. The other benefit is someone has clearly taken responsibility for dealing with the problem (i.e. a bug should be assigned to a person).



When submitting a bug, it’s a good idea to give it a short, descriptive title. It may seem like a chore, but it assists developers to quickly identify what a bug is when looking at a list of bug titles. It also helps in future when trying to recall what an old bug was about.

To create a good title, all you need to do is think to yourself “what exactly is the problem?” The first thing that pops into your mind is usually a good title. Examples of good bug titles would be: ‘my login details aren’t being saved’, ‘the photo gallery page crashes in IE’, or ‘tax totals on invoice report are not accurate’. Some examples of unhelpful titles would be: ‘error on page’, ‘the system is busted', or ‘the system doesn’t work when I use it’. A good bug title may also include what screen the error occurs on and describe only one problem at a time (e.g. “sign-up page crashes when I enter a long username”).

Probably the most important aspect of a helpful bug report is to give the steps required to reproduce the problem. All you need to do is give a step-by-step account of what you were doing when the bug appeared.

For example;

* Login as a standard user
* Go to ‘My Profile’
* In the ‘Surname’ field, enter O’Brien
* Click the ‘Save’ button.

You’ll see the page has a script error and doesn’t save the surname.

Example of an unhelpful bug report;

There’s something wrong with the profile page. It crashes, my computer reboots, and the cat goes and scratches the couch.

A few simple habits can enhance a bug submission significantly. For instance; using asterisks as bullets, giving just enough detail to be able to recreate the bug (one line is unlikely to help, a novella adds little benefit), and, at the end of the steps, say what went wrong (e.g. “the page has an error”, “data goes missing”).

When logging a bug, a priority rating is often assigned to it. Most bug tracking systems have choices like low, normal, or high. A low priority bug would be something which just looks bad (e.g. a spelling mistake, misaligned controls, etc). A high priority bug would be a page crash, data corruption, or vanishing data. All other bugs would fall into the category of normal. An accurate priority rating is worthwhile since it allows more pressing bugs to be fielded first (e.g. it would be more important to fix a broken login page before a UI issue).



Occasionally, a situation may occur where it’s hard to tell if something really is a bug, it may instead best be categorised as a ‘feature addition’. As a general rule of thumb, its a bug if; a script error occurs, data goes missing or is corrupted, the system behaves in a manner in which it was not intended too (i.e. not to spec).

How the Web Was Won

Perspectives on how business has shaped the Internet.

Before you stands a frontier of possibilities, a land which at first seems desolate. Predictably, a tumble-weed rolls by, but in the distance you spy something looming on the horizon. Upon closer inspection it dawns upon you that what you’re gazing at is in fact ‘Big Business’. What had appeared to be ‘virgin soil’ at first glance, is in fact a well established arena – is it too late? Is there no room on the web for new comers?



There was a time when the web was a novelty, but these days, no one’s impressed if you say “I got the Internet at home”. As many people know, the Internet and World Wide Web were originally conceptualised and developed for military and educational applications. The war-mongers thought to themselves “great guns! A communication system that wont go down when the nukes hit!” whilst the university boffins exclaimed “neat, now I can share my research paper on the mating habits of cockroaches with professor Irvine at Oxford!”.

If information sharing spawned the web, how is it that business is now such a dominant aspect of cyberspace? Obviously, at some point, an entrepreneurial-type thought to themselves “hang-on, what if we used this ‘Internet’ thing to sell stuff?” Now, this may have seemed like a fantastic idea on paper, but at the time this idea was conceived, the web was hardly in state to support such rich functionality (not easily at least).

So why is it that online shopping has become so pervasive? If you take a look at juggernauts like Amazon.com and eBay, it would seem like their triumph was almost pre-destined. But lets not forget follies such as Egghead.com and GovWorks (of ‘Startup.com’ movie fame). And you wouldn’t know it by looking, but there was a time when Yahoo.com was in serious trouble too. For every success, countless others must have fallen by the wayside.

The dot.com bust will certainly go down in web annals as a dark time. If you’re not familiar with why it happened, this is an example of the sort of thing which went on; web company A would buy $10m of advertising from web company B, web company B in return would purchase $10m worth of advertising from web company A. This would make it look as though each of the companies was massively profitable – but like all shoddily constructed pyramids, it all came crushing down in the end.

If the road was fraught with so much peril, then what made companies invest so much capital in ecommerce? There are a number of factors which make the web such a pliable tool for selling goods and services online. For one, an online shop is always open, a world of people can buy from you (literally), it doesn’t have rude staff, it’s cheaper then an actual shop-front (and the ‘rent’ is next to nothing), it can potentially deal with thousands of customers at once, and the list goes on.

So how has business shaped the web? Would it be too extreme to say that business is the web - perhaps, but it sounds deep upon first reading. It could be argued that commercial use of the Internet has brought it to where it is today because it facilitated advances in technological infrastructure (eg. up-take of broadband, establishment of high-speed international fibre-optic backbones, etc), but it’s not really necessary to get quite so ‘fancy’. At the end of the day, people noticed that there was a need that had to be filled, they also noticed a tool that could help satisfy that need. At this very moment, chances are you have a need which is being fulfilled by the Web.