Set things up to test, test, test

I'm sure that we've all experienced it. We go to a website that we often use, and find out that the website is unavailable, or something on the website isn't working properly. Sometimes these are websites that we pay to use, but other times they are websites that we can use for free. Some argue that free websites shouldn't be held to the same standard as other websites - using the "you get what you pay for" philosophy - but even a free website is trying to get someone to pay for it eventually, and Google isn't going to be inclined to buy your company if your website goes down all the time.

In the context of a post that describes various levels of testing, Rob Diana mentions some problems that he was having with Wordpress, and began to wonder how this could happen.

If you Google for “WordPress 2.8 editor problem” you will see a support issue with several people wondering what is wrong. As far as I can remember, WordPress 2.8 was developed fairly quickly. I remember upgrading to 2.7.1 not too long ago. The issue with the editor seems a little too common to be missed during testing, but when you are using FireFox and some plugins, anything is possible. But the real question is how and why did this happen?...

For a problem like the post editor not working properly, their integration tests failed to find the problem. Once they diagnose the problem, they need to document the issue. By having a suite of automated tests, you only need to add another test for the current defect. In the case of the editor, it could be that there is some plugin configuration for Firefox or even WordPress itself that causes the WYSIWYG editor to misbehave.

I'd like to take a step back and reinforce something that my company's testers often impress on me - for them to be able to test a requirement, a requirement has to be testable. Even in the marketing requirements stage (the stage in which I am often involved), you need to write requirements that are SMART - or Specific, Measurable, Attainable, Realistic, and Timely. One could even argue that you need a specific requirement that states that your problem will work with Firefox 3 when using the SuperDuperDooHickey extension. Yes, those types of requirements could lead to a ton of requirements and a ton of testing, but in some cases this may be warranted.