In a perfect world, the requirements would always be clearly defined prior to any software quality testing or software quality assurance-related activity. But in reality software quality assurance analysts and software quality testers are often given minimal direction by project management, software developers and business analysts. They are also allotted little time in which to complete a software project. The reason for this has partly to do with the organizational structure of many software technology companies.

Software quality assurance and software quality testing are two distinctly separate functions that should be able to work together at some level, but should never be rolled into a single entity. At some companies, software QA and testing are both couched under the same name or same department; often lumped together as QA.

Some organizations may not have software quality assurance analysts at all, and testers have the unofficial duty of outlining process methodologies for the Software Development Life Cycle (SDLC), project management workflows and/or software templates. The tester’s resources are stretched, because he is expected to conduct testing and do the work of a software quality assurance analyst.

In software development , the goal is to support the needs of the business, customer and software systems. Surprisingly, project managers and business analysts often don’t understand this goal and are focused on the overall scope of the project, its timeframe and cost. Sure, those factors are critical indications of projects’ success, but without quality the projects fail whatever their scope, length of development or cost.

In my view, software quality analysts and software quality testers should be free from the reins of any other department, thus forming their own distinct groups. The term, silo effect, usually has a negative connotation when describing business practices, but software quality assurance and software quality testing are two areas where the silo may actually be the most effective way to work. In fact, IT quality is at its best when the individuals responsible for it are removed from the influence of other departments.

Being separated from other departments eliminates the risk of having project managers or business analysts changing the software testing process and department to suit their cost, deadline and other requirements. Separation also lessens the software testers’ chance of conducting testing that is biased toward any one department, and it reduces the number of distractions that could hinder communication between the software quality assurance analysts and software testers.

In order to create outstanding products, these two entities must be able to work together while still maintaining a certain amount of exclusivity. Software quality analysts must concentrate on delivering consistent quality across the company, managing procedures, process, standards and policies; whereas software quality testers are responsible for ensuring the overall quality of the software products in terms of their functionality, security, infrastructure and performance.

Creating a governance group that monitors how software quality analysts manage and performing enterprise-wide quality assurance activities can help ensure that software quality activities are compliant with internal regulations. If compliance issues occur during the start of the process, they will only snowball as the software development lifecycle continues. Governance ensures that QA activities remain fluid throughout the SDLC from department to department. Software quality can be considered successful if software QA analysis and testing are completed without any bias infiltrating the process.

The following points depict how software QA and testing activities should be organized:

Start with process.

• Create an SDLC that the whole company is expected to follow.
• Decide how the software quality assurance and software quality testing activities will be managed.

Group acceptance and agreement

• Every key member of the SDLC should agree, must participate, and needs to acknowledge the process that was set in the above step.

Formalize documentation

• Dedicate a specific location where all documentation will be stored.
• Communicate this location and the process of creating documentation to all who will need to access/perform it.

Establish metrics

• Software quality assurance analysts should be in charge of creating audits and reports.
• Software quality analysts should be responsible only for testing reports and projections.

Review and re-establish process

• Review all software quality assurance and testing activities and put together a list of “lessons learned” to be reviewed against the SDLC.

Following these five steps keeps the responsibilities of software QA analysts and testers linear, formalized and easily managed. Remember also that although silos may help software QA and testers stay productive, the silo effect should be kept to a minimum on the enterprise-wide scale.

When Test Automation Makes Sense

Let’s start with the tests that ideally are automated. These include:

* Regression and confirmation. Rerunning a test against a new release to ensure that behavior remains unbroken—or to confirm that a bug fix did indeed fix the underlying problem—is a perfect fit for automated testing.

* Monkey (or random). Tests that fire large amounts or long sequences of data, transactions, or other inputs at a system in a random search for errors are easily and profitably automated

* Load, volume, and capacity. Sometimes, systems must support tremendous loads. On one project, we had to test how the system would respond to 50,000 simultaneous users, which ruled out manual testing! Two Linux systems running custom load-generating programs filled the bill.

* Performance and reliability. With the rise of Web-based systems, more and more automated testing is aimed at looking for slow or flaky behavior on Web systems.

* Structural, especially API-based unit, component, and integration. Most structural testing involves harnesses of some sort, which brings you most of the way into automation.

Other tests that are well-suited for automation exist, such as the static testing of complexity and code standards compliance. In general, automated tests have higher upfront costs—tools, test development, environments, and so forth—and lower costs to repeat the test.

When to Focus on Manual Testing

* High per-test or maintenance costs are one indicator that a test should be done manually. Another is the need for human judgment to assess the correctness of the result or extensive, ongoing human intervention to keep the test running. For these reasons, the following tests are a good fit for manual testing:

* Installation, setup, operations, and maintenance. In many cases, these tests involve loading CD-ROMs and tapes, changing hardware, and other ongoing hand-holding by the tester.

* Configuration and compatibility. Like operations and maintenance testing, these tests require reconfiguring systems and networks, installing software and hardware, and so forth, all requiring human intervention.

* Error handling and recovery. Again, the need to force errors—by powering off a server, for example—means that people must stay engaged during test execution.

* Localization. Only a human tester with appropriate skills can decide whether a translation makes no sense, is culturally offensive, or is otherwise inappropriate. (Currency, date, and time testing can be automated, but the need to rerun these tests for regression is limited.)

* Usability. As with localization, human judgment is needed to check for problems with the facility, simplicity, and elegance of the user interface and workflows.

* Documentation and help. Like usability and localization, checking documentation requires human judgment.

Wildcards

In some cases, tests can be done manually, be automated, or both.

* Functional. Functionality testing can often be automated, and automated functional testing is often part of an effort to create a regression test suite or smoke test. However, it makes sense to get the testing process under control manually before trying to automate functional testing. In addition, you’ll want to keep some of the testing manual.

* Use cases (user scenarios). By stringing together functional tests into workflows, you can create realistic user scenarios, whether manual or automated. The trick here is to avoid automation if many workflows involve human intervention.

* User interface. Basic testing of the user interface can be automated, but beware of frequent or extensive changes to the user interface that can incur high maintenance costs for your automated suite.

* Date and time handling. If the test system can reset the computer’s clocks automatically, then you can automate these tests.

Higher per-test costs and needs for human skills, judgment, and interaction push towards manual testing. A need to repeat tests many times or reduce the cycle time for test execution pushes towards automated testing.

Reasons to Be Careful with Automation

Automated testing is a huge investment, one of the biggest that organizations make in testing. Tool licenses can easily hit six or seven figures. Neophytes can’t use most of these tools—regardless of what any glossy test tool brochure says—so training, consulting, and expert contractors can cost more than the tools themselves. Then there’s maintenance of the test scripts, which generally is more difficult and time consuming than maintaining manual test cases.

Principle 2: Exhaustive testing is impossible
Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases. Instead of exhaustive testing, we use risks and priorities to focus testing efforts.

Principle 3: Early Testing
Testing activities should start as early as possible in the software or system development life cycle and should be focused on defined objectives.

Principle 4: Defect clustering
A small number of modules contain most of the defects discovered during pre-release testing or show the most operational failures.

Principle 5: Pesticide paradox
If the same tests are repeated over and over again, eventually the same set of test cases will no longer find any new bugs. To overcome this ‘pesticide paradox’, the test cases need to be regularly reviewed and revised, and new and different tests need to be written to exercise different parts of the software or system to potentially find more defects.

Principle 6: Testing is context dependent
Testing is done differently in different contexts. For example, safety-critical software is tested differently from an e-commerce site.

Principle 7: Absence-of-errors fallacy
Finding and fixing defects does not help if the system built is unusable and does not fulfill the users’ needs and expectations.