Acceptance Testing Introduction and Basics

Fundamental Relationships

While it is certainly true that the roles and responsibilities of the tester and the business analyst are different, it is also true that their activities are complementary; work done by one group may greatly affect, either positively or negatively, that of the other. This is especially true in acceptance testing which is performed to assess the system’s readiness for deployment and its use by the customer (end-user). Good collaboration between business analysts and testers is particularly important for a proper consideration of the business implications at this test level.

Business Goals, Business Needs and Requirements

Business analysts first must understand the organisation’s overall business goals and identify current business processes and stakeholders. Once that is done, they describe specific business needs and determine a business case that addresses those needs. Once this high-level work has been completed, requirements can be elicited for the business solution that shall be developed.

Business goals, business needs, business requirements, and product requirements describe, at different levels of abstraction, what shall be achieved. In Agile development, the same principles apply, but different terms may be used (for example features and user stories).

In this document, the term “requirements” refers both to business requirements and to product requirements.

Requirements / User Stories, Acceptance Criteria and Acceptance Tests

During requirements elicitation, business analysts and testers (possibly together with developers) should begin to create specific acceptance criteria and develop acceptance tests as a joint effort. This ensures that there is a mutual understanding of what “acceptable” means from the business, development, and testing perspectives, right from the beginning of the project.

Acceptance criteria relate directly to a specific requirement or user story. They are either part of the detailed description or an attribute of the related requirement. If user stories are used, acceptance criteria are part of the user story’s definition and extend the story. 

In all cases, acceptance criteria are measurable criteria, formulated as a statement (or a set of statements), which can be either true or false. They are used to check whether a requirement or user story has been implemented as expected. Acceptance criteria represent the test conditions which determine “what” to test. They do not contain the detailed test procedures. 

Acceptance test cases are derived from acceptance criteria. These tests specify how the verification of the acceptance criteria should be performed. 

The Importance of the Quality of the Requirements

If acceptance criteria and tests are based on requirements, user stories, and/or acceptance criteria that are vague or ambiguous, it is likely that testers will make assumptions about stakeholder expectations and business needs. In this case, the resulting acceptance tests may be flawed. This will lead to rework or, even worse, the running of invalid tests, thus creating unnecessary costs as well as risks and uncertainty about product quality assurance. 

It is critical for testers to work closely with business analysts to make sure that requirements are clear and well understood by all stakeholders concerned. Ambiguities should be resolved and assumptions should be clarified so that the resulting acceptance tests are valid and are a meaningful way to determine the product’s readiness for release. 

In Agile development, the INVEST criteria define a set of criteria, or checklist, to assess the quality of a user story. These may be used by business analysts / product owners, developers, and testers to ensure the quality of user stories.

Business Analysis and Acceptance Testing

Too often, business analysts and testers work in their own separate silos, which can lead to misunderstandings about business and customer expectations. Those misunderstandings may stay hidden until the release approaches. By taking advantage of the complementary skills and by working together, business analysts and testers can positively affect the development process. This can be accomplished both by considering acceptance criteria and acceptance testing as early as possible and by coordinating efforts to make sure that the product has been tested appropriately prior to release at acceptance test level. 

Relationship between Business Analysis and Testing Activities

The following are the main elements of the business analysis activities: 

  • Strategy definition
  • Management of the business analysis processes
  • Requirements engineering in business analysis
  • Solution evaluation and optimisation

The business analyst is responsible for identifying business needs of stakeholders and for determining solutions to business problems with the aim of introducing change which adds value to the business. An important aspect of the business analyst’s role is to establish consensus between quality engineers, testers, developers, system integrators, product managers and project managers.

A test process consists of the following main groups of activities:

  • Test planning
  • Test monitoring and control
  • Test analysis
  • Test design
  • Test implementation
  • Test execution
  • Test completion

Quite a few of the associated activities and tasks relate to both business analysis and testing. The following examples illustrate the relationship between the two disciplines in the context of acceptance testing:

Requirements engineering in business analysis vs. test planning, test analysis and test design:

  • During the requirements engineering activities in business analysis, business analysts prepare detailed business and product requirements. These requirements are part of the test basis for the test planning, test analysis and test design activities, as testers define their objectives and plan their work, evaluate the specifications and requirements, identify test conditions and design test cases and test procedures.
  • Testers can contribute to the definition and verification of acceptance criteria as part of test analysis and test design activities. Working together, the two roles ascertain that there is proper understanding of the solution and agree on the appropriate approach to acceptance testing.
  • When requirements change, business analysts and testers can work together to assess the impact of the changes.

Solution evaluation in business analysis vs. test implementation, test execution and test completion:

  • During the solution evaluation phase in business analysis, business analysts support test implementation and test execution activities. They review the testers’ procedures/scripts, clarify issues and potentially help with creation of test data to support business-related tests.
  • Business analysts can assist with the implementation and execution of the acceptance tests. They may also support testers by evaluating test results. In addition, they may assist testers in test completion activities.

There is a strong and symbiotic relationship between the two roles and their respective activities, starting at the very beginning of a project and continuing until acceptance or release of the solution.

Collaboration between Business Analysts and Testers in Acceptance Testing

The common goal for business analysts and testers is to support the production of products with the highest possible value for the customer. Given their position within the organisation, business analysts and testers have various opportunities to collaborate during the acceptance testing activities described in the previous section. Apart from joint discussions and reviews of generated artefacts, business analysts and testers collaborate in other areas. For example, collaboration on test planning based on risk analysis is a good opportunity to ensure that the appropriate test cases will be developed and prioritised.

In addition to the direct benefits of working together and supporting each other’s efforts during acceptance testing, there is an important opportunity to cross-train team members. The more testers know about business needs and stakeholder requirements, and the more business analysts know about structured testing, the more likely the two groups will understand and appreciate each other’s work and better collaborate within the project.

How Acceptance Testing Can Drive the Development Process: ATDD and BDD

The wide acceptance of Agile software development practices has influenced how acceptance testing relates to requirements elicitation and other business analysis activities. In sequential lifecycle models, acceptance test analysis, design, and implementation are activities to be handled by the testers after the requirements are finalised. With the Agile lifecycle model, acceptance criteria and acceptance test cases are created during requirements analysis, requirements refinement sessions, and product backlog refinement. This allows the implementation of the “Early Testing” principle by using the design of test cases as part of the requirements definition activities. 

In the following two approaches, acceptance test analysis and design are formally part of the requirements engineering process: 

  • In Acceptance Test-Driven Development (ATDD), acceptance tests are produced collaboratively during requirements analysis by business analysts, product owners, testers and developers.
  • Behaviour-Driven Development (BDD) uses a domain-specific scripting language, Gherkin, that is based on natural language statements. The requirements are defined in a ‘Given – When – Then’ format. These requirements become the acceptance test cases and also serve as the basis for test automation.
  • Both of these approaches engage the entire Agile team and help to focus the development efforts on the business goals. The approaches also treat the acceptance test cases as living documentation of the product because they can be read and understood by business analysts and other stakeholders. Acceptance test cases represent scenarios of usage of the product.
  • The two approaches are similar and the two terms are sometimes used interchangeably. In practice, BDD is associated with the use of Gherkin to support writing acceptance tests, while ATDD relies on different forms of textual or graphic acceptance test design. For example, the graphical representation of application workflows may be used to implement a visual ATDD approach.