Usability Testing Basic Concepts

Fundamentals

This section considers the following fundamental concepts: 

  • Usability
  • User experience
  • Accessibility

Usability

Usability is the extent to which a software product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. Usability testers should be aware that other definitions may be used in organisations.

The user interface consists of all components of a software product that provide information and controls for the user to accomplish specific tasks with the system.

Usability evaluation includes the following principal activities:

  • Usability reviews
  • Usability testing
  • User surveys

A usability problem is a software defect which results in difficulty in performing tasks via the user interface. This affects the user’s ability to achieve their goals effectively, or efficiently, or with satisfaction. Usability problems can lead to confusion, error, delay or outright failure to complete some task on the part of the user. In safety-critical systems such as medical systems, usability problems can also lead to injuries or death. 

A software product can work exactly to specification and still have serious usability problems, as shown by the following examples: 

  • A car rental mobile app has a dead link. This is a defect which results in a usability problem.
  • A car rental mobile app allows users to cancel a reservation, but the users perceive the cancellation procedure as unreasonably complicated. This is a usability problem which affects the efficiency of the mobile app.
  • A car rental mobile app conforms to the specification and works both effectively and efficiently, but users think it looks unprofessional. This is a usability problem which affects user satisfaction when using the mobile app.

Usability always relates to the context of use and can be considered in different components. As the following examples show, user expectations of usability are rather different for these components. 

ComponentComponent Name
1Users
2Tasks
3Equipment
4Environment

Description of Component in Context of Use 

1. A user is a person who interacts with a software product by providing inputs, or by using the output of the software product.

2. Particular activities performed by users or particular groups of users (e.g., inexperienced users, administrators). 

3. Equipment relates to the hardware, software and materials required to use a software product.

4.  The environment consists of the physical, social and technical conditions in which a user interacts with a software product. The social conditions include the organisational conditions.

The following scenarios describe different contexts of use for the same software product: 

  • Administrative staff use Microsoft Word ® to write documents in a consultancy firm
  • An elderly person uses Microsoft Word® for the first time to write an invitation to her birthday

User Experience Concepts

User experience describes a person’s perceptions and responses that result from the use and/or anticipated use of a product, system or service.

User experience includes the following user characteristics that occur before, during and after use: 

  • emotions
  • beliefs
  • preferences
  • perceptions
  • physical and psychological responses 
  • behaviours and accomplishments

User experience is influenced by: 

  • brand image (i.e., the users’ trust in the manufacturer)
  • presentation (i.e., the appearance of the software product, including packaging and documentation)
  • functionality
  • software product performance
  • interactive behaviour
  • the helpfulness of the software product, including help system, support and training
  • learnability
  • the user’s internal and physical state resulting from prior experiences, attitudes, skills, personality, education and intelligence
  • the context of use

Usability criteria such as effectiveness, efficiency and satisfaction can be used to assess aspects of user experience such as brand image and presentation (satisfaction), functionality (effectiveness) and software product performance (efficiency).

Accessibility

Accessibility is the degree to which a product or system can be used by people with the widest range of characteristics and capabilities to achieve a specified goal in a specified context of use.

Evaluating Usability, User Experience and Accessibility

The key objectives of usability evaluation, user experience evaluation and accessibility evaluation are compared in the following table and discussed in more detail in subsequent sections.

Type of evaluation 

Usability evaluation 

User experience evaluation 

Accessibility evaluation 

Target group 

All users 

Key objective 

 Evaluate the direct interaction between users and the software product. 

  • Evaluate the services received prior to the use of the software product.
  • Evaluate the direct interaction between users and the software product.
  • Evaluate the services received after the use of the software product.

Evaluate the direct interaction between users and the software product, focusing on understanding problems related to accessibility barriers, rather than general efficiency or satisfaction. 

The principal techniques applied in usability evaluation, user experience evaluation and accessibility evaluation are shown in the following table and discussed in more detail in later chapters. 

Technique 

Usability review 

Usability testing 

User surveys 

Users involved? 

Optionally and Yes 

Key characteristic 

Experts and users evaluate the user interface of a software product for usability problems; the evaluation is based on their experience. 

Users are observed while they perform typical tasks with the software product. 

Users fill out questionnaires regarding their satisfaction with the software product. 

Specific techniques 

Informal usability review 

Expert usability review 

Heuristic evaluation 

Think aloud testing 

International Software Testing

Qual = Qualitative usability evaluation

Quant = Quantitative usability evaluation 

Usability Evaluation

A process through which information about the usability of a system is gathered in order to improve the system (known as formative evaluation) or to assess the merit or worth of a system (known as summative evaluation). 

There are two types of usability evaluation: 

  • Formative (or “exploratory”) evaluation is conducted to understand usability issues. Formative evaluation is often conducted early on in the development lifecycle during the design and prototyping stages to get ideas and to guide (or “form”) the design by identifying usability design problems.
  • Summative evaluation is conducted late in the development lifecycle shortly before or after implementation to measure the usability of a component or software product. Summative usability testing is quantitative; it focuses on obtaining measurements for the effectiveness, efficiency or satisfaction of a software product. A summative usability evaluation can be used to evaluate a design based on usability requirements so that the design’s acceptability can be established from the users’ point of view.

Both types of evaluation can be conducted iteratively.

Usability evaluation relating to software products. Usability evaluation can also be applied to other products or services where usability is important, such as with user guides, vending machines, aircraft cockpits, medical systems and train stations.

Usability evaluation addresses the direct interaction between users and the software product. The direct interaction occurs via a screen dialogue or other form of system use. Usability evaluation can be based on a software application, on design documents and on prototypes.

The objectives of usability evaluation are: 

  • to assess whether usability requirements have been met
  • to uncover usability problems so they can be corrected
  • to measure the usability of a software product (see below)

Usability evaluation addresses the following: 

Effectiveness:

  • The extent to which correct and complete goals are achieved
  • Answers the question: “Does the software product do what I want?” 

Efficiency:

  • Resources expended to achieve specified goals
  • Answers the question: “Does the software product solve my tasks quickly?”

Satisfaction:

  • Freedom from discomfort, and positive attitudes towards the use of the software
    product
  • Answers the question: “Do I feel comfortable while using the software product?”

If users are involved, a usability evaluation can be carried out by performing usability testing, conducting user surveys and performing usability reviews. If users are not present, usability reviews may still be performed. If software will be used by disabled individuals, include them early in usability reviews (i.e., color blind users). 

A qualitative usability evaluation enables identification and analysis of usability problems, focusing on understanding user needs, goals and reasons for the observed user behaviour. 

A quantitative usability evaluation focuses on obtaining measurements for the effectiveness, efficiency or satisfaction of a software product.

User Experience Evaluation

User experience describes a person’s perceptions and responses resulting from the use or anticipated use of a software product. 

Usability is part of the user experience. Consequently, usability evaluation is a part of user experience evaluation. The principal techniques used for user experience evaluation are the same as those used for usability evaluation. 

User experience evaluation addresses the whole user experience with the software product, not just the direct interaction. User experience includes: 

  • Advertisements that make users aware of the software product
  • Training in the use of the software product
  • Touch-points with the software product other than screen dialogue, such as encounters with support, letters or goods received as a result of interaction with the software product
  • Problems that are not handled by the user interface of the software product, such as the notifications of delays, handling of complaints and unsolicited calls

User experience can be evaluated using the principal techniques outlined in the tables above. In a user experience test, time gaps can be bridged during a usability test session.

Accessibility Evaluation

Accessibility evaluation is a usability evaluation which focuses on the accessibility of a software product. It addresses the direct interaction between a user with disabilities or limitations and the software product. 

The following advice applies specifically to accessibility evaluation: 

1. Define the ambition level for accessibility
The Web Content Accessibility Guidelines (WCAG) document defines three priority levels for accessibility; A, AA and AAA. It is recommended to adopt conformance level AA, which implies satisfying the most basic requirements for web accessibility and the biggest barriers for users with disabilities. 

2. Create or adept guidelines for accessible design.
These guidelines should comply with legal requirements. They should also be in accordance with the chosen ambition level for accessibility. Additionally, the usability of the guidelines for developers should be verified. 

  • Review the guidelines for accuracy
  • Establish an accessibility hotline, where accessibility questions from development teams can be answered competently within an agreed time limit

3. Train development teams in order to prevent as many accessibility problems as possible. This includes factors such as: 

  • Legal requirements for accessibility
  • Guidelines for accessible design and how to interpret and apply them
  • Tools and techniques to use when evaluating accessibility
  • The relationship between usability and accessibility

4. Accessibility testing focuses on the following aspects:

  • Use of a think aloud technique to understand the test participant’s thoughts and vocabulary during accessibility testing
  • Focus on understanding mistakes related to accessibility barriers, rather than on efficiency or satisfaction
  • Use tasks that concentrate on specific areas of concern for potential accessibility problems, rather than on general software product usage

Accessibility evaluation should consider relevant accessibility standards.

Usability Evaluation in Human-Cantered Design

Human-cantered design activities and their interdependence. Human-cantered design is an approach to design that aims to make software products more usable by focusing on the use of the software products and applying human factors, ergonomics, and usability knowledge and techniques.

The human-cantered design process can be summarised as follows: 

  • Analyze: Talk with people and discover “what is the problem?”
  • Design: Prototype what you assume is a solution
  • Evaluate: Watch people use the prototype and learn from their experiences
  • Iterate: Repeat until the usability requirements are achieved
Human-centered design activities and their interdependence

The human-cantered design activities are based on the following three key elements: 

1. Users 

Observe and interview users in their work environment. Users are involved throughout the design stage by discussing designs and alternatives with them directly (where possible), or with representative users. In agile software development, representative users are typically the product owners, who are an integral part of the development team and enable frequent feedback to be given to designers and developers on usability issues. 

2. Evaluation 

Perform usability evaluation on the software product. A usability evaluation may take place at any time during human-cantered design, from early analysis through software product delivery and beyond. A usability evaluation may be based on a prototype, as mentioned above, or on a completed software product. Usability evaluations that are conducted in the design phase can be cost effective by finding usability problems early. 

3. Iterations 

Iterate between design and usability evaluation. 

Considering the human-cantered design process, the most frequent iterations take place between the activities “Produce design solutions” and “Evaluate design solutions”. This generally involves the successive development of a prototype, which is a representation of all or part of a software product’s user interface. Although prototypes are limited in some way, they can be useful for usability evaluation. Prototypes may take the form of paper sketches or display mock-ups, as well as software products under design. Starting with an initial prototype, the following activities are performed:

  • The prototype is evaluated. The person who performs the evaluation conducts usability testing on the prototype.
  • The prototype is improved and refined based on the results of the evaluation. The person who performs the evaluation helps the developers evolve the prototype by incorporating user feedback into the design.

These activities are repeated until the usability requirements are achieved. When prototypes are developed in iterations, the steady refinement gives the user a more realistic impression of how the finished product will look and feel. Additionally, the risk of forgetting or ignoring usability issues is reduced.
Both usability and accessibility must be considered during the design phase. Usability testing often takes place during system integration and continues through system testing and into acceptance testing. 

Usability Requirements

A usability requirement is a requirement on the usability of a component or system. 

It provides the basis for the evaluation of a software product to meet identified user needs. Usability requirements may have a variety of sources:

  • They may be stated explicitly, such as in requirements documentation or a user story
  • They may be implicit, undocumented user expectations (e.g., a user might implicitly expect that an application provides shortcut keys for particular user actions)
  • They may be included in adopted or required standards

Examples of usability requirements (in this case described as user stories) are:

  • “As a frequent user of the airline’s booking portal, an overview of my currently booked flights shall be automatically shown after I log on. This shall enable me to get a quick overview of my booked flights and quickly make any updates.”
    This usability requirement is about the effectiveness component of usability.
  •  “As a help-desk assistant, I must be able to enter and log the details of a customer request into the Customer Relations database in no more than two simple steps. This shall enable me to focus on the customer request and provide them with optimum support.” This usability requirement is about the efficiency component of usability.

Agile Usability Evaluation

Usability evaluations are also suitable in agile software development. 

Agile software development is a group of software development methodologies based on iterative incremental development, where requirements and solutions evolve through collaboration between members of a self-organising team. 

In agile software development, teams work in short iterations, each of which has the goal of designing, implementing and testing a group of features. 

The following usability evaluation approaches work well with agile software development: 

  • Rapid Iterative Testing and Evaluation (RITE) is a qualitative usability test method where changes to the user interface are made as soon as a usability problem is identified and a solution is clear. The RITE method focuses on instant redesign to fix problems and then confirming that the solution works with new test participants (real users or representative users). Changes can occur after observing as few as one test participant. Once the data for a test participant has been collected, the usability tester and the stakeholders decide if any changes are needed prior to the next test participant. The modified user interface is then tested with the remaining test participants.
  • Informal and quick usability test sessions are useful where many potential users can be accessed (e.g., a cafe, a conference or a trade show). Such forms of usability test sessions typically last less than fifteen minutes and apply techniques such as think aloud and heuristic evaluation.
  • Weekly testing. Test participants are recruited well in advance and scheduled for a particular day of the week (e.g., each Tuesday), so that the software build can be usability tested on that day. Usability tasks are prepared shortly before the scheduled testing day and may include exploratory testing sessions, where the knowledge of the tester and heuristic checklists are used to focus on usability issues.
  • Usability reviews.

Acceptance Testing Business Process and Business Rules Modelling

Modelling Business Processes and Rules

Organisations need confidence that critical business processes, such as order-to-cash procedures, human resource on-boarding, or production planning, can be performed without disruption. This is known as “business process assurance” and it is an essential objective of acceptance testing. In this context, two standards exist that provide a common language for business analysts and testers for graphically representing business processes and business rules: Business Process Model and Notation (BPMN) and Decision Model and Notation (DMN). These models support the design and implementation of tests and help to determine the priority for execution.

Business process/rule models describe the business flow and the expected behaviour of the test object. Representing business processes and rules to be tested using a graphical notation helps to establish a common understanding of what is expected. A business process corresponds to a flow of tasks, alternative paths, and the various events at the start, the end or possibly during the control flow. Business rules define explicit criteria for guiding behaviour, shaping judgments, or making decisions. 

Business Process Model and Notation (BPMN), maintained by the Object Management Group (OMG), is a recognised standard for business process modelling which uses a flowcharting technique. In this article, a subset of the Business Process Model and Notation (BPMN) notation is used that is sufficient to draw simple business process models in the context of acceptance testing activities.

Decision Model and Notation (DMN), also standardised by the Object Management Group (OMG), is complementary to the BPMN standard. While Business Process Model and Notation (BPMN) is used to represent workflows, DMN is used to represent decisions, business rules and outcomes/output within the workflow. In this article, a subset of the Decision Model and Notation (DMN) notation is used that is sufficient to define business rules in conjunction with simple business process models in Business Process Model and Notation (BPMN).

Deriving Acceptance Tests from Business Process/Rule Models

A business process model with business rules, described with the Business Process Model and Notation (BPMN) and/or Decision Model and Notation (DMN) notations, provides a precise definition of the scenarios to be tested, including the cases related to business rules. It is a good basis for generating acceptance tests using coverage-based test selection criteria as defined in a model-based testing approach. 

Coverage-based test selection follows the principle that the business analyst and tester agree on the coverage items that shall be fully tested. Typical coverage items for business process models when generating acceptance tests include the following: 

  • User stories, requirements, and risks annotated in the business process model.
  • Decisions in the decision tables describing the business rules.
  • User scenarios defined by different paths through the business process model.
  • All paths (usually without loops) through the business process model.

Once the coverage items are defined, the tester then identifies a set of test cases that covers those items. Full coverage is achieved if the test suite covers each occurrence of the coverage item in the model at least once during execution.

Different coverage criteria may be combined to meet the acceptance testing objectives. For example, the objective may be to cover all paths of a given main scenario, but only one path of each alternative scenario.

Business Process Modelling for Acceptance Testing

Business process/rule models describe the business flow and the expected behaviour of the test object. The use of business process/rule modelling in the context of acceptance testing is based on good modelling practices and supports visual ATDD practices.

Good Practices for Business Process Modelling for Acceptance Testing

The following good practices should be considered when using Business Process Model and Notation (BPMN) and Decision Model and Notation (DMN) for acceptance testing: 

  • It is not necessary to describe everything in a business process model. The graphical representations of business processes in BPMN should focus on requirements to be tested. Therefore, workflow descriptions that only partially cover the behaviour of related software systems are acceptable, as long as they represent what is to be tested.
  • Especially for rule-based business processes, using decision tables helps manage dependencies. DMN supports the definition of conditions and outcomes corresponding to the business rules under test.
  • Diagrams should be as simple as possible and be structured in sub-processes when needed to limit the number of graphical elements in a single business process diagram. This improves readability and facilitates reviews.
  • Business process modelling for acceptance testing should be a collaborative work between business analysts and testers. Artefacts produced should be shared between and reviewed by both roles. Early and close communication between those two roles improves the quality of requirements or user stories as well as tests. (This is true for all test levels.)
  • Additional information such as links to user stories, requirements, risks, priorities and any other information useful for acceptance testing should be added to the diagrams using annotations. By keeping all relevant information in a single location, it becomes easier to make decisions and reasons are better documented.

Using Business Process Models for ATDD

During the refinement sessions for requirements and user stories, the business process and business rule models will help the team to get into the details of the expected behaviour and the acceptance criteria. The representation of workflows in Business Process Model and Notation (BPMN) and of rules in Decision Model and Notation (DMN) directly enable testers to design appropriate test cases that verify the acceptance criteria.

Business process modelling for ATDD is based on the following principles:

  • Business analysts and testers collaborate to model workflows and business rules using graphical notations such as BPMN and DMN.
  • These business process/rule models are reviewed with relevant stakeholders and contribute to the validation of the requirements and acceptance criteria.
  • Testers derive tests from these business process/rule models to ensure and demonstrate the required coverage through the different paths and business rules.
  • Business analysts and testers may also use the models to identify changes that necessitate test case maintenance and to select regression test cases.
  • Business process/rule models created and maintained for ATDD can be viewed as living documentation used by business analysts to present the actual behaviour of
    the test object.
  • Automated test generation techniques can be used to produce and maintain automated test scripts. The model-based testing approach can also be combined with keyword-driven testing and data-driven testing approaches.

Business process/rule modelling in ATDD provides a visualisation of the workflows to be tested. This is the major difference from the Gherkin language used in BDD.

Acceptance Criteria, Acceptance Tests and Experience-Based Practices

Writing Acceptance Criteria

Specifying acceptance criteria is an important acceptance testing task. It helps to refine requirements or user stories and provides the basis for acceptance tests. Business analysts and testers should collaborate closely on the specification of these criteria. This collaboration ensures high business value from the acceptance testing phase and increases the chance of a successful iteration or product release. 

Writing acceptance criteria forces business analysts and testers to think about functionality, performance, and other characteristics from a stakeholder or usage perspective. This supports early verification and validation of the related requirement or user story and provides a better chance of detecting inconsistencies, contradictions, missing information or other problems. 

The following good practices should be considered when writing acceptance criteria:

  • Well-written acceptance criteria are precise, measurable and concise. Each criterion must be written in a way that enables the tester to measure whether or not the test object complies with the acceptance criterion.
  • Well-written acceptance criteria do not include technical solution details. They concentrate on the question “What shall be achieved?” rather than on the question “How shall it achieved?”.
  • Acceptance criteria should address non-functional requirements (quality characteristics) as well as functional requirements.

As with requirements and user stories, acceptance criteria should be reviewed through walkthroughs, technical reviews, iteration planning meetings or other methods (if necessary).

Designing Acceptance Tests

This section addresses the test techniques and approaches frequently used for acceptance testing.

Test Techniques for Acceptance Testing

In a requirements-based approach to acceptance testing, the tester derives test cases from the acceptance criteria related to each requirement or user story using black-box techniques such as equivalence partitioning or boundary value analysis.

Acceptance testing may be augmented with other test techniques or approaches:

  • Business process-based testing, possibly combined with decision table testing, validates business processes and rules.
  • Experience-based testing leverages the tester’s experience, knowledge and intuition.
  • Risk-based testing is based on risk types and levels. Prioritisation and thoroughness of testing depends on previously identified product risks.
  • Model-based testing uses graphical (or textual) models to obtain acceptance tests.

Acceptance criteria should be verified by acceptance tests and traceability between the requirements / user story and related test cases should be managed.

Using the Gherkin Language to Write Test Cases

In ATDD and BDD, acceptance tests are often formulated in a structured language, referred to as the Gherkin language. Using the Gherkin language, test cases are phrased declaratively using a standardised pattern:

  • Given [a situation]
  • When [an action on the system]
  • Then [the expected result]

The pattern allows business analysts, testers and developers to write test cases in a way that is easily shared with stakeholders and may be translated into automated tests. 

The “Given” block aims to put the test object in a state before performing test actions in the “When” block. The “Then” block specifies the consequences that can be observed from the actions defined in the “When” block. Test cases written in Gherkin do not refer to user interface elements but rather to user actions on the system. They are structured natural language test cases that can be understood by all relevant stakeholders. 

In addition, the structure “Given – When – Then” can be parsed in an automated way. This allows automated test script creation using a keyword-driven testing approach. 

Initially, Gherkin was specific to some software tools supporting BDD, but it is now synonymous with the “Given – When – Then” acceptance test design pattern. 

Experience-based Approaches for Acceptance Testing

All experience-based test techniques described in are relevant for acceptance testing. This section is focused on how exploratory testing can be used for acceptance tests, and on beta testing as a source of feedback on system usage. 

Exploratory Testing

Exploratory testing is an experience-based test technique that is not based on detailed predefined test procedures. In exploratory testing, all activities are carried out within an uninterrupted period of time called a session. The testers are domain experts. They are familiar with user needs, requirements and business processes, but they are not necessarily familiar with the product under test. 

During an exploratory testing session, the tester accomplishes the following:

  • Learns how to work with the product
  • Designs the tests
  • Performs the tests
  • Interprets the results

It is a good practice in exploratory testing to use a test charter. The test charter is prepared prior to the testing session (possibly jointly by the business analyst and the tester) and is used by the person in charge of the exploratory session (either a business analyst, tester or another stakeholder). It includes information about the purpose, target, and scope of the exploratory session, the test setup, the duration of the session, and possibly some tactics to be used during the session (such as the type of user that shall be simulated during the exploratory session). Time-boxed sessions help to control the time and effort dedicated to the exploratory session. It is also good practice to perform exploratory testing in pairs or as team work. 

In Agile development, exploratory test sessions can be conducted during an iteration by the product owner and/or the testers for acceptance testing of user stories assigned to the iteration. 

Exploratory testing should be used to complement other more formal techniques in acceptance testing. For example, it may be used to provide rapid feedback on new features before methodical testing is applied. 

Beta Testing

Beta testing is a form of acceptance testing that is often used for Commercial Off-the-Shelf Software (COTS) or for Software as a Service (SaaS) platforms. It is conducted to obtain feedback from the market after development and in-house testing are completed. 

Unlike other acceptance testing forms, beta testing is performed by potential or existing users at their own location. Beta tests neither impose predefined test procedures nor a test charter. Apart from the observed findings, the test activities are usually not documented at all. 

Because the product is tested in various realistic configurations by actual users in their business process context, beta testing may discover defects that escaped during the development process and previous test levels. Resolving issues found by beta tests helps organisations avoid costly hot-fixes or product recalls on a larger scale. 

Acceptance testing should not be limited to beta testing. Beta testing is not systematic or measurable. There is no guarantee that all requirements or user stories are covered by the tests. Moreover, beta testing is performed late in the development process whereas tests based on acceptance criteria support the “Early Testing” principle. 

The Gaming Industry Ecosystems

Testing phases within the Gaming Software Development Lifecycle

Test types during the gaming quality assurance phase

Compliance testing is a very involved process. There are lengthy submission forms to be filled and ITL compliance fees are often much higher than gaming quality assurance testing fees. Therefore, prior to submitting a product to an ITL, a machine manufacturer should ensure the quality of the product being submitted by performing gaming quality assurance testing.

Gaming quality assurance testing activities align with the fundamental test process. The following test activities are performed to ensure the product is tested for quality:

  • Test planning
  • Test monitoring and control
  • Test analysis
  • Test design
  • Test implementation
  • Test execution
  • Test completion

Gaming quality assurance testing is an iterative process with development. Defects are logged and then the product is returned to development to fix the logged defects. Once the logged defects are fixed, the product comes back to gaming quality assurance for further testing. This cycle continues until the product reaches the quality levels desired, as defined by the gaming quality assurance testing exit criteria.

The test types performed during gaming quality assurance testing include, but are not limited to:

  • Localisation testing – testing to determine that all screens have proper language translations and any other localisation items such as date/time or numbering formats are done correctly.
  • Functional testing – testing to determine that all functions work as designed and intended.
  • Performance testing – testing to determine that response time is acceptable.
  • Memory leak testing – testing to determine no performance issues arise due to lack of proper memory management by the software.
  • Install-ability testing – testing to determine that the product can be easily installed by the casino operator without any issues.
  • Portability testing – testing to determine that the product can be installed on all platforms it needs to run on.
  • System integration testing – testing to determine that the product can integrate with systems/components, different versions of an application interface or able to communicate to a system.
  • Operational testing – testing to determine that the product can function and be operated in a production environment, including reliability, security, recovery and failover testing.
  • Pre-compliance testing – testing to determine that all the regulations and standards are met prior to submitting the product to the ITL. This helps ensure a minimal number of submissions of the product to compliance testing.
  • Customer acceptance testing – Prior to submitting to compliance testing, some products are submitted to the client (i.e., lottery or casino) for customer acceptance testing to ensure all features function as the client expected.

Compliance testing

Compliance testing occurs when a machine manufacturer wants to enter a jurisdiction with a new or modified product, be it a game, platform, hardware or system. Based on the jurisdiction, the machine manufacturer needs to submit the product with a request for certification to either an ITL or government-based compliance test lab. In some jurisdictions, it goes through both. 

The formal submission documents are fully detailed, what is being submitted and what certifications are being requested for the product.

Once the product has passed compliance testing, the test lab will provide a certificate of compliance evidencing the certification of the rules and regulations that were to be met.

Once the regulatory commission has seen proof of the required certifications, it will allow the product to be installed in the gambling establishments in their jurisdictions.

When performing compliance testing, standard test plans are created and specific compliance checklists are used. These may include, but are not limited to: 

  • Jurisdictional specifications – usually defined by a governing body such as the federal, state and/or provincial government.
  • ITL defined standards – defined by an ITL in the gambling industry.
  • Other gaming related standards – some jurisdictions require other standards be adhered to. For example, some jurisdictions may require that gaming machines and systems in a jurisdiction are Game to System (G2S) protocol compliant. The G2S compliance checklist is defined by the Gaming Standards Association, the association that has defined the G2S
    protocol.

Many areas of the compliance testing will be the same as those performed in gaming quality assurance testing, but they are tested against the jurisdictional specifications and not the game specifications.
Some of the areas that are covered during compliance testing include: 

  • Rules of play – testing to determine that the rules meet the jurisdictional specifications.
  • RNG, Payout Percentages, odds and non-cash awards – testing to determine that the payout percentage is within the range regulated in that jurisdiction.
  • Bonus games – testing to determine that the game meets bonus regulations.
  • Electronic metering – testing to determine that all meters required to be
    monitored within that jurisdiction are being reported.
  • Game history – testing to determine that the game history tracks, at a minimum, the number of games required by the jurisdiction.
  • Power-up and power-down – testing to determine that the power up and down functionality works as per the jurisdictional specifications.
  • Setup and Configuration – testing to determine that only configurations that are permitted within the jurisdiction can be enabled.

The Gaming Ecosystem

The gaming industry ecosystem overview

The gaming industry ecosystem is composed of the following organisations: 

  • Game Developers – develop casino games not specific to a gaming machine model. These games are usually distributed by a manufacturer or casino.
  • Machine Manufacturers – make and sell the hardware, platforms, operating systems and games, developed in house or sub-contracted.
  • Independent Test Labs – test and certify that the game software, hardware, firmware, platform and operating system follow all the jurisdictional rules for each location where the game will be played.
  • Regulatory Commissions – approve every game played in their jurisdiction after the ITL certifies that the game meets the commission’s jurisdictional specifications.

The regulatory commission licenses the machine manufacturer to deploy the game in casinos or on online gaming sites in that specific jurisdiction. A game may be shipped to a casino before licensing; however, it cannot be deployed. The game must be licensed by the regulatory commission before it is deployed into the jurisdiction. Should any major defects be found in the casino, the regulatory commission can force the machine manufacturer to pull their game out of all casinos or demand that the online sites remove access to the game in that jurisdiction.

Video lottery terminals and their ecosystem

As indicated by the name, VLTs always have a video display for the game. VLTs either have standalone or server-based outcome architectures. In the standalone model, each VLT contains an RNG from which game outcomes are generated. In the server-based outcome architecture, VLTs obtain their outcomes from the server. This architecture has two possible models: the RNG model or the pre-determined finite pool model. In the server-based RNG model, the server generates the outcome it will provide to the VLT using an RNG located in the host. In the pre-determined finite pool model, the server obtains the outcome from a database of pre-determined outcomes. This model is similar to instant tickets and is often referred to as electronic instant tickets.

The types of games typically found on a VLT are: mechanical reel games, poker games and keno games. Most VLTs are multi-game machines, meaning multiple games are available for a player to choose from through a screen menu.

VLTs are frequently operated in a distributed environment over a Wide Area Network. For example, a few VLTs deployed in bars and/or pubs are connected to a central server through a Wide Area Network connection. 

The VLT ecosystem is composed of: 

  • The EGM
  • The site controller and/or bank controller
  • The systems/servers used for monitoring and/or managing functionality

The EGMs are the machines on which the players choose to play the games. Each machine communicates to a site controller and/or bank controller and one or more central servers through a communication interface board using an electronic communication language referred to as a protocol. When VLTs are installed in a distributed environment, each retail location has a site controller to which the VLTs at that location are connected. The site controller serves multiple functions:

  • Communicates and monitors VLTs to ensure they are online.
  • Records game play transactions, cash-in/cash-out transactions and security
    exceptions.
  • May act as a protocol converter by translating the protocol implemented on the VLT to the protocol understood by the central server.
  • Provide retailers with the ability to:
    • Register players for player tracking cards 
    • Validate and pay out cash tickets 

When VLTs are installed in a venue environment (i.e., a non-distributed environment), they are connected to a bank controller which functions like a site controller minus the retailer functions. A bank controller can support connection of several hundred VLTs, whereas one site controller typically supports fewer than 100 connected VLTs. 

The VLTs and bank controllers and/or site controllers are connected to various central servers based on the functionality offered by a jurisdiction. At a minimum, VLTs installed in a venue environment include the following: 

  • A casino accounting system, which is responsible for monitoring the amounts wagered and paid on each VLT.
  • A VLT CMS, which provides the ability to monitor game play, track, record and report security exceptions at the VLT and/or site controller, and monitor network availability in order to ensure continuous VLT operations in the event of communication loss.

Other central servers may include additional features, not limited to:

  • A cashless wagering server, which allows for cashless transactions either through ticket-in/ticket-out (TITO) functionality or through electronic funds transfer (EFT).
  • A distributed game content management server, which controls the selection, scheduling, distribution and auditing of VLT software to VLTs at remote retail sites.
  • A player services server, which supports player loyalty, player rewards and
    responsible gaming functionality.
  • A progressive server, which manages progressive game play.
  • A business intelligence server, which provides data warehousing and business analytics.

The other servers available are based on the functionality offered in the jurisdiction.

Slot machines and their ecosystem

Slot machines may have a video display or mechanical reels which have actual physical reels that spin. 

Slot machines outcome architectures come from the Indian Gaming Regulatory Act, a 1988 US Federal law that establishes the jurisdictional framework that governs Indian Gaming in the US, This law provides definitions for Class I, Class II and Class III architectures. Class I relates to traditional Indian gaming and will not be discussed further in the context of casino gaming. Class II and Class III define the two outcome architectures used by slot machines. Class II (also know as electronic bingo) is defined in the Act as “the game of chance commonly known as bingo whether or not electronic, computer or other technological aids are used”. Class III (also known as traditional slot machines) has a broad definition in the Act. It states “all forms of games that are neither Class I nor Class II. Games commonly played at casinos, such as slot machines and table games, e.g., blackjack, craps, roulette, etc., fall in the Class III category. 

The types of games typically found on a slot machine are: mechanical reel games, bingo games, poker games and keno games. Many slot machines are single-game machines, meaning only one game is available for play on the gaming machine. 

Slot machines are typically operated in a venue environment such as a casino. 

Slot machines (also known as Vegas style slot machines) are: 

  • Casino gaming machines with mechanical reels or a video display.
  • Machines that have an RNG that is local to that machine.

Machines usually include a currency input device, such as a coin acceptor or a note acceptor, and a currency output device, such as a coin hopper.

The slot machine ecosystem is composed of:

  • The slot machines
  • A slot machine interface board (SMIB)
  • A data collection unit or bank controller
  • Central servers

Each slot machine contains an SMIB that is linked to the data collection unit or bank controller. Historically, an SMIB was a small board that was put into every mechanical or electro-mechanical machines. These early SMIBs connected to a wiring harness that would detect when a mechanical meter was incremented, or a mechanical door switch was opened. As time passed, these SMIBs evolved and now communicate electronically with the gaming machine and are often responsible for implementing the protocol that is used to communicate with the data collection unit or bank controllers and the remote central servers. The SMIBs, at a minimum, capture:

  • The amounts wagered by the player.
  • The amounts paid out to the player.
  • And if the player is using a player card, any player data tracked by the casino.

Data collection units or bank controllers, as suggested by the name, are used to collect and store the data obtained from the SMIBs.

The data collected by the data collection unit or bank controller is communicated to the servers to update the data needed for the functionality provided by the servers.

Bingo machines (also know as electronic bingo machines), are: 

  • Machines that look and feel like slot games but are actually a game of electronic bingo.
  • Machines on which the outcomes are obtained from a centralised bingo server.
  • Machines that offer cashless input methods such as TITO or EFT. These
    gambling machines do not have currency input/output devices.
  • Machines on which the games:
    • Are played exclusively against other players rather than against the house or against a player acting as a bank.
    • Are based on multiple players playing for a common prize. 
    • Continue until there is a winner.

Each bingo machine contains an SMIB that is linked to the bingo server and other servers. The SMIBs, at a minimum, capture: 

  • The amounts wagered by the player.
  • The outcome obtained from the server and the corresponding results.
  • The amount paid out to the player.

The Network Switches are used to provide multi-player capabilities. Once a minimum number of players required for a specific game is met, the actual bingo game can start. The bingo server is the system that allows players to join a game until the group is at the minimum required and that provides the outcomes to the slot machines.

There are other central servers, such as the casino accounting server that tracks the amounts wagered and amount won, and the reporting server that allows the casino operators to report on the collected data.

Lottery and its ecosystem

The lottery ecosystem is composed of systems and devices deployed at the lottery and at each retail location. 

The main device is the point of sale (POS) lottery terminal. The POS lottery terminal facilitates the sale of traditional lottery tickets by allowing the retail employee to either scan a selection slip containing the player selected numbers, or to select a Quick Pick option where the POS lottery terminal randomly selects numbers for the player. The POS lottery terminal then prints the tickets on the attached printer. The POS lottery terminal facilitates the sale of instant tickets by scanning the instant ticket sold. The attached customer display unit (CDU) allows the customer to view all steps of the sales transaction. The POS lottery terminal must coordinate all lottery ticket sale transactions with the lottery CMS. The POS Lottery terminal, printer and CDU/PDU unit are either separate devices or, in some cases, these devices can be integrated into one unit. When a player is ready to validate a ticket, the player can choose to have the retail employee scan the ticket using the POS Lottery terminal or perform the validating themselves on a POS Self-Serve Terminal. 

The final device at the retail location is the multimedia display. The multimedia display is used for in-store advertising of lottery products, upcoming lottery promotions and winning numbers. 

Once the numbers are drawn, the numbers are entered into the lottery CMS. Using the data stored in the database of the CMS, reports can be generated indicating how many winning traditional lottery tickets were sold and which retail location sold the ticket(s). For instant tickets, the barcode data of each ticket is stored in the CMS database. This allows the lottery employees to generate reports indicating how many tickets remain unsold and manage replenishment of physical tickets to retailers. The lottery CMS is responsible for storing all transactional data of tickets sold at each retail location. It also manages the advertising content to be displayed on the multimedia display units at the retail locations and downloads the appropriate content to the POS lottery terminal from which the multimedia display unit displays the content. 

Lotteries are beginning to introduce alternative means by which to purchase lottery tickets. For example, purchases can be made at self-serve vending machines for instant tickets, self-serve ATMs like kiosks for traditional lottery tickets or online at their website for lottery tickets. These alternative devices and components must also coordinate all sale transactions with the lottery CMS. 

Some of the areas that are covered during functional testing of the lottery ecosystem include:

  • Online game rules and functionality.
  • Scratch ticket management.
  • Player account management. 
  • Geolocation functionality.
  • Player services manager.
  • Multi-player gaming
  • Player loyalty and rewards.
  • Account based play “cashless”.
  • Responsible gaming functionality
  • Game play functionality and playability.
  • RNG algorithm and math rules.
  • Artwork versus requirements
  • Host accounting and reporting – determine that the game pays out what it should and that the money at play goes to the client if they win, or to the casino if the client loses.
  • Tournament and real time event setup and management.
  • Multiple game engines functionality and capabilities.
  • Integrations with external gaming sites and mobile devices.

Introduction to the Gaming Industry

Objectives and Overview

Understand the objective of the gaming tester

Being a gaming industry tester means that you must understand both testing in general, and the unique set of skills for the gaming industry ecosystem. This ecosystem is filled with proprietary, complex, multifaceted gaming software, hardware, platforms, firmware and operating systems. The objective of this article  is to provide the regular Tester graduates with the specific knowledge that is required for a career in gaming industry testing. 

Why the gaming industry requires a specialist tester

Some of the specific testing for the gaming industry, not present in other testing areas, include the following: 

1. Gaming industry ecosystem – The unique hardware, firmware and operating systems that are proprietary to the gaming industry. 

2. Gaming industry compliance testing – There are over 440 different certification boards worldwide for gaming industry games. These boards have rules that games in the gaming industry must comply with. These rules impact hardware, software, platform, operating systems, visual and auditory functionality, mathematics, and return to player (RTP) calculations. One gaming industry game can be played in multiple gaming jurisdictions and needs to comply with the laws of each location. 

3. Fun factor or player perspective testing – This is something unique to gaming industry games, since they are an entertainment product. Not only are casino games supposed to work intuitively and provide the player pleasure, they must also be fun to play. This requires a unique insight into game design, with experience and information about the user group and what that group enjoys. 

4. Math testing – Testing the multitude of pay tables, permutations, Random Number Generator (RNG) results and RTP computations. This type of testing requires the tester to understand what triggers different types of payout behaviour and to understand financial return to the player and how these triggers can be treated by different parameters. Understanding math testing is critical to succeed in this field. 

5. Audio testing – Creating sound or playing media is common in software. However, gaming industry game music must engage the user in the game and enhance the game play. Not only should the audio play without stuttering or missing elements, it should also add to the game play. This requires extensive audio skills and specific understanding of game audio. 

6. Multiplayer testing – This type of testing is performed when many players are simultaneously interacting with casino games, with computer-controlled opponents, with game servers, and with each other. Typical  risk-based testing is followed to ensure against using unlimited amounts of time testing different scenarios. Understanding multiplayer game design, and how to test it efficiently, is required knowledge for this type of testing. 

7. Interoperability Testing is common in all software that communicates with other software, systems and/or components. Casino/Video Lottery games have a unique aspect in that they must implement interoperability using gaming industry open protocol standards or proprietary protocols as per the specifications of the central server deployed in the jurisdiction to which the game is deployed.

Gaming Activities and Artefacts

Background 

To understand gaming industry testing and its ecosystem specificities requires a review of the business model, activities, and artefacts as they pertain to the gaming industry. 

What is gaming?

Gaming can be defined as follows: 

  • The wagering of money or something of value, also called stakes, on an event
  • Where the outcome of the event is unknown
  • Where the whole intent is winning additional money, material goods or trips 

What is a gaming machine? A gaming machine is a machine that enables the wagering of money or something of value. Examples of gaming machines are: electronic or mechanical slot machines, a roulette table or even a computer for online gaming. 

Types of Gaming

Casino games

There are three categories of casino games: table games, electronic gaming machines (EGMs) and random number ticket games. 

Examples of table games are roulette, blackjack, baccarat or poker, which typically are not tested unless they are an electronic table game version of these games. 

The second group are EGMs, typically known as video lottery terminals (VLTs) or slot machines. These are usually played by one player at a time and do not require the involvement of casino employees to play. These games need to be tested, i.e., the game software, the machines, the operating systems, and platforms that they are based on. 

VLTs and slot machines are both gaming machines that allow players to bet on the outcome of a game. Physically, VLTs and slot machines are very similar in nature. The main difference between a VLT and a slot machine is that VLTs are gaming machines that are operated by government lotteries while slot machines are gaming machines operated by private organisations such as casinos. 

Both VLTs and slot machines are regulated and require licenses to be operated within their jurisdictions. Many countries around the world offer legalised VLT or slot play. For example: 

  • In the United States, a 1988 federal law established three classes of games for Native American casinos, with different regulatory schemes for each. Each state government follows variations of these classes to define their regulations.
  • In Canada, the provincial or territorial governments are responsible for regulating gaming operations. All provinces offer the ability to play, each with their own regulations.
  • In Australia, the laws regulating the use of gaming machines are the responsibility of the state governments.

Other terms by which a VLT and slot machine are referred to: EGM, Video Gaming Terminal, Video Gaming Device, Video Slot Machines and Interactive Video Terminal.

The third casino game category is random number ticket games such as Keno and simulated racing. These games are based on the selection of random numbers, either from a computerised RNG or from other gaming equipment. 

Lottery systems

A lottery is a form of gaming that involves selling numbered tickets and giving prizes to holders of winning tickets. The prize can be a fixed amount of cash or goods, but more commonly, the prize fund is a fixed percentage of the revenues from the tickets sold. 

There are typically two-forms of lottery products sold: traditional lottery tickets and instant tickets. 

Traditional lottery tickets are numbered tickets that are sold for regularly scheduled draws, most often weekly. On the draw date, random numbers are drawn either using a ball drop machine or electronically. Most lotteries that have moved to electronic draws still have ball drop machines as a backup in case of failures with the software solution. Once the numbers are drawn from the ball drop machine, they are entered into the lottery central management system (CMS). 

The chances of winning a lottery jackpot can vary widely depending on the lottery design, and are determined by several factors, including: 

  • The count of possible numbers
  • The count of winning numbers drawn
  • Whether or not the order is significant
  • Whether drawn numbers are returned for the possibility of further drawing

Instant tickets are numbered tickets from a pre-determined finite pool of outcomes. The most common form of instant tickets is the scratch card. Scratch cards are typically made of paper, with the outcome printed and hidden by an opaque substance that needs to be scratched off, hence the name of these tickets. The cards usually present the information in the form of a game, such as Tic-Tac-Toe, Bingo, Crossword or some other puzzle, to help add entertainment value. A variation of the scratch card is the break-open (also know as pull-tab) ticket in which, instead of scratching off an opaque substance to reveal the outcome, the player opens a perforated cardboard cover which is hiding the outcome. Since outcomes of scratch and break-open tickets are pre-determined, the cards do not need to be scratched or opened to be validated.

A barcode on the ticket can be scanned by the lottery CMS to determine if it is a winner or not. The scratching or breaking open is there for entertainment value to the player only. 

The chances of winning on a scratch card are typically much higher than on a traditional lottery, but prize amounts are typically much smaller. The probability of winning on a scratch card can be calculated using the odds found on the back of the scratch ticket. 

When it comes to lottery operations, it is critical that all parties are confident with the process. For everyone involved, including players, to feel confident, those running the lottery operations must acquire and uphold a secure environment that is documented and accessible. To address this, the Security Control Standard was put in place by the World Lottery Association and lottery organisations are audited against this standard on a regular basis. 

Race and sports gaming

Race and sports wagering is also called sports betting. It is the activity of predicting sports results and placing a wager on the outcome. Although most sports betting wagers are placed against amateur and professional level sports, sports betting is sometimes extended to non-athletic events such as reality show contests and political elections, or sometimes to non-human athletics such as horse racing and greyhound racing. 

Sports betting can be performed at the sports betting outlet in a casino, with bookmakers (also know as a sports-book) or online through a computer or mobile device. The types of sports bets include: 

  • Money-line Bet
  • Spread Betting
  • Proposition Bet
  • Over / Under Bet
  • Parlay
  • Progressive Parlay
  • Future Wager

Money-line bets (also known as win bets) are bets in sports wagering. It is one of the most popular wagers that can be placed and is easy to understand. It is used in almost every sport a player can bet on and is a wager on who the player thinks is going to win a match, game or other event. It does not have a spread or handicap (explained below). It should be noted that the predicted winner, i.e., the competitor expected to win, pays lower odds then does an underdog. 

Spread betting is defined as wagers that are made against the spread. The spread is a number assigned by the bookmaker which handicaps one team and favours another. This type of betting is similar to the Money-line win, in that the player is choosing which team he/she thinks will win, but there is a significant difference. A point spread is created to effectively make the two teams equal favourites in terms of betting. This means the player either backs the favourite to win by at least the size of the spread, or the player backs the underdog to win or lose by no more than the size of the spread. For example, the odds for this week’s National Football League games are posted and the point spread in the Washington Redskins versus Dallas Cowboys game looks like this: Dallas-4.5 Washington +4.5. The favourite team is associated with a minus (-) value, so Dallas is favoured by 4.5 points in this game. Consequently, the underdog is shown with a plus (+) value, which means Washington are 4.5-point underdogs. A wager on Dallas would be made if a player believe Dallas can win the game by 5 points or more. So, if Dallas wins the game 20-14, then the team not only wins by 6 points but also covers the 4.5-point spread as the favourite. However, if Dallas wins the game 20-17, then they win by 3 points and have NOT covered the 4.5 points, but Washington has because they stayed within the spread. 

Proposition bets (also known as Props or Specials) are wagers made on events that are not related to the final outcome. Example events are: who will win the first round of a boxing match or which team will score first in a match. 

Over/Under bets (also known as Totals) are wagers made on whether an outcome will be under or over an estimated outcome set by the bookmaker. For example, how many three-point shots will LeBron James make tonight?

– Over 2.5
– Under 2.5

In this example, notice how the Prop takes the form of a traditional game total wager. This is a simple wager to understand – if the person making the wager thinks that LeBron James can make three or more three-point shots tonight, bet on the over. If the player making the wager thinks LeBron cannot do that, take the under. 

There are specific odds for both the over and under bet. Payments depend on the odds at the time the bet is made. 

Parlays (also known as accumulators) involve multiple bets and rewards a successful player with a large payout. These types of bets are hard to predict because they involve making more than one selection as part of a single wager. For example, the player might place a single wager on what team will win the next five football matches. If the player successfully wagers, the payout is substantially higher than if the player had wagered on each game separately. The downside is that the player would lose his/her complete wager if the team he/she selected lost any one of the five games. Based on the number of selections, the parlay can receive a unique name. For example, “Double” when it contains two games, or “Treble” when it is composed of three games. 

Progressive Parlays are similar to parlays in that they involve making more than one selection as part of a single wager. However, they differ from a Parlay in that a player will be rewarded even if some of the bets lose. If all bets are won, the player will be awarded the full payout which is not as large as a regular parlay but will receive a reduced payout if some of the selections within the parlay lose. 

Future Wagers (also known as Outright wagers) are wagers placed on future events. Although all sports wagers are on future events, with a future bet, there is a long-term horizon measured in weeks or months. Future wagers usually are made before the season starts. Winning bets will not pay off until the end of the season. For example, the player might make a futures wager on a team winning the National Hockey League (NHL) Stanley Cup. The wager must be placed before the regular NHL season begins and the payoff will not be made until after the Stanley Cup playoffs end. 

Online and mobile gaming

Online gaming includes all areas of gaming offered via Internet, mobile, wireless in-venue, and interactive-TV channels. The online gaming space contains all the different types of gaming that have been discussed thus far, i.e., slot games, table games, lottery, and sports betting 

Online gaming has become one of the most popular and lucrative businesses present on the internet. Legalisation of online gaming varies based on the type of online gaming product and the jurisdictions in which they are offered. For example, purchasing traditional lotto tickets through online websites is legal in many jurisdictions. However, not all jurisdictions have legalised casino style gaming such as poker or slot games through online gaming websites. 

Mobile gaming is online gaming on a mobile device such as a tablet or smart phone. There are two types of mobile gaming. The first is the online gaming at casino websites that can be accessed through a mobile device either through a website or through a mobile app. The second is in-venue mobile gaming which allows on premise casinos to add mobile technology and content to their existing offerings. Products are accessible to players on the gaming machines on the casino floor and on mobile devices inside the casino. 

For the online and mobile gaming ecosystem, the player needs to be able to access the casino’s online gaming products. This can be done in two ways: 

  • Browser-based
  • Downloadable application

If the player chooses to play through a browser-based casino website, the games are available through the player’s browser while on the online casino’s website. 

If the player chooses to play through a downloadable application, he/she must first install the online casino’s software to his/her computer or mobile device. This option usually offers better graphics, sound and game play than the browser-based option. Then, in order to play at the online casino, the player must have a means of transferring money to and from the online casino. This can be accomplished by an electronic wallet (also known as a digital wallet), such as PayPal. When performing mobile in-venue gaming, some casinos have internal electronic wallets as part of the casino management system which are often associated to a player’s account. In this scenario, the player would deposit funds into or withdraw funds from the casino’s electronic wallet solution at the cashier booth.

To ensure online or mobile gaming is performed only where it is legal, geolocation, micro-technology and triangulation are used to confirm the location of the player. Geolocation is the estimation of the real-world geographic location of an object, i.e., the computer or mobile device a player is using to play online gaming. Micro-location technology is used for in-venue mobile gaming. This technology works by using the casino’s existing WIFI network or Bluetooth beacons to give accuracy of a player’s location to within a few feet. For out-of-venue online gaming, some jurisdictions have decided on mobile phone triangulation to confirm the location of players. This triangulation method determines which cellular towers are closest to the player’s mobile phone and ensures that the player is in the right geographical location. Mobile phone triangulation technology is accurate to within a mile of where the client resides. Other jurisdictions have decided to use Wi-Fi to verify geolocation for out-of-venue online gaming. This geolocation technology is accurate to within a few feet of the user’s residence. 

Individuals looking to circumvent restricting online gaming to specific locations use technical measures such as proxy servers to try to bypass restrictions imposed by geolocation software. Some online gaming sites can detect the use of proxies and anonymises and block their access to the online gaming systems.

Key Concepts in the Gaming Industry

Progressive jackpots

A progressive jackpot is a prize or payout which increases each time the game is played but the jackpot is not won. A small percentage of each wager placed by a player on the game contributes to the jackpot award amount. The game that the progressive jackpot is attached to can be any type of game (e.g., mechanical reels, poker, etc.). 

When the progressive jackpot is won, the jackpot for the next play is reset to a predetermined value, and resumes increasing under the same conditions. The progressive jackpot win is often associated with the highest winning combination on the gaming machine in which it is being played. In order to win the progressive jackpot, in most games, the player needs to have placed a maximum bet as the wager for the play. 

Progressive jackpots are available both on VLTs and slot machines. There are three types of progressive jackpots: 

  • Standalone progressive
  • Local area linked progressive
  • Multi-site linked progressive

A standalone progressive has a jackpot on the individual EGM. Only bets placed on that specific EGM increment the jackpot.
Local area linked progressives are games within a venue that are linked together to contribute to a common progressive jackpot. This type of jackpot is usually found in a casino. This type of network can include as few as a dozen EMGs and as many as hundreds of these.
Multi-site (also known as Wide Area) linked progressives link gaming machines from multiple venues to participate in the progressive jackpot. Due to jurisdictional rules being different, Multi-site linked progressives usually only link machines within the same jurisdiction, often across casinos operated by the same organisation. However, some examples of multi-jurisdiction progressive jackpots exist. For example, in July 2006, the Multi-State Lottery Association in the US introduced the first multi-jurisdictional progressive jackpot called Ca$hola. This progressive jackpot linked EGMs at nine lottery run casinos; three in Delaware, two in Rhode Island, and four in West Virginia. This linked progressive was replaced in 2011 by the Megabits jackpot and now includes two additional states: Ohio and Maryland.

A linked progressive jackpot solution adds some additional devices to VLT and slot machine ecosystems: 

  • A progressive jackpot display or sign
  • A progressive jackpot controller
  • A progressive jackpot server 

The progressive jackpot display or sign is used to display the current amount of the progressive jackpot. 

The progressive jackpot controller is used by the venue to manage the progressive jackpot. The jackpot controller links the games contributing to the progressive jackpot and communicates the jackpot value to the progressive jackpot display.

The progressive jackpot server is used to manage multiple jackpot controllers and different progressive jackpot games that may exist across a venue. It will also monitor and collect all progressive related data to allow for analytics and auditing of progressive jackpots.

Random Number Generator (RNG)

The Random Number Generator is a computational or physical device designed to generate a sequence of numbers that lack any pattern, so they are random, or they appear unrelated. RNGs are used in gaming, statistical sampling, computer simulations and other areas where producing an unpredictable result is desirable. Any machine-base gaming involves an RNG. 

The RNG is a vital part of all gaming machine operations. Where unpredictability is essential, such as in security applications, hardware generators are generally preferred over pseudo-random algorithms. 

The RNG is certified by either an ITL or by the jurisdiction’s regulatory board. 

The win selection flow

The selection process or the “did I win?” process is another key concept of the gaming industry. All gaming machines such as EGMs use some type of win selection process to determine and display the outcome of the game. This means if the player pulls a lever or presses a button, something happens on the screen and then there’s an outcome that says “Yeah! I’ve won!” or” No, I’ve lost!”. 

What is also important about the selection process is that it can be performed on the EGM itself or on a server. In some cases, the whole process from “spin the wheel”, “get a response”, “you won or lost” is done on a standalone EGM. 

The technology being used and the specific jurisdictional rules of where the game is being played will influence the selection process and whether it is performed on the EGM or on the servers.

This selection process will involve the following: 

  1. Start of spin
  2. A raw random number is generated by the RNG
  3. The raw random number is scaled to a usable number
  4. The number is mapped to a game element (e.g., is it a star? is it a 7? is it Wheel of Fortune?)
  5. There is an evaluation of the outcome of the results of that random number generation
  6. The prize is awarded to the player with that outcome. Either credits are taken away from the player in the case of a loss or credits are given because of a win.
  7. There is a display of the outcome to the player
  8. The prize is paid, if applicable
  9. End of spin

Player privacy and geolocation

Privacy laws in most jurisdictions mandate that any player’s information being tracked, whether for responsible gaming or player loyalty program purposes, adheres to the storage and use of personal information regulations set forth by those laws. An example of testing player privacy is verifying that the solution makes the player information available to only those that should have access, and that any such information is encrypted when being transferred between devices and systems. 

Some responsible gaming and player loyalty programs require knowing where the player is located. Testing this function consists of ensuring the geolocation functions accurately restrict play based on the rules mandated by the location from which the player is playing. 

Regulatory commissions, jurisdictions and associations

Compliance testing is also called jurisdictional testing. Each jurisdiction has its own rules, regulations, guidelines (also known as regulatory or jurisdictional specifications or rules) that must be tested. This testing is usually performed by an ITL. 

In the United States, there are over four hundred regulators and jurisdictions. Canada has at least one per province. South America has at least one jurisdiction per country that has legalised gaming. Europe, Asia and Africa also usually have one jurisdiction per country. Germany has lottery companies by province. Australia has at least one per state. Within these jurisdictions, there is usually an organisation that is responsible for issuing licenses and regulating the licensee or the people who have the licensee. These organisations are typically known as licensing authorities. 

Every jurisdiction controls the potential manufactures who need a licensee to operate in that jurisdiction. Manufactures cannot legally operate in any jurisdiction where they do not have a licensee. If a product fails compliance testing, it must be fixed and returned to the ITL for certification testing until it passes 100% of the mandatory certification tests. The product can be returned many times before it passes the compliance tests. 

Before gaming products are ready for compliance testing, a full range of gaming QA testing must occur. Some examples of test types and test techniques that are done for the gaming industry includes exploratory testing, functional testing, regression testing, pre-compliance testing, system integration testing, performance testing, penetration testing and failover testing.

Gaming Industry Metrics

Background 

Gaming industry testing uses many of the common test metrics. However, there are a few that are specific to the gaming industry. 

First pass percentage

First pass percentage identifies the percentage of games that receive certification from the ITL on the first submission of the product. 

The importance of receiving a first pass for a gaming product is related to both product cost and its time-to-market. If the product does not receive a first pass, there are extra costs for additional development, testing and product certification. A gaming product that does not receive a first pass is delayed from entering the market until it is certified. 

Escape compliance defects

These metrics measure data relating too escaped defects that do not comply with the jurisdictional rules or regulations and are found by the ITL or in the field. 

The resubmit factor is the number of times a game must be resubmitted to the ITL to pass certification testing. For example, if on average each game is resubmitted 4.5 times to achieve certification, the resubmits factor would be equal to 4.5. 

The number of revocations tracks how many games have been pulled from the field per time period, due to escaped compliance defects. For example, if two games have been removed from the field in a year, that would mean two revocations for the year. If a jurisdiction asks for a game to be removed due to an escaped compliance defect the manufacturer has a limited amount of time to remove the game. 

These two metrics are important because having escaped defects in a jurisdiction can impact a manufacturer’s right to be in that jurisdiction, negatively impacting their brand, making them lose revenue if the EGM and table games is not working on the casino floor. There are a fixed number of EGMs and table games in any casino. Manufacturers fight for floor space amongst themselves, so a revocation might also mean that a manufacturer loses that floor space to a competitor. 

Gaming Software Development Lifecycle

The gaming software development lifecycle overview

The Gaming Software Development Lifecycle follows the sequential development model. 

Game Concept and Design is the first phase of the gaming software development lifecycle. It starts with a game idea that is storyboarded and is reviewed. Game and sound designers, artists, video, and gaming experts, software architects and game developers, and gaming jurisdictional experts create a game prototype. The prototype is then scrutinised for innovation and playability by the targeted audience focus group. This group may be composed of internal (IT-professionals), external (non-employees, sometimes non-IT professionals), or a mix of both resources. The Game Concept and Design phase is an iterative process. The Game Concept and Design phases’ ultimate deliverables are documents which become the blueprint for the development team, artists, mathematicians, and sound designers. 

The Game Concept and Design documents include the following: 

  • Game Concept 
  • Game and Technical Design

The Alpha phase, not to be confused with alpha testing, is next. During this phase, game play functionality is developed and implemented, math functionality is completed, video and audio components are partially finished, and the game contains the major features. Black-box testing, especially functional, usability testing, exploratory testing, regression testing, math testing, and RTP testing occur.

The Code Complete phase is next. All features, audio, video and math components are finalised. At this phase, code is no longer added to the game, unless a change is needed to fix defects. Standard black-and white-box testing are typically performed at this phase. The emphasis is on test automation, testing for memory leaks, confirmation testing, and regression testing.

The Beta Build phase, not to be confused with Beta Testing, continues until no failures occur that prevent the game from being certified. Pre-certification testing is performed by the internal gaming quality assurance test team to assess the game versus the requirements of each jurisdiction. This phase is not a formal certification test cycle. It is a precursor to the ITL certification testing. Any defects discovered at this time will be corrected and the new builds are tested, and regression tests are performed. 

The Release Build phase is the one that is sent to the ITLs to ensure that the game complies with the requirements of each required jurisdiction. This build receives the final certification sign-off, which allows the game to be sent to casinos or be made available online. If the game fails this certification, it is sent back to the game developer and the process starts over. 

The role of the independent test lab (ITL)

Once the pre-certification phase is completed by the machine manufacturer, the game is ready to be certified by an ITL (also known as the Authorised Test Facility). If this is a game that will be played in North America and in Australia, it must be tested for all applicable jurisdictions which means approximately 450 jurisdictions for these two parts of the world.

Once the ITL has tested the game for all applicable jurisdictions, if it fails in any of the jurisdictions, the game is returned to the machine manufacturer or game developer who make the changes in the game or in the EGM and return it for another ITL certification test.

The only way to be an accredited ITL is to be accepted by each gaming regulatory commission. This is a lengthy and costly process and thus there are only a few ITLs who can certify games world-wide. A few of the jurisdictions have government-based certification test labs that play the role of the ITL. 

The role of regulatory commissions

Once the ITL has certified a game, the regulatory commission allows the game to be played in all casinos in their jurisdiction. However, the regulatory commission will revoke or pull a game from all its casinos if a major field issue arises. A major field issue is usually a defect that stops the game from playing, provides erroneous payouts or deviates from any of the rules of engagement that are required for certification. The machine manufacturer will have to immediately remove that game from every installation in the jurisdiction.

There are also minor field issues that will force the machine manufacturer to modify a game that is in the field, within a given timeframe. In this case the game must be certified again at an ITL and approved by the regulatory commission. 

Acceptance Testing Introduction and Basics

Fundamental Relationships

While it is certainly true that the roles and responsibilities of the tester and the business analyst are different, it is also true that their activities are complementary; work done by one group may greatly affect, either positively or negatively, that of the other. This is especially true in acceptance testing which is performed to assess the system’s readiness for deployment and its use by the customer (end-user). Good collaboration between business analysts and testers is particularly important for a proper consideration of the business implications at this test level.

Business Goals, Business Needs and Requirements

Business analysts first must understand the organisation’s overall business goals and identify current business processes and stakeholders. Once that is done, they describe specific business needs and determine a business case that addresses those needs. Once this high-level work has been completed, requirements can be elicited for the business solution that shall be developed.

Business goals, business needs, business requirements, and product requirements describe, at different levels of abstraction, what shall be achieved. In Agile development, the same principles apply, but different terms may be used (for example features and user stories).

In this document, the term “requirements” refers both to business requirements and to product requirements.

Requirements / User Stories, Acceptance Criteria and Acceptance Tests

During requirements elicitation, business analysts and testers (possibly together with developers) should begin to create specific acceptance criteria and develop acceptance tests as a joint effort. This ensures that there is a mutual understanding of what “acceptable” means from the business, development, and testing perspectives, right from the beginning of the project.

Acceptance criteria relate directly to a specific requirement or user story. They are either part of the detailed description or an attribute of the related requirement. If user stories are used, acceptance criteria are part of the user story’s definition and extend the story. 

In all cases, acceptance criteria are measurable criteria, formulated as a statement (or a set of statements), which can be either true or false. They are used to check whether a requirement or user story has been implemented as expected. Acceptance criteria represent the test conditions which determine “what” to test. They do not contain the detailed test procedures. 

Acceptance test cases are derived from acceptance criteria. These tests specify how the verification of the acceptance criteria should be performed. 

The Importance of the Quality of the Requirements

If acceptance criteria and tests are based on requirements, user stories, and/or acceptance criteria that are vague or ambiguous, it is likely that testers will make assumptions about stakeholder expectations and business needs. In this case, the resulting acceptance tests may be flawed. This will lead to rework or, even worse, the running of invalid tests, thus creating unnecessary costs as well as risks and uncertainty about product quality assurance. 

It is critical for testers to work closely with business analysts to make sure that requirements are clear and well understood by all stakeholders concerned. Ambiguities should be resolved and assumptions should be clarified so that the resulting acceptance tests are valid and are a meaningful way to determine the product’s readiness for release. 

In Agile development, the INVEST criteria define a set of criteria, or checklist, to assess the quality of a user story. These may be used by business analysts / product owners, developers, and testers to ensure the quality of user stories.

Business Analysis and Acceptance Testing

Too often, business analysts and testers work in their own separate silos, which can lead to misunderstandings about business and customer expectations. Those misunderstandings may stay hidden until the release approaches. By taking advantage of the complementary skills and by working together, business analysts and testers can positively affect the development process. This can be accomplished both by considering acceptance criteria and acceptance testing as early as possible and by coordinating efforts to make sure that the product has been tested appropriately prior to release at acceptance test level. 

Relationship between Business Analysis and Testing Activities

The following are the main elements of the business analysis activities: 

  • Strategy definition
  • Management of the business analysis processes
  • Requirements engineering in business analysis
  • Solution evaluation and optimisation

The business analyst is responsible for identifying business needs of stakeholders and for determining solutions to business problems with the aim of introducing change which adds value to the business. An important aspect of the business analyst’s role is to establish consensus between quality engineers, testers, developers, system integrators, product managers and project managers.

A test process consists of the following main groups of activities:

  • Test planning
  • Test monitoring and control
  • Test analysis
  • Test design
  • Test implementation
  • Test execution
  • Test completion

Quite a few of the associated activities and tasks relate to both business analysis and testing. The following examples illustrate the relationship between the two disciplines in the context of acceptance testing:

Requirements engineering in business analysis vs. test planning, test analysis and test design:

  • During the requirements engineering activities in business analysis, business analysts prepare detailed business and product requirements. These requirements are part of the test basis for the test planning, test analysis and test design activities, as testers define their objectives and plan their work, evaluate the specifications and requirements, identify test conditions and design test cases and test procedures.
  • Testers can contribute to the definition and verification of acceptance criteria as part of test analysis and test design activities. Working together, the two roles ascertain that there is proper understanding of the solution and agree on the appropriate approach to acceptance testing.
  • When requirements change, business analysts and testers can work together to assess the impact of the changes.

Solution evaluation in business analysis vs. test implementation, test execution and test completion:

  • During the solution evaluation phase in business analysis, business analysts support test implementation and test execution activities. They review the testers’ procedures/scripts, clarify issues and potentially help with creation of test data to support business-related tests.
  • Business analysts can assist with the implementation and execution of the acceptance tests. They may also support testers by evaluating test results. In addition, they may assist testers in test completion activities.

There is a strong and symbiotic relationship between the two roles and their respective activities, starting at the very beginning of a project and continuing until acceptance or release of the solution.

Collaboration between Business Analysts and Testers in Acceptance Testing

The common goal for business analysts and testers is to support the production of products with the highest possible value for the customer. Given their position within the organisation, business analysts and testers have various opportunities to collaborate during the acceptance testing activities described in the previous section. Apart from joint discussions and reviews of generated artefacts, business analysts and testers collaborate in other areas. For example, collaboration on test planning based on risk analysis is a good opportunity to ensure that the appropriate test cases will be developed and prioritised.

In addition to the direct benefits of working together and supporting each other’s efforts during acceptance testing, there is an important opportunity to cross-train team members. The more testers know about business needs and stakeholder requirements, and the more business analysts know about structured testing, the more likely the two groups will understand and appreciate each other’s work and better collaborate within the project.

How Acceptance Testing Can Drive the Development Process: ATDD and BDD

The wide acceptance of Agile software development practices has influenced how acceptance testing relates to requirements elicitation and other business analysis activities. In sequential lifecycle models, acceptance test analysis, design, and implementation are activities to be handled by the testers after the requirements are finalised. With the Agile lifecycle model, acceptance criteria and acceptance test cases are created during requirements analysis, requirements refinement sessions, and product backlog refinement. This allows the implementation of the “Early Testing” principle by using the design of test cases as part of the requirements definition activities. 

In the following two approaches, acceptance test analysis and design are formally part of the requirements engineering process: 

  • In Acceptance Test-Driven Development (ATDD), acceptance tests are produced collaboratively during requirements analysis by business analysts, product owners, testers and developers.
  • Behaviour-Driven Development (BDD) uses a domain-specific scripting language, Gherkin, that is based on natural language statements. The requirements are defined in a ‘Given – When – Then’ format. These requirements become the acceptance test cases and also serve as the basis for test automation.
  • Both of these approaches engage the entire Agile team and help to focus the development efforts on the business goals. The approaches also treat the acceptance test cases as living documentation of the product because they can be read and understood by business analysts and other stakeholders. Acceptance test cases represent scenarios of usage of the product.
  • The two approaches are similar and the two terms are sometimes used interchangeably. In practice, BDD is associated with the use of Gherkin to support writing acceptance tests, while ATDD relies on different forms of textual or graphic acceptance test design. For example, the graphical representation of application workflows may be used to implement a visual ATDD approach.