Objectives and Overview
Understand the objective of the gaming tester
Being a gaming industry tester means that you must understand both testing in general, and the unique set of skills for the gaming industry ecosystem. This ecosystem is filled with proprietary, complex, multifaceted gaming software, hardware, platforms, firmware and operating systems. The objective of this article is to provide the regular Tester graduates with the specific knowledge that is required for a career in gaming industry testing.
Why the gaming industry requires a specialist tester
Some of the specific testing for the gaming industry, not present in other testing areas, include the following:
1. Gaming industry ecosystem – The unique hardware, firmware and operating systems that are proprietary to the gaming industry.
2. Gaming industry compliance testing – There are over 440 different certification boards worldwide for gaming industry games. These boards have rules that games in the gaming industry must comply with. These rules impact hardware, software, platform, operating systems, visual and auditory functionality, mathematics, and return to player (RTP) calculations. One gaming industry game can be played in multiple gaming jurisdictions and needs to comply with the laws of each location.
3. Fun factor or player perspective testing – This is something unique to gaming industry games, since they are an entertainment product. Not only are casino games supposed to work intuitively and provide the player pleasure, they must also be fun to play. This requires a unique insight into game design, with experience and information about the user group and what that group enjoys.
4. Math testing – Testing the multitude of pay tables, permutations, Random Number Generator (RNG) results and RTP computations. This type of testing requires the tester to understand what triggers different types of payout behaviour and to understand financial return to the player and how these triggers can be treated by different parameters. Understanding math testing is critical to succeed in this field.
5. Audio testing – Creating sound or playing media is common in software. However, gaming industry game music must engage the user in the game and enhance the game play. Not only should the audio play without stuttering or missing elements, it should also add to the game play. This requires extensive audio skills and specific understanding of game audio.
6. Multiplayer testing – This type of testing is performed when many players are simultaneously interacting with casino games, with computer-controlled opponents, with game servers, and with each other. Typical risk-based testing is followed to ensure against using unlimited amounts of time testing different scenarios. Understanding multiplayer game design, and how to test it efficiently, is required knowledge for this type of testing.
7. Interoperability Testing is common in all software that communicates with other software, systems and/or components. Casino/Video Lottery games have a unique aspect in that they must implement interoperability using gaming industry open protocol standards or proprietary protocols as per the specifications of the central server deployed in the jurisdiction to which the game is deployed.
Gaming Activities and Artefacts
Background
To understand gaming industry testing and its ecosystem specificities requires a review of the business model, activities, and artefacts as they pertain to the gaming industry.
What is gaming?
Gaming can be defined as follows:
- The wagering of money or something of value, also called stakes, on an event
- Where the outcome of the event is unknown
- Where the whole intent is winning additional money, material goods or trips
What is a gaming machine? A gaming machine is a machine that enables the wagering of money or something of value. Examples of gaming machines are: electronic or mechanical slot machines, a roulette table or even a computer for online gaming.
Types of Gaming
Casino games
There are three categories of casino games: table games, electronic gaming machines (EGMs) and random number ticket games.
Examples of table games are roulette, blackjack, baccarat or poker, which typically are not tested unless they are an electronic table game version of these games.
The second group are EGMs, typically known as video lottery terminals (VLTs) or slot machines. These are usually played by one player at a time and do not require the involvement of casino employees to play. These games need to be tested, i.e., the game software, the machines, the operating systems, and platforms that they are based on.
VLTs and slot machines are both gaming machines that allow players to bet on the outcome of a game. Physically, VLTs and slot machines are very similar in nature. The main difference between a VLT and a slot machine is that VLTs are gaming machines that are operated by government lotteries while slot machines are gaming machines operated by private organisations such as casinos.
Both VLTs and slot machines are regulated and require licenses to be operated within their jurisdictions. Many countries around the world offer legalised VLT or slot play. For example:
- In the United States, a 1988 federal law established three classes of games for Native American casinos, with different regulatory schemes for each. Each state government follows variations of these classes to define their regulations.
- In Canada, the provincial or territorial governments are responsible for regulating gaming operations. All provinces offer the ability to play, each with their own regulations.
- In Australia, the laws regulating the use of gaming machines are the responsibility of the state governments.
Other terms by which a VLT and slot machine are referred to: EGM, Video Gaming Terminal, Video Gaming Device, Video Slot Machines and Interactive Video Terminal.
The third casino game category is random number ticket games such as Keno and simulated racing. These games are based on the selection of random numbers, either from a computerised RNG or from other gaming equipment.
Lottery systems
A lottery is a form of gaming that involves selling numbered tickets and giving prizes to holders of winning tickets. The prize can be a fixed amount of cash or goods, but more commonly, the prize fund is a fixed percentage of the revenues from the tickets sold.
There are typically two-forms of lottery products sold: traditional lottery tickets and instant tickets.
Traditional lottery tickets are numbered tickets that are sold for regularly scheduled draws, most often weekly. On the draw date, random numbers are drawn either using a ball drop machine or electronically. Most lotteries that have moved to electronic draws still have ball drop machines as a backup in case of failures with the software solution. Once the numbers are drawn from the ball drop machine, they are entered into the lottery central management system (CMS).
The chances of winning a lottery jackpot can vary widely depending on the lottery design, and are determined by several factors, including:
- The count of possible numbers
- The count of winning numbers drawn
- Whether or not the order is significant
- Whether drawn numbers are returned for the possibility of further drawing
Instant tickets are numbered tickets from a pre-determined finite pool of outcomes. The most common form of instant tickets is the scratch card. Scratch cards are typically made of paper, with the outcome printed and hidden by an opaque substance that needs to be scratched off, hence the name of these tickets. The cards usually present the information in the form of a game, such as Tic-Tac-Toe, Bingo, Crossword or some other puzzle, to help add entertainment value. A variation of the scratch card is the break-open (also know as pull-tab) ticket in which, instead of scratching off an opaque substance to reveal the outcome, the player opens a perforated cardboard cover which is hiding the outcome. Since outcomes of scratch and break-open tickets are pre-determined, the cards do not need to be scratched or opened to be validated.
A barcode on the ticket can be scanned by the lottery CMS to determine if it is a winner or not. The scratching or breaking open is there for entertainment value to the player only.
The chances of winning on a scratch card are typically much higher than on a traditional lottery, but prize amounts are typically much smaller. The probability of winning on a scratch card can be calculated using the odds found on the back of the scratch ticket.
When it comes to lottery operations, it is critical that all parties are confident with the process. For everyone involved, including players, to feel confident, those running the lottery operations must acquire and uphold a secure environment that is documented and accessible. To address this, the Security Control Standard was put in place by the World Lottery Association and lottery organisations are audited against this standard on a regular basis.
Race and sports gaming
Race and sports wagering is also called sports betting. It is the activity of predicting sports results and placing a wager on the outcome. Although most sports betting wagers are placed against amateur and professional level sports, sports betting is sometimes extended to non-athletic events such as reality show contests and political elections, or sometimes to non-human athletics such as horse racing and greyhound racing.
Sports betting can be performed at the sports betting outlet in a casino, with bookmakers (also know as a sports-book) or online through a computer or mobile device. The types of sports bets include:
- Money-line Bet
- Spread Betting
- Proposition Bet
- Over / Under Bet
- Parlay
- Progressive Parlay
- Future Wager
Money-line bets (also known as win bets) are bets in sports wagering. It is one of the most popular wagers that can be placed and is easy to understand. It is used in almost every sport a player can bet on and is a wager on who the player thinks is going to win a match, game or other event. It does not have a spread or handicap (explained below). It should be noted that the predicted winner, i.e., the competitor expected to win, pays lower odds then does an underdog.
Spread betting is defined as wagers that are made against the spread. The spread is a number assigned by the bookmaker which handicaps one team and favours another. This type of betting is similar to the Money-line win, in that the player is choosing which team he/she thinks will win, but there is a significant difference. A point spread is created to effectively make the two teams equal favourites in terms of betting. This means the player either backs the favourite to win by at least the size of the spread, or the player backs the underdog to win or lose by no more than the size of the spread. For example, the odds for this week’s National Football League games are posted and the point spread in the Washington Redskins versus Dallas Cowboys game looks like this: Dallas-4.5 Washington +4.5. The favourite team is associated with a minus (-) value, so Dallas is favoured by 4.5 points in this game. Consequently, the underdog is shown with a plus (+) value, which means Washington are 4.5-point underdogs. A wager on Dallas would be made if a player believe Dallas can win the game by 5 points or more. So, if Dallas wins the game 20-14, then the team not only wins by 6 points but also covers the 4.5-point spread as the favourite. However, if Dallas wins the game 20-17, then they win by 3 points and have NOT covered the 4.5 points, but Washington has because they stayed within the spread.
Proposition bets (also known as Props or Specials) are wagers made on events that are not related to the final outcome. Example events are: who will win the first round of a boxing match or which team will score first in a match.
Over/Under bets (also known as Totals) are wagers made on whether an outcome will be under or over an estimated outcome set by the bookmaker. For example, how many three-point shots will LeBron James make tonight?
– Over 2.5
– Under 2.5
In this example, notice how the Prop takes the form of a traditional game total wager. This is a simple wager to understand – if the person making the wager thinks that LeBron James can make three or more three-point shots tonight, bet on the over. If the player making the wager thinks LeBron cannot do that, take the under.
There are specific odds for both the over and under bet. Payments depend on the odds at the time the bet is made.
Parlays (also known as accumulators) involve multiple bets and rewards a successful player with a large payout. These types of bets are hard to predict because they involve making more than one selection as part of a single wager. For example, the player might place a single wager on what team will win the next five football matches. If the player successfully wagers, the payout is substantially higher than if the player had wagered on each game separately. The downside is that the player would lose his/her complete wager if the team he/she selected lost any one of the five games. Based on the number of selections, the parlay can receive a unique name. For example, “Double” when it contains two games, or “Treble” when it is composed of three games.
Progressive Parlays are similar to parlays in that they involve making more than one selection as part of a single wager. However, they differ from a Parlay in that a player will be rewarded even if some of the bets lose. If all bets are won, the player will be awarded the full payout which is not as large as a regular parlay but will receive a reduced payout if some of the selections within the parlay lose.
Future Wagers (also known as Outright wagers) are wagers placed on future events. Although all sports wagers are on future events, with a future bet, there is a long-term horizon measured in weeks or months. Future wagers usually are made before the season starts. Winning bets will not pay off until the end of the season. For example, the player might make a futures wager on a team winning the National Hockey League (NHL) Stanley Cup. The wager must be placed before the regular NHL season begins and the payoff will not be made until after the Stanley Cup playoffs end.
Online and mobile gaming
Online gaming includes all areas of gaming offered via Internet, mobile, wireless in-venue, and interactive-TV channels. The online gaming space contains all the different types of gaming that have been discussed thus far, i.e., slot games, table games, lottery, and sports betting
Online gaming has become one of the most popular and lucrative businesses present on the internet. Legalisation of online gaming varies based on the type of online gaming product and the jurisdictions in which they are offered. For example, purchasing traditional lotto tickets through online websites is legal in many jurisdictions. However, not all jurisdictions have legalised casino style gaming such as poker or slot games through online gaming websites.
Mobile gaming is online gaming on a mobile device such as a tablet or smart phone. There are two types of mobile gaming. The first is the online gaming at casino websites that can be accessed through a mobile device either through a website or through a mobile app. The second is in-venue mobile gaming which allows on premise casinos to add mobile technology and content to their existing offerings. Products are accessible to players on the gaming machines on the casino floor and on mobile devices inside the casino.
For the online and mobile gaming ecosystem, the player needs to be able to access the casino’s online gaming products. This can be done in two ways:
- Browser-based
- Downloadable application
If the player chooses to play through a browser-based casino website, the games are available through the player’s browser while on the online casino’s website.
If the player chooses to play through a downloadable application, he/she must first install the online casino’s software to his/her computer or mobile device. This option usually offers better graphics, sound and game play than the browser-based option. Then, in order to play at the online casino, the player must have a means of transferring money to and from the online casino. This can be accomplished by an electronic wallet (also known as a digital wallet), such as PayPal. When performing mobile in-venue gaming, some casinos have internal electronic wallets as part of the casino management system which are often associated to a player’s account. In this scenario, the player would deposit funds into or withdraw funds from the casino’s electronic wallet solution at the cashier booth.
To ensure online or mobile gaming is performed only where it is legal, geolocation, micro-technology and triangulation are used to confirm the location of the player. Geolocation is the estimation of the real-world geographic location of an object, i.e., the computer or mobile device a player is using to play online gaming. Micro-location technology is used for in-venue mobile gaming. This technology works by using the casino’s existing WIFI network or Bluetooth beacons to give accuracy of a player’s location to within a few feet. For out-of-venue online gaming, some jurisdictions have decided on mobile phone triangulation to confirm the location of players. This triangulation method determines which cellular towers are closest to the player’s mobile phone and ensures that the player is in the right geographical location. Mobile phone triangulation technology is accurate to within a mile of where the client resides. Other jurisdictions have decided to use Wi-Fi to verify geolocation for out-of-venue online gaming. This geolocation technology is accurate to within a few feet of the user’s residence.
Individuals looking to circumvent restricting online gaming to specific locations use technical measures such as proxy servers to try to bypass restrictions imposed by geolocation software. Some online gaming sites can detect the use of proxies and anonymises and block their access to the online gaming systems.
Key Concepts in the Gaming Industry
Progressive jackpots
A progressive jackpot is a prize or payout which increases each time the game is played but the jackpot is not won. A small percentage of each wager placed by a player on the game contributes to the jackpot award amount. The game that the progressive jackpot is attached to can be any type of game (e.g., mechanical reels, poker, etc.).
When the progressive jackpot is won, the jackpot for the next play is reset to a predetermined value, and resumes increasing under the same conditions. The progressive jackpot win is often associated with the highest winning combination on the gaming machine in which it is being played. In order to win the progressive jackpot, in most games, the player needs to have placed a maximum bet as the wager for the play.
Progressive jackpots are available both on VLTs and slot machines. There are three types of progressive jackpots:
- Standalone progressive
- Local area linked progressive
- Multi-site linked progressive
A standalone progressive has a jackpot on the individual EGM. Only bets placed on that specific EGM increment the jackpot.
Local area linked progressives are games within a venue that are linked together to contribute to a common progressive jackpot. This type of jackpot is usually found in a casino. This type of network can include as few as a dozen EMGs and as many as hundreds of these.
Multi-site (also known as Wide Area) linked progressives link gaming machines from multiple venues to participate in the progressive jackpot. Due to jurisdictional rules being different, Multi-site linked progressives usually only link machines within the same jurisdiction, often across casinos operated by the same organisation. However, some examples of multi-jurisdiction progressive jackpots exist. For example, in July 2006, the Multi-State Lottery Association in the US introduced the first multi-jurisdictional progressive jackpot called Ca$hola. This progressive jackpot linked EGMs at nine lottery run casinos; three in Delaware, two in Rhode Island, and four in West Virginia. This linked progressive was replaced in 2011 by the Megabits jackpot and now includes two additional states: Ohio and Maryland.
A linked progressive jackpot solution adds some additional devices to VLT and slot machine ecosystems:
- A progressive jackpot display or sign
- A progressive jackpot controller
- A progressive jackpot server
The progressive jackpot display or sign is used to display the current amount of the progressive jackpot.
The progressive jackpot controller is used by the venue to manage the progressive jackpot. The jackpot controller links the games contributing to the progressive jackpot and communicates the jackpot value to the progressive jackpot display.
The progressive jackpot server is used to manage multiple jackpot controllers and different progressive jackpot games that may exist across a venue. It will also monitor and collect all progressive related data to allow for analytics and auditing of progressive jackpots.
Random Number Generator (RNG)
The Random Number Generator is a computational or physical device designed to generate a sequence of numbers that lack any pattern, so they are random, or they appear unrelated. RNGs are used in gaming, statistical sampling, computer simulations and other areas where producing an unpredictable result is desirable. Any machine-base gaming involves an RNG.
The RNG is a vital part of all gaming machine operations. Where unpredictability is essential, such as in security applications, hardware generators are generally preferred over pseudo-random algorithms.
The RNG is certified by either an ITL or by the jurisdiction’s regulatory board.
The win selection flow
The selection process or the “did I win?” process is another key concept of the gaming industry. All gaming machines such as EGMs use some type of win selection process to determine and display the outcome of the game. This means if the player pulls a lever or presses a button, something happens on the screen and then there’s an outcome that says “Yeah! I’ve won!” or” No, I’ve lost!”.
What is also important about the selection process is that it can be performed on the EGM itself or on a server. In some cases, the whole process from “spin the wheel”, “get a response”, “you won or lost” is done on a standalone EGM.
The technology being used and the specific jurisdictional rules of where the game is being played will influence the selection process and whether it is performed on the EGM or on the servers.
This selection process will involve the following:
- Start of spin
- A raw random number is generated by the RNG
- The raw random number is scaled to a usable number
- The number is mapped to a game element (e.g., is it a star? is it a 7? is it Wheel of Fortune?)
- There is an evaluation of the outcome of the results of that random number generation
- The prize is awarded to the player with that outcome. Either credits are taken away from the player in the case of a loss or credits are given because of a win.
- There is a display of the outcome to the player
- The prize is paid, if applicable
- End of spin
Player privacy and geolocation
Privacy laws in most jurisdictions mandate that any player’s information being tracked, whether for responsible gaming or player loyalty program purposes, adheres to the storage and use of personal information regulations set forth by those laws. An example of testing player privacy is verifying that the solution makes the player information available to only those that should have access, and that any such information is encrypted when being transferred between devices and systems.
Some responsible gaming and player loyalty programs require knowing where the player is located. Testing this function consists of ensuring the geolocation functions accurately restrict play based on the rules mandated by the location from which the player is playing.
Regulatory commissions, jurisdictions and associations
Compliance testing is also called jurisdictional testing. Each jurisdiction has its own rules, regulations, guidelines (also known as regulatory or jurisdictional specifications or rules) that must be tested. This testing is usually performed by an ITL.
In the United States, there are over four hundred regulators and jurisdictions. Canada has at least one per province. South America has at least one jurisdiction per country that has legalised gaming. Europe, Asia and Africa also usually have one jurisdiction per country. Germany has lottery companies by province. Australia has at least one per state. Within these jurisdictions, there is usually an organisation that is responsible for issuing licenses and regulating the licensee or the people who have the licensee. These organisations are typically known as licensing authorities.
Every jurisdiction controls the potential manufactures who need a licensee to operate in that jurisdiction. Manufactures cannot legally operate in any jurisdiction where they do not have a licensee. If a product fails compliance testing, it must be fixed and returned to the ITL for certification testing until it passes 100% of the mandatory certification tests. The product can be returned many times before it passes the compliance tests.
Before gaming products are ready for compliance testing, a full range of gaming QA testing must occur. Some examples of test types and test techniques that are done for the gaming industry includes exploratory testing, functional testing, regression testing, pre-compliance testing, system integration testing, performance testing, penetration testing and failover testing.
Gaming Industry Metrics
Background
Gaming industry testing uses many of the common test metrics. However, there are a few that are specific to the gaming industry.
First pass percentage
First pass percentage identifies the percentage of games that receive certification from the ITL on the first submission of the product.
The importance of receiving a first pass for a gaming product is related to both product cost and its time-to-market. If the product does not receive a first pass, there are extra costs for additional development, testing and product certification. A gaming product that does not receive a first pass is delayed from entering the market until it is certified.
Escape compliance defects
These metrics measure data relating too escaped defects that do not comply with the jurisdictional rules or regulations and are found by the ITL or in the field.
The resubmit factor is the number of times a game must be resubmitted to the ITL to pass certification testing. For example, if on average each game is resubmitted 4.5 times to achieve certification, the resubmits factor would be equal to 4.5.
The number of revocations tracks how many games have been pulled from the field per time period, due to escaped compliance defects. For example, if two games have been removed from the field in a year, that would mean two revocations for the year. If a jurisdiction asks for a game to be removed due to an escaped compliance defect the manufacturer has a limited amount of time to remove the game.
These two metrics are important because having escaped defects in a jurisdiction can impact a manufacturer’s right to be in that jurisdiction, negatively impacting their brand, making them lose revenue if the EGM and table games is not working on the casino floor. There are a fixed number of EGMs and table games in any casino. Manufacturers fight for floor space amongst themselves, so a revocation might also mean that a manufacturer loses that floor space to a competitor.
Gaming Software Development Lifecycle
The gaming software development lifecycle overview
The Gaming Software Development Lifecycle follows the sequential development model.
Game Concept and Design is the first phase of the gaming software development lifecycle. It starts with a game idea that is storyboarded and is reviewed. Game and sound designers, artists, video, and gaming experts, software architects and game developers, and gaming jurisdictional experts create a game prototype. The prototype is then scrutinised for innovation and playability by the targeted audience focus group. This group may be composed of internal (IT-professionals), external (non-employees, sometimes non-IT professionals), or a mix of both resources. The Game Concept and Design phase is an iterative process. The Game Concept and Design phases’ ultimate deliverables are documents which become the blueprint for the development team, artists, mathematicians, and sound designers.
The Game Concept and Design documents include the following:
- Game Concept
- Game and Technical Design
The Alpha phase, not to be confused with alpha testing, is next. During this phase, game play functionality is developed and implemented, math functionality is completed, video and audio components are partially finished, and the game contains the major features. Black-box testing, especially functional, usability testing, exploratory testing, regression testing, math testing, and RTP testing occur.
The Code Complete phase is next. All features, audio, video and math components are finalised. At this phase, code is no longer added to the game, unless a change is needed to fix defects. Standard black-and white-box testing are typically performed at this phase. The emphasis is on test automation, testing for memory leaks, confirmation testing, and regression testing.
The Beta Build phase, not to be confused with Beta Testing, continues until no failures occur that prevent the game from being certified. Pre-certification testing is performed by the internal gaming quality assurance test team to assess the game versus the requirements of each jurisdiction. This phase is not a formal certification test cycle. It is a precursor to the ITL certification testing. Any defects discovered at this time will be corrected and the new builds are tested, and regression tests are performed.
The Release Build phase is the one that is sent to the ITLs to ensure that the game complies with the requirements of each required jurisdiction. This build receives the final certification sign-off, which allows the game to be sent to casinos or be made available online. If the game fails this certification, it is sent back to the game developer and the process starts over.
The role of the independent test lab (ITL)
Once the pre-certification phase is completed by the machine manufacturer, the game is ready to be certified by an ITL (also known as the Authorised Test Facility). If this is a game that will be played in North America and in Australia, it must be tested for all applicable jurisdictions which means approximately 450 jurisdictions for these two parts of the world.
Once the ITL has tested the game for all applicable jurisdictions, if it fails in any of the jurisdictions, the game is returned to the machine manufacturer or game developer who make the changes in the game or in the EGM and return it for another ITL certification test.
The only way to be an accredited ITL is to be accepted by each gaming regulatory commission. This is a lengthy and costly process and thus there are only a few ITLs who can certify games world-wide. A few of the jurisdictions have government-based certification test labs that play the role of the ITL.
The role of regulatory commissions
Once the ITL has certified a game, the regulatory commission allows the game to be played in all casinos in their jurisdiction. However, the regulatory commission will revoke or pull a game from all its casinos if a major field issue arises. A major field issue is usually a defect that stops the game from playing, provides erroneous payouts or deviates from any of the rules of engagement that are required for certification. The machine manufacturer will have to immediately remove that game from every installation in the jurisdiction.
There are also minor field issues that will force the machine manufacturer to modify a game that is in the field, within a given timeframe. In this case the game must be certified again at an ITL and approved by the regulatory commission.