Are you building a new website? Be sure to discuss the importance of performance early on and set targets. That way, you have a faster load speed from the beginning and don’t have to implement fixes later.
Step 1: test, step 2: test…
Are you seeing a pattern here? 😉 Testing is crucial! Before you launch, load and test your website multiple times to make sure you can handle the traffic of real site visitors. This is especially important for sites with complex hosting, such as load-balanced configuration.
Implement some “quick wins”
To be clear, there’s no “get fast quick” scheme for site load speeds. But there is a tried-and-true template that will put you ahead of the curve. That includes making use of modern image formats, enabling compression on the server via Gzip, and leveraging browser cache. Find some more low-hanging fruit here.
Careful of your images!
Good websites have great graphic content – but they also take into account how images impact load speed. You can improve image performance by considering file formats, image compression, and lazy loading.
Think of the mobile visitors
More and more people surf the web on their phone these days, which makes mobile-optimized sites a huge priority! Since mobile users tend to use slower, less stable Internet connections, Accelerated Mobile Pages (AMPs) are a great way to get them content faster.
Prioritize above-the-fold
First impressions matter – and your above-the-fold content can make or break them! Consider inline styling for above-the-fold, then loading your code in chunks. This type of asynchronous loading can create a faster perceived load time for the user.
Assess your external scripts
Third-party scripts are a great tool – but can make your website feel a little crowded. Assess the performance of external scripts on your site load speed, and replace or remove those that are negatively impacting user experience.
Improving your site load speed is always a brilliant idea!
Here’s why you should take a look at it now:
With every year that passes, web surfers are more impatient with slow-loading sites 🌊
That means once a site visitor lands on your site you have a very small window of time to convince them not to leave.
In fact, visitors form their first impression about your site within 50 milliseconds (that’s .05 seconds)! So you want to make sure to impress.
Fast facts about site speed:
📉 Page load times longer than 2 seconds result in a 50% drop in conversion rates.
📱 53% of mobile visits end when a page takes longer than 3 seconds to load.
😱 For every extra second of page load time, 10% of users will leave.
Don’t let it happen to you! Check out our partner-approved tips for improving your page load speed 👇
A fast website never goes out of style.
Since the advent of the World Wide Web, website creators have been able to count on one thing: The faster your page load speed, the better experience you provide site visitors 🎉
Our partners have experience improving their clients’ site speeds on the daily. Take it from the experts at leading in tech and more…
…setting up your site for success might take time at first, but it will save you time and grow your revenue for the long term! 📈
The expectations of you are probably lower than you think, because, hey, you’re brand new!
You’ll find a wealth of ready-made packages and libraries of code online to use at your disposal. Do your research and be sure to sense-check the quality, but don’t be afraid to use these resources to help you spend less time “reinventing in the wheel” and more time developing your skills and knowledge in other areas.
Which ties nicely with the next tip:
2 Make Google your friend
Seeking a solution online is often the most efficient first step towards a solution.
A great piece of advice is to “get good at Googling”. Someone has run into the same problem as you, you just need to find it. Once you’ve found it, try to understand the what, why and how before copying and pasting it. This is an opportunity to learn and develop your knowledge.
3 Be kind to yourself (and your team!)
It may sound cliché – and perhaps a little cheesy – but it’s important to be kind to yourself when starting out in your development career, as nobody becomes an award-winning developer overnight 🤷♀️
While it is sometimes easier said than done, don’t put too much pressure on yourself and make sure you allow yourself the time to learn, grow and most importantly, make mistakes!
And you will make mistakes – just remember that it’s solving these mistakes that will help you become a stronger developer. And try not to strive for perfection – aim to write clean, reusable and easy to read code in a timely manner.
And don’t forget to be kind to your team too and remember nobody comes to work to do a bad job. The key to a successful development team is helping and supporting each other. A happy team will always produce the best work – and it’s less likely to feel like a job!
Ouch – this one can be a tough one for some. It’s only natural that you don’t want to look ignorant but you must fight this urge and speak up.
If you don’t understand something or haven’t heard of a term or technology – ask. If you don’t, it’s a missed opportunity to learn and verify your understanding. Software development is a multifaceted industry, you can’t know everything and you’re not expected to, but you can always gain knowledge by speaking up.
2 Communication is key
This one might surprise you, but your communication skills are just as important as your software development skills. Take the time to practice writing – you’ll use it more in your job than you might think.
And get comfortable explaining what you do to non-developers. Especially in the world of consulting and cross-team projects, you’ll likely be communicating with people who don’t have the same technical background as you do.
Miscommunication is perhaps the biggest threat to any project. You need to be able to effectively communicate with other developers, project managers and clients. Clear, concise and timely written or verbal communication can go a long way. It might take some practice, but if you’re aware of this from the start, it will become a strong skill for you going forward!
3 Develop your project management skills
Similar to social skills and communication, you need to be able to communicate your progress on development tasks.
Tools like Trello, Jiraand Azure DevOps support developers in task management, planning and scheduling. These skills will help you when you’re fixing a bug or writing a new piece of functionality; breaking down a larger task into smaller pieces making it more manageable for you as well as making it easier to present an overview to your manager or other team members.
When starting out, create your own GitHub account where you can start adding your own projects and snippets of code as you go along. Not only is this a great place to build up a reference library of code, it also helps when showcasing your work to potential employers too.
You’ll find that when you’re interviewing for roles, most employers appreciate being able to see some code you have written.
2 It’s important to know what’s cooking now – and in the future
Keep yourself up to date with whatever develops within your field – it’s crucial to know what’s cooking.
Explore and try out different areas within web development and different technologies. If you want to work with web development, try working with one CMS and becoming an expert in that – e.g. sqaeb. It will help you get a better idea of where you want to focus on later.
In the long run, I think you need to pick a specific area and master it – and this also means keeping yourself updated on this particular area!
3 Be curious – learn from others
The support you can get from your colleagues, friends and the online developer communities (like us) is invaluable, and you should never be afraid to ask for help.
If you’re struggling with some code, the chances are that someone has struggled before you and has already solved your exact problem! By having the confidence to reach out to those around you or online, you’ll find solutions much more quickly, increasing your knowledge in the process.
Perks and benefits that save employees money in the long run are always a valuable addition to a paycheck. Addition being the keyword here.
Because no amount of pizza parties can supplement the 10% increase in salary that people could get at the other agency across the street. Except, that’s not the case, the statistics surrounding this, point in the exact opposite direction:
32% of people polled in the US would take a 10% pay cut to work at a company where they like the culture
58% of workers will stay at a lower-paying job if it means having a great boss
And 60% of workers would even take half of the potential paycheck if it meant working at a job they love
So if culture makes up for the differences in salary between your agency and the agency next door, how do you structure the salaries in your company to both attract and retain top talent?
Don’t buy stars, build them – Have a partnership with the local media and technical schools that provides internships and part-time positions for promising students. If you follow our onboarding tips and you build a functional onboarding program, after a couple of weeks, your time investment in onboarding them should already be paying you back. And in a few months? You might just have your hands on your newest superstar.
Have a clear progression path – be upfront and transparent with the salary structure. It will eventually become the biggest motivator for the employees in the lower tiers. If you split your progression path into layers where everyone gets paid the same, you can skip long management discussions like: ’’Is a Senior Backend Developer with 4 years of experience worth the same as a Senior Art Director with 5?’’ An example of how to structure your progression path could be:
Intern > unpaid, but gaining real-life skills and experience from an agency by working on real projects
Trainee > paid, part-time or full time; self-taught, certified or freshly graduated
Apprentice > Same credentials as a trainee, but with some successful commercial projects
Junior > Proven 1-3 years of experience with commercial projects
Senior > 3+ years of experience with commercial projects and proficientwith project management and delegating tasks
Management > If you’re doing linear progression, this step is simple. But if you want to do non-linear progression, it’s worth differentiating at management level. a. Senior members with multiple specializations and experience with managing teams b. Senior members with extra non-managerial responsibilities (product development, decision making, etc.)
Equity tier > Management whose investment with the company is substantial enough to warrant equity in the company
Promotions, raises and employees who feel undervalued – if you adopt the aforementioned salary structure, your employees should have a clear overview of where they fall and what they need to achieve to move up to the next salary level. But as it goes with highly ambitious people, you will always have individuals who take on more than their fair share of responsibility and then don’t feel adequately compensated. The answer should be obvious. If the employee performs above the set expectations, has the data to back it up, and asks for an increase in pay, they should get one. Sadly, when working with more than one person, it will never be that easy. Ben Horowitz summed it up the best in his class on Y combinator – how to start a startup.
A point he brings up is: If you give that employee a raise, will you give everyone else who is also performing well a raise as well? What about the employees who are performing just as well, but their personality prevents them from asking directly?
Apart from being approachable overall, managers and senior agency members can adopt these two methods to focus these conversations and help employees feel more valued and heard:
1. Monthly walk and talk:A manager and employee go for a half-hour walk outside of the office, talking about current projects, plans for future projects, the progress of the employee and any problems they might be having
2. Yearly progress conversation: Performance reviews are usually seen as a negative process because of the negative associations that people usually have with them. Walk and talks remove the need for quarterly performance reviews at a scary meeting room table.
But a walk and talk is not really the place to sign contracts and obsess over spreadsheets. So how about a yearly progress review, close to the end of the year, talking strictly about the employee’s progression path and salary?
That way, both current problems can be addressed from month to month, and larger issues or achievements can be accumulated over time.
When hearing the words ’’non-linear’’, if your mind immediately jumps towards video games, you already sort of get the point.
In a non-linear game progression system, you start at the same spot as every other player. But when you arrive at a crossroads, instead of going straight down the first path like you usually would, you get to choose if you want to go left, right, or even take a step back and see if you can get to your current position again, by taking another path. This progression helps you pick up new skills and new experiences that will make the path ahead much easier.
This is also how the current trend in career progression looks. Companies no longer expect people to stay in the same career path for decades, slowly working their way up the corporate ladder. This rings especially true for agencies, where skills from different career paths transfer almost seamlessly and complement each other with a broader outlook on the problems being solved.
As an example, if you have a frontend developer who discovered she likes designing more than she likes coding, you should give her a chance because:
She already knows the limitations that code can have on some designs
She can design with systems and reusable assets in mind
She can give better estimates on project length and the overall development time
If she wants to progress further into something like art direction, the added coding skills are always a plus when communicating to both clients and developers alike
If your agency has people who have invested in their craft to the point where they are considered experts, top talent, or masters, their progression will eventually hit a plateau.
And while just existing at the top and using your skills to their full potential is a fantastic feeling… ultimately, the need for self-improvement and innovation that got them to the top of the talent pool will make them want to progress further. But you can’t really go further up than the top, so where do you go?
This is where people start considering switching jobs or pursuing entrepreneurship because it seems like the only challenging way forward.
The classic solution to this “problem” is to promote them to the management level. Clearly, if someone is performing exceptionally well as a specialist they will automatically become an exceptional manager… Right?
The solution is not always that simple and pushing someone to become a manager (or a manager of a bigger team than before) is not for everyone. Some top talent enjoy being a specialist and would rather spend their time performing their tasks, than managing a team.
“In a hierarchy, every employee tends to rise to hislevel of incompetence.”
– Laurence J. Peter, Author of The Peter Principle
The previous quote refers to what is known as the Peter principle, a concept of management developed by Laurence J. Peter. The principle suggests that people tend to get promoted outside of their skillset and competence, based on previous success.
Meaning: Your best front-end developer is first and foremost… a front- end developer. Having 10 award-winning projects under his belt does not make him an instant candidate for managing the next project. That requires knowledge of front-end and an additional management skill set, lack of which could lead to disaster down the line.
The modern solution to the problem is working with non-linear progression and promotion. Instead of the career path only going one way – towards management – you can set an alternative path. This could be anything from giving your top talent more influence on projects or a seat at the table when tough decisions are made to simply giving more freedom to perform tasks their own way. Once you start thinking outside the box you’ll be amazed at the possibilities there are for non-linear progression.
And the result? Happier top talent that gets a truly unique position at your agency, which they won’t be able to find anywhere else.
At SQAEB, most of our junior employees start out in the SWAT department, helping our users with day to day issues. This helps them naturally and quickly get an overview of all the other departments, the products, and how everything fits together. Later they can choose to transition into newly opened positions in the company that they find interesting or get places incompletely new positions based on their specializations.
Fun is a fickle thing. Everyone inherently knows what fun is, but if you had to define fun at the workplace, it would not be as easy as it first sounds. Looking up the definition of fun will also get you reprimanded by the dictionary, and there is no one sure way to define it. The only sure thing is that if the most interesting thing at the office on the first day is the photocopier, the new employee getting the tour will probably start looking for another job during the lunch break.
The overall feeling of fun at the workplace impacts productivity. And so it’s a topic without any specific bullet points, but a topic to think about and discuss nonetheless. If you want to have fun at the workplace but can’t manage to play chess on one screen while maintaining your focus on coding… or your keyboard shortcut hand is also your balloon tying and juggling hand… you will probably need to interact with other people eventually. But there is only a limited level of friendship and camaraderie that you can build with people when talking about code and sending each other design files.
When was the last time someone asked a different water cooler question than: ’’So, how was the weekend/any plans for the weekend?’’ In most agencies, it has probably been a while. And that’s expected. If you work in a consistent and focused environment, there are only so many topics that can come to mind.
But if you change up the setting, if you do different activities together, you might build more than just classic coworker bonds. You might build friendships. And what could be nicer than looking forward to Monday morning at the office to see your friends?
But not everyone comes to work looking for friendship. Especially top performers who just want to put on their headphones and forget that they are in an office environment.
Sadly headphones run out of battery, the wifi goes down, and progress meetings exist. Eventually, even the most focused people have to talk to their coworkers. And since you spend most of your day at work, people would prefer to cut down on the dry, corporate jargon and instead discuss or do something… fun.
This again brings us to the topic of shared values. The job of a back-end developer and the job of a UX designer require different personalities. So if your agency wants to have a varied offering of skills and backgrounds, you will have to find values that connect with every group.
But not just the ’’standard’’ values that are put on the agency “about us” page. The values that make up the constantly evolving personality of your agency. If you do this, you will eventually have an agency full of like minded individuals that don’t need to act corporate 24/7 and might even joke around from time to time.
Sadly, there is a thin line between having fun at the workplace and being overly quirky and disrupting everyone’s work. Unfortunately, you can also never get full value-alignment with every person that has been hired. But an agency where people think of each other as nothing more than colleagues and only spend time together at work is an agency that will have trouble scaling and keeping up with the more friendly teams later on.
Your culture and environment both have an impact on the quality of your work.
You have to spend money to make money. And you have to invest in top talent to retain top talent. Achieving maximum focus in an office setting where a million things are gunning for your attention is tough.
All of that can be managed with a good work culture and processes. But if you don’t have the right equipment and tools, you’ll never be as efficient as you could be.
Maybe a chair is not comfortable. Maybe you can still hear your sales team in the other room, even with your headphones on. Maybe you found a SaaS tool that would save you hours upon hours of repetitive tasks.
If someone asks for a new keyboard, new tool, or new screen, it’s never a good idea to dismiss them right away. The person asking rarely brings up an issue like this on a whim, it has to be premeditated in some way, and that means that the problem they are facing is a recurring one.
“The way management treats their associates is exactly how the associates will treat the customers.”
– Sam Walton, Founder of Walmart
A one-time investment, no matter how large, is actually pretty small when looking at it as a long term investment in focus and productivity. If an agency shows that it cares about its employees in all the ways that matter, the employees will return it multiple times over. Here are some small or large things in no particular order that could make or break an employee relationship with the company:
IT equipment. If you ask someone to work in front of a computer 8 hours each day, you better make sure they have the proper equipment to do their job. This includes everything from computer equipment to noise- cancelling headphones and online tools to do their job.
Chair and desk. This one is connected to the one above; spending a third of their day in uncomfortable working conditions will severely hurt their productivity and health.
Coffee, refreshments and snacks. We know it might not sound like much, but making sure that your employees have access to all the basics like coffee, cold water (or soda) and some fruit can drastically increase their productivity and improve health.
Indoor climate. The stereotype of a developer might be: someone sitting in a dark basement with a hoodie on – but nothing could be further from the truth if you want them to be productive. Proper lighting, some plants and good ventilation are all tiny details that have a huge impact.
A promotion: While most talented people love what they do, as they repeat the same tasks day after day, eventually, they will find ways of improving the process or get ideas for new ventures that the team should pursue. And there is only so much one can do from the bottom of the corporate ladder. Career growth is a key part of goal setting strategies for high performersand agencies need to provide these opportunities if they want to retain their top talent. Otherwise those people might look for those higher positions elsewhere. Please note, that a “regular” promotion is not always the best option; we’ll cover that later in our post “Non-linear progression”.
A raise: Usually going hand in hand with a promotion. However, while every promotion should come with a raise, not every raise has to come with a promotion. Many people are not after the responsibility that comes with a promotion, they just like what they do, and so they take on more tasks, spend more time at the office or even work weekends. But maybe they aren’t looking to delegate their tasks to their would-be replacements. Maybe they just want to feel like their extra time is seen as valuable by the agency. And seeing as time is money, sometimes the answer is as simple as that.
While all of the above will probably make your agency employees happy and get your agency valuable, educated and dedicated employees for a long time to come, there are also smaller ways to improve productivity faster.
Courses and conferences: There are always new books and courses popping up, covering the latest and greatest developments in the industry.
If your top performers ask about you helping fund their education, it’s one of the best ways to show them that you are counting on them in the future.
Maybe there is a developer conference coming up that would help them meet some like minded people and gather industry knowledge?
While it may seem like a big investment to send one or multiple developers away for a few days, the new knowledge and energy they bring back will pay dividends now as well as in the future. If they have valid arguments for going, why not give it a shot?
Schools and degrees: A similar approach to the one about courses and conferences, to an even higher degree (forgive the pun), should be taken if an employee asks about the possibility of returning to school.
Maybe they got this job straight after finishing their bachelor’s degree. Maybe they want to go for a manager position and think that an MBA would greatly improve their outlook.
Or maybe they want to slowly transition to another position, but wish to stay at the agency. Customer lifetime value and return on investment are some of the most important metrics that agencies need to keep an eye out. But try to imagine the “employee lifetime value”, of someone who you helped put through school.
Every movie about an office work environment has managed to, in one way or another, demonize the monotony of sitting at a cubicle doing the same work every single day. And who can blame them? Doing the same thing over and over again is widely referred to as the definition of insanity.
No one wants to feel like they aren’t progressing in their job. And this rings especially true when we are talking about top talent. If someone wants to stay at the top (where you probably want to keep them), they need to continually have an eye on the newest developments in their field.
The information gathering and processing is on them – allowing for an environment where they can test new ideas, that’s on the agency.
There are many ways to help talented employees fuel their passion for their work. Every person is looking for something different, but we have a few ideas that should be universally interesting for most people.
Allowing for a full five-day remote work schedule is not something that can be implemented instantly, it’s something that agencies have to build towards over time.
For a large portion of agencies, a full week of remote work might not even make sense at all. But giving people the freedom to work from home as needed on special occasions can remove a lot of unnecessary stress. If a person needs to take care of some errands, look after the kids, or maybe they are not feeling well enough to drive to the office, but well enough to work, why not have the option of working from home?
Let’s say you have a single developer dedicated to taking care of your agency website. He has tasks that he doesn’t actively collaborate with anyone else on. He gets a mockup of the website, some copy, and gets to work. He might also be actively trying to sell his apartment. In most companies, this would mean that he has to run back and forth between the apartment and the office, sometimes multiple times a day, to deal with the buyers, real estate agents and contractors. But does he really have to?
Would it not be more comfortable for him to stay at home and work between meetings? And would it not make it easier for his team members and managers not to have to keep track of his travel schedule? And if the work gets done in the right time frame, does his physical presence at the office really matter? I’ll discuss this further in “Is it time to go fully remote?” post.
SQAEB TIP
At SQAEB, everyone has a setup that allows for secure remote work, and in case of sickness, family emergencies, schoolwork or other unforeseen events, they are always welcome to work from home. We give people the benefit of the doubt / assume positive intent, and so far, it has always paid off.
Freedom is often hailed as the ultimate solution to happy employees. But most people have an easier time being creative when there are some restrictions in place.
Example: If your agency needs you to write as many slogans as possible selling pineapples in the next 10 minutes. When do you think you will produce more? A) If the 10 minutes is the only restriction. B) If you have a 10 minute restriction, you cannot use the word pineapple and all the slogans have to be under 10 words or less?
Studies show that B is the right answer – even though you have more freedom in A.Sidenote: We tried it at our office and we are currently considering a new venture in ’’Spiky yellow fruit’’advertising.
So does this prove that freedom may not be the answer to an infinitely creative and productive workplace culture?
Of course not – because we had the freedom to choose those restrictions.
Client expectations and agency needs dictate the tasks that have to be solved. Every agency also needs to have some time and budget restrictions to prevent a project getting out of hand.
Other than that, the freedom to solve the problem in any way possible is one of the most significant benefits you can grant your employees:
The most efficient way to a problem takes all the learning and experimentation out of the process
Using less billable hours and achieving maximum efficiency will inevitably mean that the client should probably expect cookie-cutter deliverables instead of innovative solutions
If there is a framework, guideline or brand book for everything, proposing new solutions and approaches might be perceived as too much of a hassle to even suggest
If you find the perfect balance in the above, you should have the How and Why of task management covered. But freedom in the workplace is a complicated thing. The How and Why are questions that have to be answered or the work will never get done. But why not take more weight off of people’s shoulders by not having them stress over the When and Where as well?
Hiring and onboarding new employees is one thing. But as we know, the costs of employee turnover is high. If you don’t work on having a great environment where your employees thrive, then it’s going to be very costly for you to keep replacing everyone.
Employees changing jobs is impossible to stop – especially in the tech industry – but there are things you can do to keep your turnover rate low.
This post could just be called ’’culture in the agency space’’ because that is the true key to acquiring and keeping top talent.
But what is company culture?
The 17-word, aka the short answer: Company culture is the combination of all the values, social interactions, and psychological behavior in an organization.
The 340-word, aka the long answer:Company culture is hard to define in specific terms, because unlike most essential things in business, it is entirely intangible, a feeling. Branding is closely intertwined with culture in every interaction that the company makes with any of its outside stakeholders. And if you want your brand to be consistent across all channels, you have to work towards a work culture that aligns with your corporate messaging.
A brand is a reflection of your company in the minds of your stakeholders.
That is why it takes on new forms in every piece of content shared on social media, every meeting with a possible client, and every shared lunch break with Debbie from the agency next door. A brand consists of many moving parts, some tangible, some not. The tangible can be boiled down to visual identity, messaging, and imagery, if need be. These can all be changed with a new set of guidelines, a new designer, or a new marketing department, but how do you control a culture?
Culture is not just a code of conduct, communication strategy, or a list of processes. Company culture includes all the small details:
The tone of voice the CEO uses to address a reporter while discussing a new acquisition
If your employees feel comfortable to talk about non-work related issues with their manager
If the new sales intern feels like waking up in the morning on his second week on the jobAnd that’s why culture is one of the hardest things to get right in an agency, as it can not be acquired, mandated or forced.
Culture has to be built and continuously monitored and maintained.
You can tell a lot about an agency culture:
In the way, your company treats employees, customers and the surrounding community
In the degree that your employees are committed to the company values and goals
By how comfortable employees are with innovating, making decisions and expressing their opinions
In how information is broadcasted from one department to another and from the higher-ups to the lower-level employees
There are many things a person needs to know on their first day at a company. And there are a lot of things that they will definitely not remember. To prevent information overload, it’s preferable to keep some essential things for the rest of the week so the fresh hire will pick them all up eventually. So what should they know on their first day?
Give them an “onboarding buddy”. This should be someone from their team, who they can ask any and all questions to, without feeling like you are bothering them
The values or the ’’WHY’’ of the company
The names of their closest coworkers
The tech stack your department is using
Where to find the best coffee machine in the building, as well as any other refreshments they can get (fruit, cold water, etc.)
How the company intranet or CMS works
The most efficient way to get to their desk
The information and communication flow of your company (emails, chat, phone calls, etc.)
Where the bathrooms are (you’d be surprised how often this is an issue)
What task management solution your team uses to keep track of tasks
When lunch is
Their first real work-related task
That’s about it, any other information would probably be too much, and as we all know, if you go for a handshake tour with every department immediately, you forget the first person’s name while shaking the third one’s hand.
Onboarding a new person to the team is a masterclass in taking your own medicine for a lot of agencies. Every good agency prides itself on an in- depth understanding of user journeys and user experience, but what is the experience of joining your agency like?
Placing someone behind a desk, giving them access to your password manager, and asking them to start developing right away is the equivalent of ordering a pizza and giving the delivery guy just your zip code. It takes so much more, and a good onboarding experience can make or break your company’s ability to foster new top talent.
Generally, tech companies started adopting ’’a multiple interview approach’’ that not only gives applicants a coding test or some homework, but also goes over their background and culture fit in the same depth. More and more agencies are now doing the same. This is where our hiring journey once again splits into two paths, this time, based on if you chose the internal hiring strategy or the headhunter/recruiter strategy.
The recruiter can take care of the searching, first impressions and the technical fit, but you should always have the most promising candidates meet the current team for a short and sweet meet and greet before you consider hiring them.
If the agency conducts the entire hiring process in-house, there is a lot of leeway in the process. Try new approaches and strategies, and eventually, you will find what works for you. But if you want a hint from a company that put culture first and has been doing so for 3 years, here’s how we do it at SQAEB:
Collaborative effort to identify skills required. Once we are sure we need a new addition to a department, the team goes over the exact skills we are looking for. This ensures that the team knows which new skills are coming in, instead of a manager deciding it themselves.
Job posting. When the manager has the final job posting ready, it is posted and shared online internally as well as externally. We know the value of a good network, so employees from all departments are asked to share it with anyone they might think is a good fit. To help gauge personality in the first screening process we usually ask for a short video introduction, along with a resumé, just to get an idea of who you are as a person even before we meet you.
Screening of candidates. As soon as we have enough candidates, the first screening process starts. This consists of sorting out any that does not have the required skills or did not adequately show that they would be a good cultural fit.
First interview. All candidates that pass our first screening are invited to a first interview. The purpose of the first interview is to get to know them as a person and figure out if they would be a good cultural fit. This includes having a current team member talk to them for 10 minutes one- on-one, without those involved with the hiring present. If the personality is a match to our culture, they are given homework and invited to a second interview.
Homework. While the first interview is focused on the cultural fit, the second is about technical skills. And to judge that, each candidate is given homework to complete before the second interview. This consists of various work-related tasks where they have a chance to showcase their skills. The homework also includes writing a movie review. This is an added curveball to see how they approach problem solving of tasks they probably haven’t done since high school.
Second interview. We have the second interview to go over the homework and technical questions. This is where their skills are assessed and the main goal is to ensure that the chosen candidate has the necessary skills to handle the tasks they would be given in the position.
Hiring. After the second round of interviews it is often clear which candidate is the best cultural fit and whether or not they have the necessary skills.
Now that you’re done recruiting and have hired the right person, the real work starts: onboarding. Hiring the right candidate is one thing; but if you don’t manage to give them a proper onboarding experience they will not perform as well as they could. Onboarding is the first step towards nurturing top talent.
Agencies have a lot of ways to get new talent in the door. You might do all the recruitment in-house, outsource it to a headhunter/recruiter or grow to a point where a dedicated HR department or in-house recruitment person is the way to go.
But no matter which option is the most viable for you, always keep the cultural fit in mind. You might find out that the person with the most extensive resume might be too far in their career to adapt to the workflow that works for the rest of the team. There are also cases of people with less impressive qualifications, who fit in so well with the rest of the team, that they hit the ground running and start producing work way above their estimated skill-level right away.
Making your agency a cultural paradise for top talent pays off in more than one way:
On one hand, you will attract those who have already proven to be top talent, which can give the quality and speed of work an instant boost. And if they are the ones who come to you looking to join, you’ll have a much larger talent pool to choose from.
On the other hand, you will be nurturing potential top performers from their career infancy and help them grow into top talent with the right personality traits to perform at your company. That has a ROI that can only be beaten by time travelers going back in time and buying stocks in Apple.
This whole train of thought is where agencies might learn something from the world of sports, where it’s a common philosophy in some football clubs (or soccer if that’s the term you prefer to use):
”We don’t sign superstars, we make them”. – Arsène Wenger, Manager of the Arsenal F.C.
But how do you make sure that your candidates are a cultural fit? And how can you make sure that they can do the work once they get hired?
Contrary to what you might think from our previous arguments about “personality > skills”, it’s important to start with the skills first. At the end of the day you need to know which skills you’re looking for before you can start evaluating personality and cultural fit.
When the hiring process is handled by the department or team that is looking for a new member, the senior members or managers are usually in charge of the process. If there is an obvious need for a specialist that the team doesn’t yet have, creating the requirements should be as easy as simply writing down the tasks that need to be done and translating them into skills. However, if there is just more work coming in for a specific skill set (UX, .NET Developer, etc.), the existing team members should be consulted so that the new hire can complement their skill set.
Once you are settled on the skills it’s time to consider the personality you’re looking for. Are you looking for a person with an extraordinary drive to grind it out 50 hours a week? Or maybe a true team player that makes everyone around them better? There’s no right or wrong answers here – but it’s important to have an idea of which personalities you’re looking for.
The tone of voice varies from agency to agency and even from team to team, and the structure of a job posting can vary quite a bit. But there are still some evergreen tips that could save you and potential candidates some time:
When a job has language or certification requirements that make or break the application, start with those
Don’t get caught up in the technical requirements and skills needed for the job.
Present the personality traits you are looking for on equal footing with skills, education and experience
When dealing with entry-level jobs, a portfolio of work could be supplemented with school projects that have a similar scope
Don’t put unnecessary year requirements on non-senior jobs
With software that has a steeper learning curve, ask for a specific platform that your team uses (Sketch/Adobe XD/InVision) instead of listing experience with prototyping software in general
Don’t ask for 8 years of experience in a language that has been around for 3 years
A good starting point for your ’’first point of recruitment’’ (not the first point of contact, because that’s probably your landing page) is to create a clear value proposition for the inbound job candidates. Until your agency reaches a certain size, you can’t cater to everyone’s wishes concerning work-life balance. Your hiring decisions should always be based on a cultural fit more than a technical fit.
While technical skills are clearly important, it’s much easier to improve a skill than it is to change a personality. If we want to go into specifics, we can go back to the user experience analogy. When writing a value proposition on the careers page, you need to think about what kind of agency you really are.
’’We are looking for dedicated people to help bring the most innovative web solutions to life for our clients by day, and help us put up new shelves for all these awards by night…’’
That statement will attract a certain kind of people:
Fresh graduates with a lot of ambition looking for validation of their skills
Experienced professionals who want an environment for their talents to be utilized
People looking for a challenge and don’t even consider crunch time a negative word
Career-building professionals who are looking for a place that gets them more awards to their resume
People who live for their jobs and look forward to evenings and Saturdays at the office filled with pizza and fixing the kinks in the code
Then on the other side of the spectrum, you could have:
’’ You bring the talent, we bring the perks. At AUE Inc. (Agency Used as an Example), we value strategy and planning above everything else. And thanks to our in-depth research and planning, clients always get the solution they need, instead of the solution they think they want. This also means that our employees never have to worry about scope creep or staying at work past 5 PM. Oh, and did we mention possibilities of
working from home or the 4 day work week?”
A few sentences like this on your career page could go a long way towards attracting people that:
Love their jobs, but don’t want to sacrifice time with their family for work
Are perfect for the job, but would have had to relocate or travel multiple hours every day
Are motivated for the job, but also have other ambitions and are trying to run some sort of side-hustle or project on the side
Sections like ”International Workplace” or ”Fun Squad” shows that we care about an open and fun work environment,where your colleagues also become your friends.
Before we get any further it’s time to address the tiny elephant in the room:
What’s more important – personality or skills?
To answer that question, you only need to scroll back up a few pages to find our list of characteristics for top talent. Notice how there’s only 1 called skill, while the rest are primarily based on personality?
That’s no coincidence. While skill alone is incredibly important, it’s what makes them capable of doing their job after all, it’s not necessarily the thing that makes them top talent. If they are an amazing coder, but can’t be depended on to meet deadlines or have issues working together with their team, it’s hard to call them top talent.
At the end of the day it’s important to remember that skills can be taught and improved, but personality and culture can’t. And if you want your entire team to perform – not just the individual – it’s important to have the right mix of personalities and culture. If the right culture is there, you’ll see skills improve for everyone and soon you’ll have a team full of top talent that performs day in and day out.
SQAEB TIP
For 99 % of our job postings we use this to highlight our people- first focus:
”We care about people. That’s why the most important qualification is your personality: who you are, what values you have and how you interact with other people. We are looking for people with passion and energy to be part of something bigger than themselves and who are willing to dedicate their time and skills towards building great products and services in collaboration with talented and friendly colleagues.”
Recruitment. Love it or hate it, this is where it all starts if you want to attract top talent for your agency. But there’s so much more to recruitment than job postings and hiring recruiters. It’s in the recruitment phase that the first bit of onboarding starts. While it is 100% the candidate’s responsibility to find out as much as possible about the agency he wants to join, why not show your values and culture even at the earliest stages and make it easier for them?
We are drawn to leaders and organizations that are good at communicating what they believe. Their ability to make us feel like we belong, to make us feel special, safe and not alone is part of what gives them the ability to inspire us
– Simon Sinek, Author of ’’Start with Why’’
It’s no secret that even the most basic one-page websites have an “about us” section. But imagine being a top talent developer or specialist looking for new opportunities. They might go through 50 “about us” pages every day. Does your mission and vision statement stand out of the crowd? Do you communicate having a culture that provides a constant stream of challenging problems to solve? Do you have a hilarious video of your founder switching places with your human-sized-rabbit-office-mascot and shooting confetti at your unsuspecting support staff?
SQAEB TIP
Do you want to show your values to potential clients? Then video is the way to go. It doesn’t have to be a big production – the only thing it has to do, is to show your company values and culture.
Letting your mission, vision and culture shine through in your recruiting process helps you immensely in not only standing out from the crowd, but also in attracting the right people for your company.
Before we start our deep dive into the obvious and not-so-obvious ways of attracting and retaining top talent, let’s take a moment to define:
What exactly is top talent?
Top talent is one of those terms that does not have a clear cut definition that people can point to. However, when talking about the agency world, there are certain characteristics that come up time and time again when discussing high performers:
Skill – The go-to metric for determining top talent. Whether it’s due to natural talent or 10,000 hours of practice, if someone is exceptionally skilled, they are on the best possible path to be considered top talent at any agency.
Ambition – The goal to become the top of their field. Ambition drives people to always keep up with the newest trends and developments in their field and continuously improve their skills.
Integrity – When they say something will get done, it gets done at all costs. And if both the managers and team members know they can count on someone when the going gets tough, that person becomes irreplaceable.
Communication – Knowing how to clearly communicate with managers and executives that speak the language of money on one side, while communicating with the technical team members who speak in code and high fidelity mockups on the other is a skill that should be paid in gold.
Teamwork – Everyone can excel at their individual tasks, but sharing a task or working efficiently in a team is a must-have for those that want to become the top performers in any agency
Creativity – Some creatives are a constant source of ideas during a brainstorming session. Some always see a problem from 3 more angles than everyone else. And while creativity manifests in a lot of ways, sometimes it’s the main thing behind a person’s top-talent status.
Leadership – Leadership is not just a skill for managers or team leads. People who join fresh out of college can find themselves at the top of the pyramid in any team within a few months, even with no direct effort. If an individual is approachable, facilitates a good workflow, or solves problems with a leveled head, they will soon become respected by their peers as a leader, even with no title involved.
Devotion – The green ’’you can talk to me’’-light next to the monitor turns red. The headphones go on. 6 hours, 3 cups of coffee, 1 missed lunch, and a single stretching session later, one individual just saved a 10-person project from being one week late. That’s how people become legends. And top talent.
Being considered top talent does not mean that a person has to have all of these qualities fully formed. It doesn’t even mean that top talent and top performers have to achieve all of these qualities eventually. A person who fully masters 3-4 of these qualities should quickly rise to become a prime asset to any agency. And if your agency finds itself hiring a person that displays most or all of these qualities, then you should do everything you can to keep them around until they decide it’s time to retire.
This section considers the following fundamental concepts:
Usability
User experience
Accessibility
Usability
Usability is the extent to which a software product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. Usability testers should be aware that other definitions may be used in organisations.
The user interface consists of all components of a software product that provide information and controls for the user to accomplish specific tasks with the system.
Usability evaluation includes the following principal activities:
Usability reviews
Usability testing
User surveys
A usability problem is a software defect which results in difficulty in performing tasks via the user interface. This affects the user’s ability to achieve their goals effectively, or efficiently, or with satisfaction. Usability problems can lead to confusion, error, delay or outright failure to complete some task on the part of the user. In safety-critical systems such as medical systems, usability problems can also lead to injuries or death.
A software product can work exactly to specification and still have serious usability problems, as shown by the following examples:
A car rental mobile app has a dead link. This is a defect which results in a usability problem.
A car rental mobile app allows users to cancel a reservation, but the users perceive the cancellation procedure as unreasonably complicated. This is a usability problem which affects the efficiency of the mobile app.
A car rental mobile app conforms to the specification and works both effectively and efficiently, but users think it looks unprofessional. This is a usability problem which affects user satisfaction when using the mobile app.
Usability always relates to the context of use and can be considered in different components. As the following examples show, user expectations of usability are rather different for these components.
Component
Component Name
1
Users
2
Tasks
3
Equipment
4
Environment
Description of Component in Context of Use
1. A user is a person who interacts with a software product by providing inputs, or by using the output of the software product.
2. Particular activities performed by users or particular groups of users (e.g., inexperienced users, administrators).
3. Equipment relates to the hardware, software and materials required to use a software product.
4. The environment consists of the physical, social and technical conditions in which a user interacts with a software product. The social conditions include the organisational conditions.
The following scenarios describe different contexts of use for the same software product:
Administrative staff use Microsoft Word ® to write documents in a consultancy firm
An elderly person uses Microsoft Word® for the first time to write an invitation to her birthday
User Experience Concepts
User experience describes a person’s perceptions and responses that result from the use and/or anticipated use of a product, system or service.
User experience includes the following user characteristics that occur before, during and after use:
emotions
beliefs
preferences
perceptions
physical and psychological responses
behaviours and accomplishments
User experience is influenced by:
brand image (i.e., the users’ trust in the manufacturer)
presentation (i.e., the appearance of the software product, including packaging and documentation)
functionality
software product performance
interactive behaviour
the helpfulness of the software product, including help system, support and training
learnability
the user’s internal and physical state resulting from prior experiences, attitudes, skills, personality, education and intelligence
the context of use
Usability criteria such as effectiveness, efficiency and satisfaction can be used to assess aspects of user experience such as brand image and presentation (satisfaction), functionality (effectiveness) and software product performance (efficiency).
Accessibility
Accessibility is the degree to which a product or system can be used by people with the widest range of characteristics and capabilities to achieve a specified goal in a specified context of use.
Evaluating Usability, User Experience and Accessibility
The key objectives of usability evaluation, user experience evaluation and accessibility evaluation are compared in the following table and discussed in more detail in subsequent sections.
Type of evaluation
Usability evaluation
User experience evaluation
Accessibility evaluation
Target group
All users
Key objective
Evaluate the direct interaction between users and the software product.
Evaluate the services received prior to the use of the software product.
Evaluate the direct interaction between users and the software product.
Evaluate the services received after the use of the software product.
Evaluate the direct interaction between users and the software product, focusing on understanding problems related to accessibility barriers, rather than general efficiency or satisfaction.
The principal techniques applied in usability evaluation, user experience evaluation and accessibility evaluation are shown in the following table and discussed in more detail in later chapters.
Technique
Usability review
Usability testing
User surveys
Users involved?
Optionally and Yes
Key characteristic
Experts and users evaluate the user interface of a software product for usability problems; the evaluation is based on their experience.
Users are observed while they perform typical tasks with the software product.
Users fill out questionnaires regarding their satisfaction with the software product.
Specific techniques
Informal usability review
Expert usability review
Heuristic evaluation
Think aloud testing
International Software Testing
Qual = Qualitative usability evaluation
Quant = Quantitative usability evaluation
Usability Evaluation
A process through which information about the usability of a system is gathered in order to improve the system (known as formative evaluation) or to assess the merit or worth of a system (known as summative evaluation).
There are two types of usability evaluation:
Formative (or “exploratory”) evaluation is conducted to understand usability issues. Formative evaluation is often conducted early on in the development lifecycle during the design and prototyping stages to get ideas and to guide (or “form”) the design by identifying usability design problems.
Summative evaluation is conducted late in the development lifecycle shortly before or after implementation to measure the usability of a component or software product. Summative usability testing is quantitative; it focuses on obtaining measurements for the effectiveness, efficiency or satisfaction of a software product. A summative usability evaluation can be used to evaluate a design based on usability requirements so that the design’s acceptability can be established from the users’ point of view.
Both types of evaluation can be conducted iteratively.
Usability evaluation relating to software products. Usability evaluation can also be applied to other products or services where usability is important, such as with user guides, vending machines, aircraft cockpits, medical systems and train stations.
Usability evaluation addresses the direct interaction between users and the software product. The direct interaction occurs via a screen dialogue or other form of system use. Usability evaluation can be based on a software application, on design documents and on prototypes.
The objectives of usability evaluation are:
to assess whether usability requirements have been met
to uncover usability problems so they can be corrected
to measure the usability of a software product (see below)
Usability evaluation addresses the following:
Effectiveness:
The extent to which correct and complete goals are achieved
Answers the question: “Does the software product do what I want?”
Efficiency:
Resources expended to achieve specified goals
Answers the question: “Does the software product solve my tasks quickly?”
Satisfaction:
Freedom from discomfort, and positive attitudes towards the use of the software product
Answers the question: “Do I feel comfortable while using the software product?”
If users are involved, a usability evaluation can be carried out by performing usability testing, conducting user surveys and performing usability reviews. If users are not present, usability reviews may still be performed. If software will be used by disabled individuals, include them early in usability reviews (i.e., color blind users).
A qualitative usability evaluation enables identification and analysis of usability problems, focusing on understanding user needs, goals and reasons for the observed user behaviour.
A quantitative usability evaluation focuses on obtaining measurements for the effectiveness, efficiency or satisfaction of a software product.
User Experience Evaluation
User experience describes a person’s perceptions and responses resulting from the use or anticipated use of a software product.
Usability is part of the user experience. Consequently, usability evaluation is a part of user experience evaluation. The principal techniques used for user experience evaluation are the same as those used for usability evaluation.
User experience evaluation addresses the whole user experience with the software product, not just the direct interaction. User experience includes:
Advertisements that make users aware of the software product
Training in the use of the software product
Touch-points with the software product other than screen dialogue, such as encounters with support, letters or goods received as a result of interaction with the software product
Problems that are not handled by the user interface of the software product, such as the notifications of delays, handling of complaints and unsolicited calls
User experience can be evaluated using the principal techniques outlined in the tables above. In a user experience test, time gaps can be bridged during a usability test session.
Accessibility Evaluation
Accessibility evaluation is a usability evaluation which focuses on the accessibility of a software product. It addresses the direct interaction between a user with disabilities or limitations and the software product.
The following advice applies specifically to accessibility evaluation:
1. Define the ambition level for accessibility The Web Content Accessibility Guidelines (WCAG) document defines three priority levels for accessibility; A, AA and AAA. It is recommended to adopt conformance level AA, which implies satisfying the most basic requirements for web accessibility and the biggest barriers for users with disabilities.
2. Create or adept guidelines for accessible design. These guidelines should comply with legal requirements. They should also be in accordance with the chosen ambition level for accessibility. Additionally, the usability of the guidelines for developers should be verified.
Review the guidelines for accuracy
Establish an accessibility hotline, where accessibility questions from development teams can be answered competently within an agreed time limit
3. Train development teams in order to prevent as many accessibility problems as possible. This includes factors such as:
Legal requirements for accessibility
Guidelines for accessible design and how to interpret and apply them
Tools and techniques to use when evaluating accessibility
The relationship between usability and accessibility
4. Accessibility testing focuses on the following aspects:
Use of a think aloud technique to understand the test participant’s thoughts and vocabulary during accessibility testing
Focus on understanding mistakes related to accessibility barriers, rather than on efficiency or satisfaction
Use tasks that concentrate on specific areas of concern for potential accessibility problems, rather than on general software product usage
Accessibility evaluation should consider relevant accessibility standards.
Usability Evaluation in Human-Cantered Design
Human-cantered design activities and their interdependence. Human-cantered design is an approach to design that aims to make software products more usable by focusing on the use of the software products and applying human factors, ergonomics, and usability knowledge and techniques.
The human-cantered design process can be summarised as follows:
Analyze: Talk with people and discover “what is the problem?”
Design: Prototype what you assume is a solution
Evaluate: Watch people use the prototype and learn from their experiences
Iterate: Repeat until the usability requirements are achieved
Human-centered design activities and their interdependence
The human-cantered design activities are based on the following three key elements:
1. Users
Observe and interview users in their work environment. Users are involved throughout the design stage by discussing designs and alternatives with them directly (where possible), or with representative users. In agile software development, representative users are typically the product owners, who are an integral part of the development team and enable frequent feedback to be given to designers and developers on usability issues.
2. Evaluation
Perform usability evaluation on the software product. A usability evaluation may take place at any time during human-cantered design, from early analysis through software product delivery and beyond. A usability evaluation may be based on a prototype, as mentioned above, or on a completed software product. Usability evaluations that are conducted in the design phase can be cost effective by finding usability problems early.
3. Iterations
Iterate between design and usability evaluation.
Considering the human-cantered design process, the most frequent iterations take place between the activities “Produce design solutions” and “Evaluate design solutions”. This generally involves the successive development of a prototype, which is a representation of all or part of a software product’s user interface. Although prototypes are limited in some way, they can be useful for usability evaluation. Prototypes may take the form of paper sketches or display mock-ups, as well as software products under design. Starting with an initial prototype, the following activities are performed:
The prototype is evaluated. The person who performs the evaluation conducts usability testing on the prototype.
The prototype is improved and refined based on the results of the evaluation. The person who performs the evaluation helps the developers evolve the prototype by incorporating user feedback into the design.
These activities are repeated until the usability requirements are achieved. When prototypes are developed in iterations, the steady refinement gives the user a more realistic impression of how the finished product will look and feel. Additionally, the risk of forgetting or ignoring usability issues is reduced. Both usability and accessibility must be considered during the design phase. Usability testing often takes place during system integration and continues through system testing and into acceptance testing.
Usability Requirements
A usability requirement is a requirement on the usability of a component or system.
It provides the basis for the evaluation of a software product to meet identified user needs. Usability requirements may have a variety of sources:
They may be stated explicitly, such as in requirements documentation or a user story
They may be implicit, undocumented user expectations (e.g., a user might implicitly expect that an application provides shortcut keys for particular user actions)
They may be included in adopted or required standards
Examples of usability requirements (in this case described as user stories) are:
“As a frequent user of the airline’s booking portal, an overview of my currently booked flights shall be automatically shown after I log on. This shall enable me to get a quick overview of my booked flights and quickly make any updates.” This usability requirement is about the effectiveness component of usability.
“As a help-desk assistant, I must be able to enter and log the details of a customer request into the Customer Relations database in no more than two simple steps. This shall enable me to focus on the customer request and provide them with optimum support.” This usability requirement is about the efficiency component of usability.
Agile Usability Evaluation
Usability evaluations are also suitable in agile software development.
Agile software development is a group of software development methodologies based on iterative incremental development, where requirements and solutions evolve through collaboration between members of a self-organising team.
In agile software development, teams work in short iterations, each of which has the goal of designing, implementing and testing a group of features.
The following usability evaluation approaches work well with agile software development:
Rapid Iterative Testing and Evaluation (RITE) is a qualitative usability test method where changes to the user interface are made as soon as a usability problem is identified and a solution is clear. The RITE method focuses on instant redesign to fix problems and then confirming that the solution works with new test participants (real users or representative users). Changes can occur after observing as few as one test participant. Once the data for a test participant has been collected, the usability tester and the stakeholders decide if any changes are needed prior to the next test participant. The modified user interface is then tested with the remaining test participants.
Informal and quick usability test sessions are useful where many potential users can be accessed (e.g., a cafe, a conference or a trade show). Such forms of usability test sessions typically last less than fifteen minutes and apply techniques such as think aloud and heuristic evaluation.
Weekly testing. Test participants are recruited well in advance and scheduled for a particular day of the week (e.g., each Tuesday), so that the software build can be usability tested on that day. Usability tasks are prepared shortly before the scheduled testing day and may include exploratory testing sessions, where the knowledge of the tester and heuristic checklists are used to focus on usability issues.
Organisations need confidence that critical business processes, such as order-to-cash procedures, human resource on-boarding, or production planning, can be performed without disruption. This is known as “business process assurance” and it is an essential objective of acceptance testing. In this context, two standards exist that provide a common language for business analysts and testers for graphically representing business processes and business rules: Business Process Model and Notation (BPMN) and Decision Model and Notation (DMN). These models support the design and implementation of tests and help to determine the priority for execution.
Business process/rule models describe the business flow and the expected behaviour of the test object. Representing business processes and rules to be tested using a graphical notation helps to establish a common understanding of what is expected. A business process corresponds to a flow of tasks, alternative paths, and the various events at the start, the end or possibly during the control flow. Business rules define explicit criteria for guiding behaviour, shaping judgments, or making decisions.
Business Process Model and Notation (BPMN), maintained by the Object Management Group (OMG), is a recognised standard for business process modelling which uses a flowcharting technique. In this article, a subset of the Business Process Model and Notation (BPMN) notation is used that is sufficient to draw simple business process models in the context of acceptance testing activities.
Decision Model and Notation (DMN), also standardised by the Object Management Group (OMG), is complementary to the BPMN standard. While Business Process Model and Notation (BPMN) is used to represent workflows, DMN is used to represent decisions, business rules and outcomes/output within the workflow. In this article, a subset of the Decision Model and Notation (DMN) notation is used that is sufficient to define business rules in conjunction with simple business process models in Business Process Model and Notation (BPMN).
Deriving Acceptance Tests from Business Process/Rule Models
A business process model with business rules, described with the Business Process Model and Notation (BPMN) and/or Decision Model and Notation (DMN) notations, provides a precise definition of the scenarios to be tested, including the cases related to business rules. It is a good basis for generating acceptance tests using coverage-based test selection criteria as defined in a model-based testing approach.
Coverage-based test selection follows the principle that the business analyst and tester agree on the coverage items that shall be fully tested. Typical coverage items for business process models when generating acceptance tests include the following:
User stories, requirements, and risks annotated in the business process model.
Decisions in the decision tables describing the business rules.
User scenarios defined by different paths through the business process model.
All paths (usually without loops) through the business process model.
Once the coverage items are defined, the tester then identifies a set of test cases that covers those items. Full coverage is achieved if the test suite covers each occurrence of the coverage item in the model at least once during execution.
Different coverage criteria may be combined to meet the acceptance testing objectives. For example, the objective may be to cover all paths of a given main scenario, but only one path of each alternative scenario.
Business Process Modelling for Acceptance Testing
Business process/rule models describe the business flow and the expected behaviour of the test object. The use of business process/rule modelling in the context of acceptance testing is based on good modelling practices and supports visual ATDD practices.
Good Practices for Business Process Modelling for Acceptance Testing
The following good practices should be considered when using Business Process Model and Notation (BPMN) and Decision Model and Notation (DMN) for acceptance testing:
It is not necessary to describe everything in a business process model. The graphical representations of business processes in BPMN should focus on requirements to be tested. Therefore, workflow descriptions that only partially cover the behaviour of related software systems are acceptable, as long as they represent what is to be tested.
Especially for rule-based business processes, using decision tables helps manage dependencies. DMN supports the definition of conditions and outcomes corresponding to the business rules under test.
Diagrams should be as simple as possible and be structured in sub-processes when needed to limit the number of graphical elements in a single business process diagram. This improves readability and facilitates reviews.
Business process modelling for acceptance testing should be a collaborative work between business analysts and testers. Artefacts produced should be shared between and reviewed by both roles. Early and close communication between those two roles improves the quality of requirements or user stories as well as tests. (This is true for all test levels.)
Additional information such as links to user stories, requirements, risks, priorities and any other information useful for acceptance testing should be added to the diagrams using annotations. By keeping all relevant information in a single location, it becomes easier to make decisions and reasons are better documented.
Using Business Process Models for ATDD
During the refinement sessions for requirements and user stories, the business process and business rule models will help the team to get into the details of the expected behaviour and the acceptance criteria. The representation of workflows in Business Process Model and Notation (BPMN) and of rules in Decision Model and Notation (DMN) directly enable testers to design appropriate test cases that verify the acceptance criteria.
Business process modelling for ATDD is based on the following principles:
Business analysts and testers collaborate to model workflows and business rules using graphical notations such as BPMN and DMN.
These business process/rule models are reviewed with relevant stakeholders and contribute to the validation of the requirements and acceptance criteria.
Testers derive tests from these business process/rule models to ensure and demonstrate the required coverage through the different paths and business rules.
Business analysts and testers may also use the models to identify changes that necessitate test case maintenance and to select regression test cases.
Business process/rule models created and maintained for ATDD can be viewed as living documentation used by business analysts to present the actual behaviour of the test object.
Automated test generation techniques can be used to produce and maintain automated test scripts. The model-based testing approach can also be combined with keyword-driven testing and data-driven testing approaches.
Business process/rule modelling in ATDD provides a visualisation of the workflows to be tested. This is the major difference from the Gherkin language used in BDD.
Specifying acceptance criteria is an important acceptance testing task. It helps to refine requirements or user stories and provides the basis for acceptance tests. Business analysts and testers should collaborate closely on the specification of these criteria. This collaboration ensures high business value from the acceptance testing phase and increases the chance of a successful iteration or product release.
Writing acceptance criteria forces business analysts and testers to think about functionality, performance, and other characteristics from a stakeholder or usage perspective. This supports early verification and validation of the related requirement or user story and provides a better chance of detecting inconsistencies, contradictions, missing information or other problems.
The following good practices should be considered when writing acceptance criteria:
Well-written acceptance criteria are precise, measurable and concise. Each criterion must be written in a way that enables the tester to measure whether or not the test object complies with the acceptance criterion.
Well-written acceptance criteria do not include technical solution details. They concentrate on the question “What shall be achieved?” rather than on the question “How shall it achieved?”.
Acceptance criteria should address non-functional requirements (quality characteristics) as well as functional requirements.
As with requirements and user stories, acceptance criteria should be reviewed through walkthroughs, technical reviews, iteration planning meetings or other methods (if necessary).
Designing Acceptance Tests
This section addresses the test techniques and approaches frequently used for acceptance testing.
Test Techniques for Acceptance Testing
In a requirements-based approach to acceptance testing, the tester derives test cases from the acceptance criteria related to each requirement or user story using black-box techniques such as equivalence partitioning or boundary value analysis.
Acceptance testing may be augmented with other test techniques or approaches:
Business process-based testing, possibly combined with decision table testing, validates business processes and rules.
Experience-based testing leverages the tester’s experience, knowledge and intuition.
Risk-based testing is based on risk types and levels. Prioritisation and thoroughness of testing depends on previously identified product risks.
Acceptance criteria should be verified by acceptance tests and traceability between the requirements / user story and related test cases should be managed.
Using the Gherkin Language to Write Test Cases
In ATDD and BDD, acceptance tests are often formulated in a structured language, referred to as the Gherkin language. Using the Gherkin language, test cases are phrased declaratively using a standardised pattern:
Given [a situation]
When [an action on the system]
Then [the expected result]
The pattern allows business analysts, testers and developers to write test cases in a way that is easily shared with stakeholders and may be translated into automated tests.
The “Given” block aims to put the test object in a state before performing test actions in the “When” block. The “Then” block specifies the consequences that can be observed from the actions defined in the “When” block. Test cases written in Gherkin do not refer to user interface elements but rather to user actions on the system. They are structured natural language test cases that can be understood by all relevant stakeholders.
In addition, the structure “Given – When – Then” can be parsed in an automated way. This allows automated test script creation using a keyword-driven testing approach.
Initially, Gherkin was specific to some software tools supporting BDD, but it is now synonymous with the “Given – When – Then” acceptance test design pattern.
Experience-based Approaches for Acceptance Testing
All experience-based test techniques described in are relevant for acceptance testing. This section is focused on how exploratory testing can be used for acceptance tests, and on beta testing as a source of feedback on system usage.
Exploratory Testing
Exploratory testing is an experience-based test technique that is not based on detailed predefined test procedures. In exploratory testing, all activities are carried out within an uninterrupted period of time called a session. The testers are domain experts. They are familiar with user needs, requirements and business processes, but they are not necessarily familiar with the product under test.
During an exploratory testing session, the tester accomplishes the following:
Learns how to work with the product
Designs the tests
Performs the tests
Interprets the results
It is a good practice in exploratory testing to use a test charter. The test charter is prepared prior to the testing session (possibly jointly by the business analyst and the tester) and is used by the person in charge of the exploratory session (either a business analyst, tester or another stakeholder). It includes information about the purpose, target, and scope of the exploratory session, the test setup, the duration of the session, and possibly some tactics to be used during the session (such as the type of user that shall be simulated during the exploratory session). Time-boxed sessions help to control the time and effort dedicated to the exploratory session. It is also good practice to perform exploratory testing in pairs or as team work.
In Agile development, exploratory test sessions can be conducted during an iteration by the product owner and/or the testers for acceptance testing of user stories assigned to the iteration.
Exploratory testing should be used to complement other more formal techniques in acceptance testing. For example, it may be used to provide rapid feedback on new features before methodical testing is applied.
Beta Testing
Beta testing is a form of acceptance testing that is often used for Commercial Off-the-Shelf Software (COTS) or for Software as a Service (SaaS) platforms. It is conducted to obtain feedback from the market after development and in-house testing are completed.
Unlike other acceptance testing forms, beta testing is performed by potential or existing users at their own location. Beta tests neither impose predefined test procedures nor a test charter. Apart from the observed findings, the test activities are usually not documented at all.
Because the product is tested in various realistic configurations by actual users in their business process context, beta testing may discover defects that escaped during the development process and previous test levels. Resolving issues found by beta tests helps organisations avoid costly hot-fixes or product recalls on a larger scale.
Acceptance testing should not be limited to beta testing. Beta testing is not systematic or measurable. There is no guarantee that all requirements or user stories are covered by the tests. Moreover, beta testing is performed late in the development process whereas tests based on acceptance criteria support the “Early Testing” principle.
Testing phases within the Gaming Software Development Lifecycle
Test types during the gaming quality assurance phase
Compliance testing is a very involved process. There are lengthy submission forms to be filled and ITL compliance fees are often much higher than gaming quality assurance testing fees. Therefore, prior to submitting a product to an ITL, a machine manufacturer should ensure the quality of the product being submitted by performing gaming quality assurance testing.
Gaming quality assurance testing activities align with the fundamental test process. The following test activities are performed to ensure the product is tested for quality:
Test planning
Test monitoring and control
Test analysis
Test design
Test implementation
Test execution
Test completion
Gaming quality assurance testing is an iterative process with development. Defects are logged and then the product is returned to development to fix the logged defects. Once the logged defects are fixed, the product comes back to gaming quality assurance for further testing. This cycle continues until the product reaches the quality levels desired, as defined by the gaming quality assurance testing exit criteria.
The test types performed during gaming quality assurance testing include, but are not limited to:
Localisation testing – testing to determine that all screens have proper language translations and any other localisation items such as date/time or numbering formats are done correctly.
Functional testing – testing to determine that all functions work as designed and intended.
Performance testing – testing to determine that response time is acceptable.
Memory leak testing – testing to determine no performance issues arise due to lack of proper memory management by the software.
Install-ability testing – testing to determine that the product can be easily installed by the casino operator without any issues.
Portability testing – testing to determine that the product can be installed on all platforms it needs to run on.
System integration testing – testing to determine that the product can integrate with systems/components, different versions of an application interface or able to communicate to a system.
Operational testing – testing to determine that the product can function and be operated in a production environment, including reliability, security, recovery and failover testing.
Pre-compliance testing – testing to determine that all the regulations and standards are met prior to submitting the product to the ITL. This helps ensure a minimal number of submissions of the product to compliance testing.
Customer acceptance testing – Prior to submitting to compliance testing, some products are submitted to the client (i.e., lottery or casino) for customer acceptance testing to ensure all features function as the client expected.
Compliance testing
Compliance testing occurs when a machine manufacturer wants to enter a jurisdiction with a new or modified product, be it a game, platform, hardware or system. Based on the jurisdiction, the machine manufacturer needs to submit the product with a request for certification to either an ITL or government-based compliance test lab. In some jurisdictions, it goes through both.
The formal submission documents are fully detailed, what is being submitted and what certifications are being requested for the product.
Once the product has passed compliance testing, the test lab will provide a certificate of compliance evidencing the certification of the rules and regulations that were to be met.
Once the regulatory commission has seen proof of the required certifications, it will allow the product to be installed in the gambling establishments in their jurisdictions.
When performing compliance testing, standard test plans are created and specific compliance checklists are used. These may include, but are not limited to:
Jurisdictional specifications – usually defined by a governing body such as the federal, state and/or provincial government.
ITL defined standards – defined by an ITL in the gambling industry.
Other gaming related standards – some jurisdictions require other standards be adhered to. For example, some jurisdictions may require that gaming machines and systems in a jurisdiction are Game to System (G2S) protocol compliant. The G2S compliance checklist is defined by the Gaming Standards Association, the association that has defined the G2S protocol.
Many areas of the compliance testing will be the same as those performed in gaming quality assurance testing, but they are tested against the jurisdictional specifications and not the game specifications. Some of the areas that are covered during compliance testing include:
Rules of play – testing to determine that the rules meet the jurisdictional specifications.
RNG, Payout Percentages, odds and non-cash awards – testing to determine that the payout percentage is within the range regulated in that jurisdiction.
Bonus games – testing to determine that the game meets bonus regulations.
Electronic metering – testing to determine that all meters required to be monitored within that jurisdiction are being reported.
Game history – testing to determine that the game history tracks, at a minimum, the number of games required by the jurisdiction.
Power-up and power-down – testing to determine that the power up and down functionality works as per the jurisdictional specifications.
Setup and Configuration – testing to determine that only configurations that are permitted within the jurisdiction can be enabled.
The Gaming Ecosystem
The gaming industry ecosystem overview
The gaming industry ecosystem is composed of the following organisations:
Game Developers – develop casino games not specific to a gaming machine model. These games are usually distributed by a manufacturer or casino.
Machine Manufacturers – make and sell the hardware, platforms, operating systems and games, developed in house or sub-contracted.
Independent Test Labs – test and certify that the game software, hardware, firmware, platform and operating system follow all the jurisdictional rules for each location where the game will be played.
Regulatory Commissions – approve every game played in their jurisdiction after the ITL certifies that the game meets the commission’s jurisdictional specifications.
The regulatory commission licenses the machine manufacturer to deploy the game in casinos or on online gaming sites in that specific jurisdiction. A game may be shipped to a casino before licensing; however, it cannot be deployed. The game must be licensed by the regulatory commission before it is deployed into the jurisdiction. Should any major defects be found in the casino, the regulatory commission can force the machine manufacturer to pull their game out of all casinos or demand that the online sites remove access to the game in that jurisdiction.
Video lottery terminals and their ecosystem
As indicated by the name, VLTs always have a video display for the game. VLTs either have standalone or server-based outcome architectures. In the standalone model, each VLT contains an RNG from which game outcomes are generated. In the server-based outcome architecture, VLTs obtain their outcomes from the server. This architecture has two possible models: the RNG model or the pre-determined finite pool model. In the server-based RNG model, the server generates the outcome it will provide to the VLT using an RNG located in the host. In the pre-determined finite pool model, the server obtains the outcome from a database of pre-determined outcomes. This model is similar to instant tickets and is often referred to as electronic instant tickets.
The types of games typically found on a VLT are: mechanical reel games, poker games and keno games. Most VLTs are multi-game machines, meaning multiple games are available for a player to choose from through a screen menu.
VLTs are frequently operated in a distributed environment over a Wide Area Network. For example, a few VLTs deployed in bars and/or pubs are connected to a central server through a Wide Area Network connection.
The VLT ecosystem is composed of:
The EGM
The site controller and/or bank controller
The systems/servers used for monitoring and/or managing functionality
The EGMs are the machines on which the players choose to play the games. Each machine communicates to a site controller and/or bank controller and one or more central servers through a communication interface board using an electronic communication language referred to as a protocol. When VLTs are installed in a distributed environment, each retail location has a site controller to which the VLTs at that location are connected. The site controller serves multiple functions:
Communicates and monitors VLTs to ensure they are online.
Records game play transactions, cash-in/cash-out transactions and security exceptions.
May act as a protocol converter by translating the protocol implemented on the VLT to the protocol understood by the central server.
Provide retailers with the ability to:
Register players for player tracking cards
Validate and pay out cash tickets
When VLTs are installed in a venue environment (i.e., a non-distributed environment), they are connected to a bank controller which functions like a site controller minus the retailer functions. A bank controller can support connection of several hundred VLTs, whereas one site controller typically supports fewer than 100 connected VLTs.
The VLTs and bank controllers and/or site controllers are connected to various central servers based on the functionality offered by a jurisdiction. At a minimum, VLTs installed in a venue environment include the following:
A casino accounting system, which is responsible for monitoring the amounts wagered and paid on each VLT.
A VLT CMS, which provides the ability to monitor game play, track, record and report security exceptions at the VLT and/or site controller, and monitor network availability in order to ensure continuous VLT operations in the event of communication loss.
Other central servers may include additional features, not limited to:
A cashless wagering server, which allows for cashless transactions either through ticket-in/ticket-out (TITO) functionality or through electronic funds transfer (EFT).
A distributed game content management server, which controls the selection, scheduling, distribution and auditing of VLT software to VLTs at remote retail sites.
A player services server, which supports player loyalty, player rewards and responsible gaming functionality.
A progressive server, which manages progressive game play.
A business intelligence server, which provides data warehousing and business analytics.
The other servers available are based on the functionality offered in the jurisdiction.
Slot machines and their ecosystem
Slot machines may have a video display or mechanical reels which have actual physical reels that spin.
Slot machines outcome architectures come from the Indian Gaming Regulatory Act, a 1988 US Federal law that establishes the jurisdictional framework that governs Indian Gaming in the US, This law provides definitions for Class I, Class II and Class III architectures. Class I relates to traditional Indian gaming and will not be discussed further in the context of casino gaming. Class II and Class III define the two outcome architectures used by slot machines. Class II (also know as electronic bingo) is defined in the Act as “the game of chance commonly known as bingo whether or not electronic, computer or other technological aids are used”. Class III (also known as traditional slot machines) has a broad definition in the Act. It states “all forms of games that are neither Class I nor Class II. Games commonly played at casinos, such as slot machines and table games, e.g., blackjack, craps, roulette, etc., fall in the Class III category.
The types of games typically found on a slot machine are: mechanical reel games, bingo games, poker games and keno games. Many slot machines are single-game machines, meaning only one game is available for play on the gaming machine.
Slot machines are typically operated in a venue environment such as a casino.
Slot machines (also known as Vegas style slot machines) are:
Casino gaming machines with mechanical reels or a video display.
Machines that have an RNG that is local to that machine.
Machines usually include a currency input device, such as a coin acceptor or a note acceptor, and a currency output device, such as a coin hopper.
The slot machine ecosystem is composed of:
The slot machines
A slot machine interface board (SMIB)
A data collection unit or bank controller
Central servers
Each slot machine contains an SMIB that is linked to the data collection unit or bank controller. Historically, an SMIB was a small board that was put into every mechanical or electro-mechanical machines. These early SMIBs connected to a wiring harness that would detect when a mechanical meter was incremented, or a mechanical door switch was opened. As time passed, these SMIBs evolved and now communicate electronically with the gaming machine and are often responsible for implementing the protocol that is used to communicate with the data collection unit or bank controllers and the remote central servers. The SMIBs, at a minimum, capture:
The amounts wagered by the player.
The amounts paid out to the player.
And if the player is using a player card, any player data tracked by the casino.
Data collection units or bank controllers, as suggested by the name, are used to collect and store the data obtained from the SMIBs.
The data collected by the data collection unit or bank controller is communicated to the servers to update the data needed for the functionality provided by the servers.
Bingo machines (also know as electronic bingo machines), are:
Machines that look and feel like slot games but are actually a game of electronic bingo.
Machines on which the outcomes are obtained from a centralised bingo server.
Machines that offer cashless input methods such as TITO or EFT. These gambling machines do not have currency input/output devices.
Machines on which the games:
Are played exclusively against other players rather than against the house or against a player acting as a bank.
Are based on multiple players playing for a common prize.
Continue until there is a winner.
Each bingo machine contains an SMIB that is linked to the bingo server and other servers. The SMIBs, at a minimum, capture:
The amounts wagered by the player.
The outcome obtained from the server and the corresponding results.
The amount paid out to the player.
The Network Switches are used to provide multi-player capabilities. Once a minimum number of players required for a specific game is met, the actual bingo game can start. The bingo server is the system that allows players to join a game until the group is at the minimum required and that provides the outcomes to the slot machines.
There are other central servers, such as the casino accounting server that tracks the amounts wagered and amount won, and the reporting server that allows the casino operators to report on the collected data.
Lottery and its ecosystem
The lottery ecosystem is composed of systems and devices deployed at the lottery and at each retail location.
The main device is the point of sale (POS) lottery terminal. The POS lottery terminal facilitates the sale of traditional lottery tickets by allowing the retail employee to either scan a selection slip containing the player selected numbers, or to select a Quick Pick option where the POS lottery terminal randomly selects numbers for the player. The POS lottery terminal then prints the tickets on the attached printer. The POS lottery terminal facilitates the sale of instant tickets by scanning the instant ticket sold. The attached customer display unit (CDU) allows the customer to view all steps of the sales transaction. The POS lottery terminal must coordinate all lottery ticket sale transactions with the lottery CMS. The POS Lottery terminal, printer and CDU/PDU unit are either separate devices or, in some cases, these devices can be integrated into one unit. When a player is ready to validate a ticket, the player can choose to have the retail employee scan the ticket using the POS Lottery terminal or perform the validating themselves on a POS Self-Serve Terminal.
The final device at the retail location is the multimedia display. The multimedia display is used for in-store advertising of lottery products, upcoming lottery promotions and winning numbers.
Once the numbers are drawn, the numbers are entered into the lottery CMS. Using the data stored in the database of the CMS, reports can be generated indicating how many winning traditional lottery tickets were sold and which retail location sold the ticket(s). For instant tickets, the barcode data of each ticket is stored in the CMS database. This allows the lottery employees to generate reports indicating how many tickets remain unsold and manage replenishment of physical tickets to retailers. The lottery CMS is responsible for storing all transactional data of tickets sold at each retail location. It also manages the advertising content to be displayed on the multimedia display units at the retail locations and downloads the appropriate content to the POS lottery terminal from which the multimedia display unit displays the content.
Lotteries are beginning to introduce alternative means by which to purchase lottery tickets. For example, purchases can be made at self-serve vending machines for instant tickets, self-serve ATMs like kiosks for traditional lottery tickets or online at their website for lottery tickets. These alternative devices and components must also coordinate all sale transactions with the lottery CMS.
Some of the areas that are covered during functional testing of the lottery ecosystem include:
Online game rules and functionality.
Scratch ticket management.
Player account management.
Geolocation functionality.
Player services manager.
Multi-player gaming
Player loyalty and rewards.
Account based play “cashless”.
Responsible gaming functionality
Game play functionality and playability.
RNG algorithm and math rules.
Artwork versus requirements
Host accounting and reporting – determine that the game pays out what it should and that the money at play goes to the client if they win, or to the casino if the client loses.
Tournament and real time event setup and management.
Multiple game engines functionality and capabilities.
Integrations with external gaming sites and mobile devices.
Being a gaming industry tester means that you must understand both testing in general, and the unique set of skills for the gaming industry ecosystem. This ecosystem is filled with proprietary, complex, multifaceted gaming software, hardware, platforms, firmware and operating systems. The objective of this article is to provide the regular Tester graduates with the specific knowledge that is required for a career in gaming industry testing.
Why the gaming industry requires a specialist tester
Some of the specific testing for the gaming industry, not present in other testing areas, include the following:
1. Gaming industry ecosystem – The unique hardware, firmware and operating systems that are proprietary to the gaming industry.
2. Gaming industry compliance testing – There are over 440 different certification boards worldwide for gaming industry games. These boards have rules that games in the gaming industry must comply with. These rules impact hardware, software, platform, operating systems, visual and auditory functionality, mathematics, and return to player (RTP) calculations. One gaming industry game can be played in multiple gaming jurisdictions and needs to comply with the laws of each location.
3. Fun factor or player perspective testing – This is something unique to gaming industry games, since they are an entertainment product. Not only are casino games supposed to work intuitively and provide the player pleasure, they must also be fun to play. This requires a unique insight into game design, with experience and information about the user group and what that group enjoys.
4. Math testing – Testing the multitude of pay tables, permutations, Random Number Generator (RNG) results and RTP computations. This type of testing requires the tester to understand what triggers different types of payout behaviour and to understand financial return to the player and how these triggers can be treated by different parameters. Understanding math testing is critical to succeed in this field.
5. Audio testing – Creating sound or playing media is common in software. However, gaming industry game music must engage the user in the game and enhance the game play. Not only should the audio play without stuttering or missing elements, it should also add to the game play. This requires extensive audio skills and specific understanding of game audio.
6. Multiplayer testing – This type of testing is performed when many players are simultaneously interacting with casino games, with computer-controlled opponents, with game servers, and with each other. Typical risk-based testing is followed to ensure against using unlimited amounts of time testing different scenarios. Understanding multiplayer game design, and how to test it efficiently, is required knowledge for this type of testing.
7. Interoperability Testing is common in all software that communicates with other software, systems and/or components. Casino/Video Lottery games have a unique aspect in that they must implement interoperability using gaming industry open protocol standards or proprietary protocols as per the specifications of the central server deployed in the jurisdiction to which the game is deployed.
Gaming Activities and Artefacts
Background
To understand gaming industry testing and its ecosystem specificities requires a review of the business model, activities, and artefacts as they pertain to the gaming industry.
What is gaming?
Gaming can be defined as follows:
The wagering of money or something of value, also called stakes, on an event
Where the outcome of the event is unknown
Where the whole intent is winning additional money, material goods or trips
What is a gaming machine? A gaming machine is a machine that enables the wagering of money or something of value. Examples of gaming machines are: electronic or mechanical slot machines, a roulette table or even a computer for online gaming.
Types of Gaming
Casino games
There are three categories of casino games: table games, electronic gaming machines (EGMs) and random number ticket games.
Examples of table games are roulette, blackjack, baccarat or poker, which typically are not tested unless they are an electronic table game version of these games.
The second group are EGMs, typically known as video lottery terminals (VLTs) or slot machines. These are usually played by one player at a time and do not require the involvement of casino employees to play. These games need to be tested, i.e., the game software, the machines, the operating systems, and platforms that they are based on.
VLTs and slot machines are both gaming machines that allow players to bet on the outcome of a game. Physically, VLTs and slot machines are very similar in nature. The main difference between a VLT and a slot machine is that VLTs are gaming machines that are operated by government lotteries while slot machines are gaming machines operated by private organisations such as casinos.
Both VLTs and slot machines are regulated and require licenses to be operated within their jurisdictions. Many countries around the world offer legalised VLT or slot play. For example:
In the United States, a 1988 federal law established three classes of games for Native American casinos, with different regulatory schemes for each. Each state government follows variations of these classes to define their regulations.
In Canada, the provincial or territorial governments are responsible for regulating gaming operations. All provinces offer the ability to play, each with their own regulations.
In Australia, the laws regulating the use of gaming machines are the responsibility of the state governments.
Other terms by which a VLT and slot machine are referred to: EGM, Video Gaming Terminal, Video Gaming Device, Video Slot Machines and Interactive Video Terminal.
The third casino game category is random number ticket games such as Keno and simulated racing. These games are based on the selection of random numbers, either from a computerised RNG or from other gaming equipment.
Lottery systems
A lottery is a form of gaming that involves selling numbered tickets and giving prizes to holders of winning tickets. The prize can be a fixed amount of cash or goods, but more commonly, the prize fund is a fixed percentage of the revenues from the tickets sold.
There are typically two-forms of lottery products sold: traditional lottery tickets and instant tickets.
Traditional lottery tickets are numbered tickets that are sold for regularly scheduled draws, most often weekly. On the draw date, random numbers are drawn either using a ball drop machine or electronically. Most lotteries that have moved to electronic draws still have ball drop machines as a backup in case of failures with the software solution. Once the numbers are drawn from the ball drop machine, they are entered into the lottery central management system (CMS).
The chances of winning a lottery jackpot can vary widely depending on the lottery design, and are determined by several factors, including:
The count of possible numbers
The count of winning numbers drawn
Whether or not the order is significant
Whether drawn numbers are returned for the possibility of further drawing
Instant tickets are numbered tickets from a pre-determined finite pool of outcomes. The most common form of instant tickets is the scratch card. Scratch cards are typically made of paper, with the outcome printed and hidden by an opaque substance that needs to be scratched off, hence the name of these tickets. The cards usually present the information in the form of a game, such as Tic-Tac-Toe, Bingo, Crossword or some other puzzle, to help add entertainment value. A variation of the scratch card is the break-open (also know as pull-tab) ticket in which, instead of scratching off an opaque substance to reveal the outcome, the player opens a perforated cardboard cover which is hiding the outcome. Since outcomes of scratch and break-open tickets are pre-determined, the cards do not need to be scratched or opened to be validated.
A barcode on the ticket can be scanned by the lottery CMS to determine if it is a winner or not. The scratching or breaking open is there for entertainment value to the player only.
The chances of winning on a scratch card are typically much higher than on a traditional lottery, but prize amounts are typically much smaller. The probability of winning on a scratch card can be calculated using the odds found on the back of the scratch ticket.
When it comes to lottery operations, it is critical that all parties are confident with the process. For everyone involved, including players, to feel confident, those running the lottery operations must acquire and uphold a secure environment that is documented and accessible. To address this, the Security Control Standard was put in place by the World Lottery Association and lottery organisations are audited against this standard on a regular basis.
Race and sports gaming
Race and sports wagering is also called sports betting. It is the activity of predicting sports results and placing a wager on the outcome. Although most sports betting wagers are placed against amateur and professional level sports, sports betting is sometimes extended to non-athletic events such as reality show contests and political elections, or sometimes to non-human athletics such as horse racing and greyhound racing.
Sports betting can be performed at the sports betting outlet in a casino, with bookmakers (also know as a sports-book) or online through a computer or mobile device. The types of sports bets include:
Money-line Bet
Spread Betting
Proposition Bet
Over / Under Bet
Parlay
Progressive Parlay
Future Wager
Money-line bets (also known as win bets) are bets in sports wagering. It is one of the most popular wagers that can be placed and is easy to understand. It is used in almost every sport a player can bet on and is a wager on who the player thinks is going to win a match, game or other event. It does not have a spread or handicap (explained below). It should be noted that the predicted winner, i.e., the competitor expected to win, pays lower odds then does an underdog.
Spread betting is defined as wagers that are made against the spread. The spread is a number assigned by the bookmaker which handicaps one team and favours another. This type of betting is similar to the Money-line win, in that the player is choosing which team he/she thinks will win, but there is a significant difference. A point spread is created to effectively make the two teams equal favourites in terms of betting. This means the player either backs the favourite to win by at least the size of the spread, or the player backs the underdog to win or lose by no more than the size of the spread. For example, the odds for this week’s National Football League games are posted and the point spread in the Washington Redskins versus Dallas Cowboys game looks like this: Dallas-4.5 Washington +4.5. The favourite team is associated with a minus (-) value, so Dallas is favoured by 4.5 points in this game. Consequently, the underdog is shown with a plus (+) value, which means Washington are 4.5-point underdogs. A wager on Dallas would be made if a player believe Dallas can win the game by 5 points or more. So, if Dallas wins the game 20-14, then the team not only wins by 6 points but also covers the 4.5-point spread as the favourite. However, if Dallas wins the game 20-17, then they win by 3 points and have NOT covered the 4.5 points, but Washington has because they stayed within the spread.
Proposition bets (also known as Props or Specials) are wagers made on events that are not related to the final outcome. Example events are: who will win the first round of a boxing match or which team will score first in a match.
Over/Under bets (also known as Totals) are wagers made on whether an outcome will be under or over an estimated outcome set by the bookmaker. For example, how many three-point shots will LeBron James make tonight?
– Over 2.5 – Under 2.5
In this example, notice how the Prop takes the form of a traditional game total wager. This is a simple wager to understand – if the person making the wager thinks that LeBron James can make three or more three-point shots tonight, bet on the over. If the player making the wager thinks LeBron cannot do that, take the under.
There are specific odds for both the over and under bet. Payments depend on the odds at the time the bet is made.
Parlays (also known as accumulators) involve multiple bets and rewards a successful player with a large payout. These types of bets are hard to predict because they involve making more than one selection as part of a single wager. For example, the player might place a single wager on what team will win the next five football matches. If the player successfully wagers, the payout is substantially higher than if the player had wagered on each game separately. The downside is that the player would lose his/her complete wager if the team he/she selected lost any one of the five games. Based on the number of selections, the parlay can receive a unique name. For example, “Double” when it contains two games, or “Treble” when it is composed of three games.
Progressive Parlays are similar to parlays in that they involve making more than one selection as part of a single wager. However, they differ from a Parlay in that a player will be rewarded even if some of the bets lose. If all bets are won, the player will be awarded the full payout which is not as large as a regular parlay but will receive a reduced payout if some of the selections within the parlay lose.
Future Wagers (also known as Outright wagers) are wagers placed on future events. Although all sports wagers are on future events, with a future bet, there is a long-term horizon measured in weeks or months. Future wagers usually are made before the season starts. Winning bets will not pay off until the end of the season. For example, the player might make a futures wager on a team winning the National Hockey League (NHL) Stanley Cup. The wager must be placed before the regular NHL season begins and the payoff will not be made until after the Stanley Cup playoffs end.
Online and mobile gaming
Online gaming includes all areas of gaming offered via Internet, mobile, wireless in-venue, and interactive-TV channels. The online gaming space contains all the different types of gaming that have been discussed thus far, i.e., slot games, table games, lottery, and sports betting
Online gaming has become one of the most popular and lucrative businesses present on the internet. Legalisation of online gaming varies based on the type of online gaming product and the jurisdictions in which they are offered. For example, purchasing traditional lotto tickets through online websites is legal in many jurisdictions. However, not all jurisdictions have legalised casino style gaming such as poker or slot games through online gaming websites.
Mobile gaming is online gaming on a mobile device such as a tablet or smart phone. There are two types of mobile gaming. The first is the online gaming at casino websites that can be accessed through a mobile device either through a website or through a mobile app. The second is in-venue mobile gaming which allows on premise casinos to add mobile technology and content to their existing offerings. Products are accessible to players on the gaming machines on the casino floor and on mobile devices inside the casino.
For the online and mobile gaming ecosystem, the player needs to be able to access the casino’s online gaming products. This can be done in two ways:
Browser-based
Downloadable application
If the player chooses to play through a browser-based casino website, the games are available through the player’s browser while on the online casino’s website.
If the player chooses to play through a downloadable application, he/she must first install the online casino’s software to his/her computer or mobile device. This option usually offers better graphics, sound and game play than the browser-based option. Then, in order to play at the online casino, the player must have a means of transferring money to and from the online casino. This can be accomplished by an electronic wallet (also known as a digital wallet), such as PayPal. When performing mobile in-venue gaming, some casinos have internal electronic wallets as part of the casino management system which are often associated to a player’s account. In this scenario, the player would deposit funds into or withdraw funds from the casino’s electronic wallet solution at the cashier booth.
To ensure online or mobile gaming is performed only where it is legal, geolocation, micro-technology and triangulation are used to confirm the location of the player. Geolocation is the estimation of the real-world geographic location of an object, i.e., the computer or mobile device a player is using to play online gaming. Micro-location technology is used for in-venue mobile gaming. This technology works by using the casino’s existing WIFI network or Bluetooth beacons to give accuracy of a player’s location to within a few feet. For out-of-venue online gaming, some jurisdictions have decided on mobile phone triangulation to confirm the location of players. This triangulation method determines which cellular towers are closest to the player’s mobile phone and ensures that the player is in the right geographical location. Mobile phone triangulation technology is accurate to within a mile of where the client resides. Other jurisdictions have decided to use Wi-Fi to verify geolocation for out-of-venue online gaming. This geolocation technology is accurate to within a few feet of the user’s residence.
Individuals looking to circumvent restricting online gaming to specific locations use technical measures such as proxy servers to try to bypass restrictions imposed by geolocation software. Some online gaming sites can detect the use of proxies and anonymises and block their access to the online gaming systems.
Key Concepts in the Gaming Industry
Progressive jackpots
A progressive jackpot is a prize or payout which increases each time the game is played but the jackpot is not won. A small percentage of each wager placed by a player on the game contributes to the jackpot award amount. The game that the progressive jackpot is attached to can be any type of game (e.g., mechanical reels, poker, etc.).
When the progressive jackpot is won, the jackpot for the next play is reset to a predetermined value, and resumes increasing under the same conditions. The progressive jackpot win is often associated with the highest winning combination on the gaming machine in which it is being played. In order to win the progressive jackpot, in most games, the player needs to have placed a maximum bet as the wager for the play.
Progressive jackpots are available both on VLTs and slot machines. There are three types of progressive jackpots:
Standalone progressive
Local area linked progressive
Multi-site linked progressive
A standalone progressive has a jackpot on the individual EGM. Only bets placed on that specific EGM increment the jackpot. Local area linked progressives are games within a venue that are linked together to contribute to a common progressive jackpot. This type of jackpot is usually found in a casino. This type of network can include as few as a dozen EMGs and as many as hundreds of these. Multi-site (also known as Wide Area) linked progressives link gaming machines from multiple venues to participate in the progressive jackpot. Due to jurisdictional rules being different, Multi-site linked progressives usually only link machines within the same jurisdiction, often across casinos operated by the same organisation. However, some examples of multi-jurisdiction progressive jackpots exist. For example, in July 2006, the Multi-State Lottery Association in the US introduced the first multi-jurisdictional progressive jackpot called Ca$hola. This progressive jackpot linked EGMs at nine lottery run casinos; three in Delaware, two in Rhode Island, and four in West Virginia. This linked progressive was replaced in 2011 by the Megabits jackpot and now includes two additional states: Ohio and Maryland.
A linked progressive jackpot solution adds some additional devices to VLT and slot machine ecosystems:
A progressive jackpot display or sign
A progressive jackpot controller
A progressive jackpot server
The progressive jackpot display or sign is used to display the current amount of the progressive jackpot.
The progressive jackpot controller is used by the venue to manage the progressive jackpot. The jackpot controller links the games contributing to the progressive jackpot and communicates the jackpot value to the progressive jackpot display.
The progressive jackpot server is used to manage multiple jackpot controllers and different progressive jackpot games that may exist across a venue. It will also monitor and collect all progressive related data to allow for analytics and auditing of progressive jackpots.
Random Number Generator (RNG)
The Random Number Generator is a computational or physical device designed to generate a sequence of numbers that lack any pattern, so they are random, or they appear unrelated. RNGs are used in gaming, statistical sampling, computer simulations and other areas where producing an unpredictable result is desirable. Any machine-base gaming involves an RNG.
The RNG is a vital part of all gaming machine operations. Where unpredictability is essential, such as in security applications, hardware generators are generally preferred over pseudo-random algorithms.
The RNG is certified by either an ITL or by the jurisdiction’s regulatory board.
The win selection flow
The selection process or the “did I win?” process is another key concept of the gaming industry. All gaming machines such as EGMs use some type of win selection process to determine and display the outcome of the game. This means if the player pulls a lever or presses a button, something happens on the screen and then there’s an outcome that says “Yeah! I’ve won!” or” No, I’ve lost!”.
What is also important about the selection process is that it can be performed on the EGM itself or on a server. In some cases, the whole process from “spin the wheel”, “get a response”, “you won or lost” is done on a standalone EGM.
The technology being used and the specific jurisdictional rules of where the game is being played will influence the selection process and whether it is performed on the EGM or on the servers.
This selection process will involve the following:
Start of spin
A raw random number is generated by the RNG
The raw random number is scaled to a usable number
The number is mapped to a game element (e.g., is it a star? is it a 7? is it Wheel of Fortune?)
There is an evaluation of the outcome of the results of that random number generation
The prize is awarded to the player with that outcome. Either credits are taken away from the player in the case of a loss or credits are given because of a win.
There is a display of the outcome to the player
The prize is paid, if applicable
End of spin
Player privacy and geolocation
Privacy laws in most jurisdictions mandate that any player’s information being tracked, whether for responsible gaming or player loyalty program purposes, adheres to the storage and use of personal information regulations set forth by those laws. An example of testing player privacy is verifying that the solution makes the player information available to only those that should have access, and that any such information is encrypted when being transferred between devices and systems.
Some responsible gaming and player loyalty programs require knowing where the player is located. Testing this function consists of ensuring the geolocation functions accurately restrict play based on the rules mandated by the location from which the player is playing.
Regulatory commissions, jurisdictions and associations
Compliance testing is also called jurisdictional testing. Each jurisdiction has its own rules, regulations, guidelines (also known as regulatory or jurisdictional specifications or rules) that must be tested. This testing is usually performed by an ITL.
In the United States, there are over four hundred regulators and jurisdictions. Canada has at least one per province. South America has at least one jurisdiction per country that has legalised gaming. Europe, Asia and Africa also usually have one jurisdiction per country. Germany has lottery companies by province. Australia has at least one per state. Within these jurisdictions, there is usually an organisation that is responsible for issuing licenses and regulating the licensee or the people who have the licensee. These organisations are typically known as licensing authorities.
Every jurisdiction controls the potential manufactures who need a licensee to operate in that jurisdiction. Manufactures cannot legally operate in any jurisdiction where they do not have a licensee. If a product fails compliance testing, it must be fixed and returned to the ITL for certification testing until it passes 100% of the mandatory certification tests. The product can be returned many times before it passes the compliance tests.
Before gaming products are ready for compliance testing, a full range of gaming QA testing must occur. Some examples of test types and test techniques that are done for the gaming industry includes exploratory testing, functional testing, regression testing, pre-compliance testing, system integration testing, performance testing, penetration testing and failover testing.
Gaming Industry Metrics
Background
Gaming industry testing uses many of the common test metrics. However, there are a few that are specific to the gaming industry.
First pass percentage
First pass percentage identifies the percentage of games that receive certification from the ITL on the first submission of the product.
The importance of receiving a first pass for a gaming product is related to both product cost and its time-to-market. If the product does not receive a first pass, there are extra costs for additional development, testing and product certification. A gaming product that does not receive a first pass is delayed from entering the market until it is certified.
Escape compliance defects
These metrics measure data relating too escaped defects that do not comply with the jurisdictional rules or regulations and are found by the ITL or in the field.
The resubmit factor is the number of times a game must be resubmitted to the ITL to pass certification testing. For example, if on average each game is resubmitted 4.5 times to achieve certification, the resubmits factor would be equal to 4.5.
The number of revocations tracks how many games have been pulled from the field per time period, due to escaped compliance defects. For example, if two games have been removed from the field in a year, that would mean two revocations for the year. If a jurisdiction asks for a game to be removed due to an escaped compliance defect the manufacturer has a limited amount of time to remove the game.
These two metrics are important because having escaped defects in a jurisdiction can impact a manufacturer’s right to be in that jurisdiction, negatively impacting their brand, making them lose revenue if the EGM and table games is not working on the casino floor. There are a fixed number of EGMs and table games in any casino. Manufacturers fight for floor space amongst themselves, so a revocation might also mean that a manufacturer loses that floor space to a competitor.
Gaming Software Development Lifecycle
The gaming software development lifecycle overview
The Gaming Software Development Lifecycle follows the sequential development model.
Game Concept and Design is the first phase of the gaming software development lifecycle. It starts with a game idea that is storyboarded and is reviewed. Game and sound designers, artists, video, and gaming experts, software architects and game developers, and gaming jurisdictional experts create a game prototype. The prototype is then scrutinised for innovation and playability by the targeted audience focus group. This group may be composed of internal (IT-professionals), external (non-employees, sometimes non-IT professionals), or a mix of both resources. The Game Concept and Design phase is an iterative process. The Game Concept and Design phases’ ultimate deliverables are documents which become the blueprint for the development team, artists, mathematicians, and sound designers.
The Game Concept and Design documents include the following:
Game Concept
Game and Technical Design
The Alpha phase, not to be confused with alpha testing, is next. During this phase, game play functionality is developed and implemented, math functionality is completed, video and audio components are partially finished, and the game contains the major features. Black-box testing, especially functional, usability testing, exploratory testing, regression testing, math testing, and RTP testing occur.
The Code Complete phase is next. All features, audio, video and math components are finalised. At this phase, code is no longer added to the game, unless a change is needed to fix defects. Standard black-and white-box testing are typically performed at this phase. The emphasis is on test automation, testing for memory leaks, confirmation testing, and regression testing.
The Beta Build phase, not to be confused with Beta Testing, continues until no failures occur that prevent the game from being certified. Pre-certification testing is performed by the internal gaming quality assurance test team to assess the game versus the requirements of each jurisdiction. This phase is not a formal certification test cycle. It is a precursor to the ITL certification testing. Any defects discovered at this time will be corrected and the new builds are tested, and regression tests are performed.
The Release Build phase is the one that is sent to the ITLs to ensure that the game complies with the requirements of each required jurisdiction. This build receives the final certification sign-off, which allows the game to be sent to casinos or be made available online. If the game fails this certification, it is sent back to the game developer and the process starts over.
The role of the independent test lab (ITL)
Once the pre-certification phase is completed by the machine manufacturer, the game is ready to be certified by an ITL (also known as the Authorised Test Facility). If this is a game that will be played in North America and in Australia, it must be tested for all applicable jurisdictions which means approximately 450 jurisdictions for these two parts of the world.
Once the ITL has tested the game for all applicable jurisdictions, if it fails in any of the jurisdictions, the game is returned to the machine manufacturer or game developer who make the changes in the game or in the EGM and return it for another ITL certification test.
The only way to be an accredited ITL is to be accepted by each gaming regulatory commission. This is a lengthy and costly process and thus there are only a few ITLs who can certify games world-wide. A few of the jurisdictions have government-based certification test labs that play the role of the ITL.
The role of regulatory commissions
Once the ITL has certified a game, the regulatory commission allows the game to be played in all casinos in their jurisdiction. However, the regulatory commission will revoke or pull a game from all its casinos if a major field issue arises. A major field issue is usually a defect that stops the game from playing, provides erroneous payouts or deviates from any of the rules of engagement that are required for certification. The machine manufacturer will have to immediately remove that game from every installation in the jurisdiction.
There are also minor field issues that will force the machine manufacturer to modify a game that is in the field, within a given timeframe. In this case the game must be certified again at an ITL and approved by the regulatory commission.
While it is certainly true that the roles and responsibilities of the tester and the business analyst are different, it is also true that their activities are complementary; work done by one group may greatly affect, either positively or negatively, that of the other. This is especially true in acceptance testing which is performed to assess the system’s readiness for deployment and its use by the customer (end-user). Good collaboration between business analysts and testers is particularly important for a proper consideration of the business implications at this test level.
Business Goals, Business Needs and Requirements
Business analysts first must understand the organisation’s overall business goals and identify current business processes and stakeholders. Once that is done, they describe specific business needs and determine a business case that addresses those needs. Once this high-level work has been completed, requirements can be elicited for the business solution that shall be developed.
Business goals, business needs, business requirements, and product requirements describe, at different levels of abstraction, what shall be achieved. In Agile development, the same principles apply, but different terms may be used (for example features and user stories).
In this document, the term “requirements” refers both to business requirements and to product requirements.
Requirements / User Stories, Acceptance Criteria and Acceptance Tests
During requirements elicitation, business analysts and testers (possibly together with developers) should begin to create specific acceptance criteria and develop acceptance tests as a joint effort. This ensures that there is a mutual understanding of what “acceptable” means from the business, development, and testing perspectives, right from the beginning of the project.
Acceptance criteria relate directly to a specific requirement or user story. They are either part of the detailed description or an attribute of the related requirement. If user stories are used, acceptance criteria are part of the user story’s definition and extend the story.
In all cases, acceptance criteria are measurable criteria, formulated as a statement (or a set of statements), which can be either true or false. They are used to check whether a requirement or user story has been implemented as expected. Acceptance criteria represent the test conditions which determine “what” to test. They do not contain the detailed test procedures.
Acceptance test cases are derived from acceptance criteria. These tests specify how the verification of the acceptance criteria should be performed.
The Importance of the Quality of the Requirements
If acceptance criteria and tests are based on requirements, user stories, and/or acceptance criteria that are vague or ambiguous, it is likely that testers will make assumptions about stakeholder expectations and business needs. In this case, the resulting acceptance tests may be flawed. This will lead to rework or, even worse, the running of invalid tests, thus creating unnecessary costs as well as risks and uncertainty about product quality assurance.
It is critical for testers to work closely with business analysts to make sure that requirements are clear and well understood by all stakeholders concerned. Ambiguities should be resolved and assumptions should be clarified so that the resulting acceptance tests are valid and are a meaningful way to determine the product’s readiness for release.
In Agile development, the INVEST criteria define a set of criteria, or checklist, to assess the quality of a user story. These may be used by business analysts / product owners, developers, and testers to ensure the quality of user stories.
Business Analysis and Acceptance Testing
Too often, business analysts and testers work in their own separate silos, which can lead to misunderstandings about business and customer expectations. Those misunderstandings may stay hidden until the release approaches. By taking advantage of the complementary skills and by working together, business analysts and testers can positively affect the development process. This can be accomplished both by considering acceptance criteria and acceptance testing as early as possible and by coordinating efforts to make sure that the product has been tested appropriately prior to release at acceptance test level.
Relationship between Business Analysis and Testing Activities
The following are the main elements of the business analysis activities:
Strategy definition
Management of the business analysis processes
Requirements engineering in business analysis
Solution evaluation and optimisation
The business analyst is responsible for identifying business needs of stakeholders and for determining solutions to business problems with the aim of introducing change which adds value to the business. An important aspect of the business analyst’s role is to establish consensus between quality engineers, testers, developers, system integrators, product managers and project managers.
A test process consists of the following main groups of activities:
Test planning
Test monitoring and control
Test analysis
Test design
Test implementation
Test execution
Test completion
Quite a few of the associated activities and tasks relate to both business analysis and testing. The following examples illustrate the relationship between the two disciplines in the context of acceptance testing:
Requirements engineering in business analysis vs. test planning, test analysis and test design:
During the requirements engineering activities in business analysis, business analysts prepare detailed business and product requirements. These requirements are part of the test basis for the test planning, test analysis and test design activities, as testers define their objectives and plan their work, evaluate the specifications and requirements, identify test conditions and design test cases and test procedures.
Testers can contribute to the definition and verification of acceptance criteria as part of test analysis and test design activities. Working together, the two roles ascertain that there is proper understanding of the solution and agree on the appropriate approach to acceptance testing.
When requirements change, business analysts and testers can work together to assess the impact of the changes.
Solution evaluation in business analysis vs. test implementation, test execution and test completion:
During the solution evaluation phase in business analysis, business analysts support test implementation and test execution activities. They review the testers’ procedures/scripts, clarify issues and potentially help with creation of test data to support business-related tests.
Business analysts can assist with the implementation and execution of the acceptance tests. They may also support testers by evaluating test results. In addition, they may assist testers in test completion activities.
There is a strong and symbiotic relationship between the two roles and their respective activities, starting at the very beginning of a project and continuing until acceptance or release of the solution.
Collaboration between Business Analysts and Testers in Acceptance Testing
The common goal for business analysts and testers is to support the production of products with the highest possible value for the customer. Given their position within the organisation, business analysts and testers have various opportunities to collaborate during the acceptance testing activities described in the previous section. Apart from joint discussions and reviews of generated artefacts, business analysts and testers collaborate in other areas. For example, collaboration on test planning based on risk analysis is a good opportunity to ensure that the appropriate test cases will be developed and prioritised.
In addition to the direct benefits of working together and supporting each other’s efforts during acceptance testing, there is an important opportunity to cross-train team members. The more testers know about business needs and stakeholder requirements, and the more business analysts know about structured testing, the more likely the two groups will understand and appreciate each other’s work and better collaborate within the project.
How Acceptance Testing Can Drive the Development Process: ATDD and BDD
The wide acceptance of Agile software development practices has influenced how acceptance testing relates to requirements elicitation and other business analysis activities. In sequential lifecycle models, acceptance test analysis, design, and implementation are activities to be handled by the testers after the requirements are finalised. With the Agile lifecycle model, acceptance criteria and acceptance test cases are created during requirements analysis, requirements refinement sessions, and product backlog refinement. This allows the implementation of the “Early Testing” principle by using the design of test cases as part of the requirements definition activities.
In the following two approaches, acceptance test analysis and design are formally part of the requirements engineering process:
In Acceptance Test-Driven Development (ATDD), acceptance tests are produced collaboratively during requirements analysis by business analysts, product owners, testers and developers.
Behaviour-Driven Development (BDD) uses a domain-specific scripting language, Gherkin, that is based on natural language statements. The requirements are defined in a ‘Given – When – Then’ format. These requirements become the acceptance test cases and also serve as the basis for test automation.
Both of these approaches engage the entire Agile team and help to focus the development efforts on the business goals. The approaches also treat the acceptance test cases as living documentation of the product because they can be read and understood by business analysts and other stakeholders. Acceptance test cases represent scenarios of usage of the product.
The two approaches are similar and the two terms are sometimes used interchangeably. In practice, BDD is associated with the use of Gherkin to support writing acceptance tests, while ATDD relies on different forms of textual or graphic acceptance test design. For example, the graphical representation of application workflows may be used to implement a visual ATDD approach.
