Category: agile

acceptance testing, agile, AI, BDD, bot, ci/cd, container, continues integration, development, devops, digital, speed, Technique, Tools, הראל, הראל לירם, לירם, לירם הראל

7 tips for improving load speed

Plan for performance

Are you building a new website? Be sure to discuss the importance of performance early on and set targets. That way, you have a faster load speed from the beginning and don’t have to implement fixes later.

Step 1: test, step 2: test…

Are you seeing a pattern here? 😉 Testing is crucial! Before you launch, load and test your website multiple times to make sure you can handle the traffic of real site visitors. This is especially important for sites with complex hosting, such as load-balanced configuration.

Implement some “quick wins”

To be clear, there’s no “get fast quick” scheme for site load speeds. But there is a tried-and-true template that will put you ahead of the curve. That includes making use of modern image formats, enabling compression on the server via Gzip, and leveraging browser cache. Find some more low-hanging fruit here.

Careful of your images!

Good websites have great graphic content – but they also take into account how images impact load speed. You can improve image performance by considering file formats, image compression, and lazy loading.

Think of the mobile visitors

More and more people surf the web on their phone these days, which makes mobile-optimized sites a huge priority! Since mobile users tend to use slower, less stable Internet connections, Accelerated Mobile Pages (AMPs) are a great way to get them content faster.

Prioritize above-the-fold

First impressions matter – and your above-the-fold content can make or break them! Consider inline styling for above-the-fold, then loading your code in chunks. This type of asynchronous loading can create a faster perceived load time for the user.

Assess your external scripts

Third-party scripts are a great tool – but can make your website feel a little crowded. Assess the performance of external scripts on your site load speed, and replace or remove those that are negatively impacting user experience.

agile, BDD, Management, Specslist, Talent

Salary structure in an agency

Perks and benefits that save employees money in the long run are always a valuable addition to a paycheck. Addition being the keyword here.

Because no amount of pizza parties can supplement the 10% increase in salary that people could get at the other agency across the street. Except, that’s not the case, the statistics surrounding this, point in the exact opposite direction:

  • 32% of people polled in the US would take a 10% pay cut to work at a company where they like the culture
  • 58% of workers will stay at a lower-paying job if it means having a great boss
  • And 60% of workers would even take half of the potential paycheck if it meant working at a job they love

So if culture makes up for the differences in salary between your agency and the agency next door, how do you structure the salaries in your company to both attract and retain top talent?

  • Don’t buy stars, build them – Have a partnership with the local media and technical schools that provides internships and part-time positions for promising students. If you follow our onboarding tips and you build a functional onboarding program, after a couple of weeks, your time investment in onboarding them should already be paying you back. And in a few months? You might just have your hands on your newest superstar.
  • Have a clear progression path – be upfront and transparent with the salary structure. It will eventually become the biggest motivator for the employees in the lower tiers. If you split your progression path into layers where everyone gets paid the same, you can skip long management discussions like: ’’Is a Senior Backend Developer with 4 years of experience worth the same as a Senior Art Director with 5?’’ An example of how to structure your progression path could be:
  1. Intern > unpaid, but gaining real-life skills and experience from an agency by working on real projects
  2. Trainee > paid, part-time or full time; self-taught, certified or freshly graduated
  3. Apprentice > Same credentials as a trainee, but with some successful commercial projects
  4. Junior > Proven 1-3 years of experience with commercial projects
  5. Senior > 3+ years of experience with commercial projects and proficientwith project management and delegating tasks
  6. Management > If you’re doing linear progression, this step is simple. But if you want to do non-linear progression, it’s worth differentiating at management level. a. Senior members with multiple specializations and experience with managing teams b. Senior members with extra non-managerial responsibilities (product development, decision making, etc.)
  7. Equity tier > Management whose investment with the company is substantial enough to warrant equity in the company
  • Promotions, raises and employees who feel undervalued – if you adopt the aforementioned salary structure, your employees should have a clear overview of where they fall and what they need to achieve to move up to the next salary level. But as it goes with highly ambitious people, you will always have individuals who take on more than their fair share of responsibility and then don’t feel adequately compensated. The answer should be obvious. If the employee performs above the set expectations, has the data to back it up, and asks for an increase in pay, they should get one. Sadly, when working with more than one person, it will never be that easy. Ben Horowitz summed it up the best in his class on Y combinator – how to start a startup.

A point he brings up is: If you give that employee a raise, will you give everyone else who is also performing well a raise as well? What about the employees who are performing just as well, but their personality prevents them from asking directly?

Apart from being approachable overall, managers and senior agency members can adopt these two methods to focus these conversations and help employees feel more valued and heard:

1. Monthly walk and talk:A manager and employee go for a half-hour walk outside of the office, talking about current projects, plans for future projects, the progress of the employee and any problems they might be having

2. Yearly progress conversation: Performance reviews are usually seen as a negative process because of the negative associations that people usually have with them. Walk and talks remove the need for quarterly performance reviews at a scary meeting room table.

But a walk and talk is not really the place to sign contracts and obsess over spreadsheets. So how about a yearly progress review, close to the end of the year, talking strictly about the employee’s progression path and salary?

That way, both current problems can be addressed from month to month, and larger issues or achievements can be accumulated over time.

agile, BDD, Management, Specslist, Talent

Non-linear progression

When hearing the words ’’non-linear’’, if your mind immediately jumps towards video games, you already sort of get the point.

In a non-linear game progression system, you start at the same spot as every other player. But when you arrive at a crossroads, instead of going straight down the first path like you usually would, you get to choose if you want to go left, right, or even take a step back and see if you can get to your current position again, by taking another path. This progression helps you pick up new skills and new experiences that will make the path ahead much easier.

This is also how the current trend in career progression looks. Companies no longer expect people to stay in the same career path for decades, slowly working their way up the corporate ladder. This rings especially true for agencies, where skills from different career paths transfer almost seamlessly and complement each other with a broader outlook on the problems being solved.

As an example, if you have a frontend developer who discovered she likes designing more than she likes coding, you should give her a chance because:

  • She already knows the limitations that code can have on some designs
  • She can design with systems and reusable assets in mind
  • She can give better estimates on project length and the overall development time
  • If she wants to progress further into something like art direction, the added coding skills are always a plus when communicating to both clients and developers alike

If your agency has people who have invested in their craft to the point where they are considered experts, top talent, or masters, their progression will eventually hit a plateau.

And while just existing at the top and using your skills to their full potential is a fantastic feeling… ultimately, the need for self-improvement and innovation that got them to the top of the talent pool will make them want to progress further. But you can’t really go further up than the top, so where do you go?

This is where people start considering switching jobs or pursuing entrepreneurship because it seems like the only challenging way forward.

The classic solution to this “problem” is to promote them to the management level. Clearly, if someone is performing exceptionally well as a specialist they will automatically become an exceptional manager… Right?

The solution is not always that simple and pushing someone to become a manager (or a manager of a bigger team than before) is not for everyone. Some top talent enjoy being a specialist and would rather spend their time performing their tasks, than managing a team.

“In a hierarchy, every employee tends to rise to his level of incompetence.”

– Laurence J. Peter, Author of The Peter Principle

The previous quote refers to what is known as the Peter principle, a concept of management developed by Laurence J. Peter. The principle suggests that people tend to get promoted outside of their skillset and competence, based on previous success.

Meaning: Your best front-end developer is first and foremost… a front-
end developer. Having 10 award-winning projects under his belt does not make him an instant candidate for managing the next project. That requires knowledge of front-end and an additional management skill set, lack of which could lead to disaster down the line.

The modern solution to the problem is working with non-linear progression and promotion. Instead of the career path only going one way – towards management – you can set an alternative path. This could be anything from giving your top talent more influence on projects or a seat at the table when tough decisions are made to simply giving more freedom to perform tasks their own way. Once you start thinking outside the box you’ll be amazed at the possibilities there are for non-linear progression.

And the result?
Happier top talent that gets a truly unique position at your agency, which they won’t be able to find anywhere else.

At SQAEB, most of our junior employees start out in the SWAT department, helping our users with day to day issues. This helps them naturally and quickly get an overview of all the other departments, the products, and how everything fits together. Later they can choose to transition into newly opened positions in the company that they find interesting or get places in completely new positions based on their specializations.

agile, BDD, Management, Specslist, Talent

Talent Investment

You have to spend money to make money. And you have to invest in top talent to retain top talent. Achieving maximum focus in an office setting where a million things are gunning for your attention is tough.

All of that can be managed with a good work culture and processes. But if you don’t have the right equipment and tools, you’ll never be as efficient as you could be.

Maybe a chair is not comfortable. Maybe you can still hear your sales team in the other room, even with your headphones on. Maybe you found a SaaS tool that would save you hours upon hours of repetitive tasks.

If someone asks for a new keyboard, new tool, or new screen, it’s never a good idea to dismiss them right away. The person asking rarely brings up an issue like this on a whim, it has to be premeditated in some way, and that means that the problem they are facing is a recurring one.

“The way management treats
their associates is exactly how the associates will treat the customers.”

– Sam Walton, Founder of Walmart

A one-time investment, no matter how large, is actually pretty small when looking at it as a long term investment in focus and productivity. If an agency shows that it cares about its employees in all the ways that matter, the employees will return it multiple times over. Here are some small or large things in no particular order that could make or break an employee relationship with the company:

  • IT equipment. If you ask someone to work in front of a computer 8 hours each day, you better make sure they have the proper equipment to do their job. This includes everything from computer equipment to noise- cancelling headphones and online tools to do their job.
  • Chair and desk. This one is connected to the one above; spending a third of their day in uncomfortable working conditions will severely hurt their productivity and health.
  • Coffee, refreshments and snacks. We know it might not sound like much, but making sure that your employees have access to all the basics like coffee, cold water (or soda) and some fruit can drastically increase their productivity and improve health.
  • Indoor climate. The stereotype of a developer might be: someone sitting in a dark basement with a hoodie on – but nothing could be further from the truth if you want them to be productive. Proper lighting, some plants and good ventilation are all tiny details that have a huge impact.
agile, BDD, Management, Specslist, Talent

Talent Professional growth

A promotion: While most talented people love what they do, as they repeat the same tasks day after day, eventually, they will find ways of improving the process or get ideas for new ventures that the team should pursue. And there is only so much one can do from the bottom of the corporate ladder. Career growth is a key part of goal setting strategies for high performersand agencies need to provide these opportunities if they want to retain their top talent. Otherwise those people might look for those higher positions elsewhere. Please note, that a “regular” promotion is not always the best option; we’ll cover that later in our post “Non-linear progression”.

A raise: Usually going hand in hand with a promotion. However, while every promotion should come with a raise, not every raise has to come with a promotion. Many people are not after the responsibility that comes with
a promotion, they just like what they do, and so they take on more tasks, spend more time at the office or even work weekends. But maybe they aren’t looking to delegate their tasks to their would-be replacements. Maybe they just want to feel like their extra time is seen as valuable by the agency. And seeing as time is money, sometimes the answer is as simple as that.

While all of the above will probably make your agency employees happy and get your agency valuable, educated and dedicated employees for a long time
to come, there are also smaller ways to improve productivity faster.

agile, BDD, Management, Specslist, Talent

Talent Personal growth

Courses and conferences: There are always new books and courses popping up, covering the latest and greatest developments in the industry.

If your top performers ask about you helping fund their education, it’s one of the best ways to show them that you are counting on them in the future.

Maybe there is a developer conference coming up that would help them meet some like minded people and gather industry knowledge?

While it may seem like a big investment to send one or multiple developers away for a few days, the new knowledge and energy they bring back will pay dividends now as well as in the future. If they have valid arguments for going, why not give it a shot?

Schools and degrees: A similar approach to the one about courses and conferences, to an even higher degree (forgive the pun), should be taken if an employee asks about the possibility of returning to school.

Maybe they got this job straight after finishing their bachelor’s degree. Maybe they want to go for a manager position and think that an MBA would greatly improve their outlook.

Or maybe they want to slowly transition to another position, but wish to stay at the agency. Customer lifetime value and return on investment are some of the most important metrics that agencies need to keep an eye out. But try
to imagine the “employee lifetime value”, of someone who you helped put through school.

agile, BDD, Management, Specslist, Talent

Personal and professional growth

Every movie about an office work environment has managed to, in one way or another, demonize the monotony of sitting at a cubicle doing the same work every single day. And who can blame them? Doing the same thing over and over again is widely referred to as the definition of insanity.

No one wants to feel like they aren’t progressing in their job. And this rings especially true when we are talking about top talent. If someone wants
to stay at the top (where you probably want to keep them), they need to continually have an eye on the newest developments in their field.

The information gathering and processing is on them – allowing for an environment where they can test new ideas, that’s on the agency.

There are many ways to help talented employees fuel their passion for their work. Every person is looking for something different, but we have a few ideas that should be universally interesting for most people.

agile, BDD, Management, Specslist, Talent

Is ’’When and Where’’ Important?

Allowing for a full five-day remote work schedule is not something that can be implemented instantly, it’s something that agencies have to build towards over time.

For a large portion of agencies, a full week of remote work might not even make sense at all. But giving people the freedom to work from home as needed on special occasions can remove a lot of unnecessary stress. If a person needs to take care of some errands, look after the kids, or maybe they are not feeling well enough to drive to the office, but well enough to work, why not have the option of working from home?

Let’s say you have a single developer dedicated to taking care of your agency website. He has tasks that he doesn’t actively collaborate with anyone else on. He gets a mockup of the website, some copy, and gets to work. He might also be actively trying to sell his apartment. In most companies, this would mean that he has to run back and forth between the apartment and the office, sometimes multiple times a day, to deal with the buyers, real estate agents and contractors. But does he really have to?

Would it not be more comfortable for him to stay at home and work between meetings? And would it not make it easier for his team members and managers not to have to keep track of his travel schedule? And if the work gets done in the right time frame, does his physical presence at the office really matter? I’ll discuss this further in “Is it time to go fully remote?” post.

SQAEB TIP

At SQAEB, everyone has a setup that allows for secure remote work, and in case of sickness, family emergencies, schoolwork or other unforeseen events, they are always welcome to work from home. We give people the benefit of the doubt / assume positive intent, and so far, it has always paid off.

agile, BDD, Management, Specslist, Talent

Talent Freedom

Freedom is often hailed as the ultimate solution to happy employees. But most people have an easier time being creative when there are some restrictions in place.

Example: If your agency needs you to write as many slogans as possible selling pineapples in the next 10 minutes. When do you think you will produce more? A) If the 10 minutes is the only restriction. B) If you have a 10 minute restriction, you cannot use the word pineapple and all the slogans have to be under 10 words or less?

Studies show that B is the right answer – even though you have more freedom in A. Sidenote: We tried it at our office and we are currently considering a new venture in ’’Spiky yellow fruit’’ advertising.

So does this prove that freedom may not be the answer to an infinitely creative and productive workplace culture?

Of course not – because we had the freedom to choose those restrictions.

Client expectations and agency needs dictate the tasks that have to be solved. Every agency also needs to have some time and budget restrictions to prevent a project getting out of hand.

Other than that, the freedom to solve the problem in any way possible is one of the most significant benefits you can grant your employees:

  • The most efficient way to a problem takes all the learning and experimentation out of the process
  • Using less billable hours and achieving maximum efficiency will inevitably mean that the client should probably expect cookie-cutter deliverables instead of innovative solutions
  • If there is a framework, guideline or brand book for everything, proposing new solutions and approaches might be perceived as too much of a hassle to even suggest

If you find the perfect balance in the above, you should have the How and Why of task management covered. But freedom in the workplace is a complicated thing. The How and Why are questions that have to be answered or the work will never get done. But why not take more weight off of people’s shoulders by not having them stress over the When and Where as well?

agile, BDD, Management, Specslist, Talent

NURTURING AND RETAINING TOP TALENT

Hiring and onboarding new employees is one thing. But as we know, the costs of employee turnover is high. If you don’t work on having a great environment where your employees thrive, then it’s going to be very costly for you to keep replacing everyone.

Employees changing jobs is impossible to stop – especially in the tech industry – but there are things you can do to keep your turnover rate low.

This post could just be called ’’culture in the agency space’’ because that is the true key to acquiring and keeping top talent.

But what is company culture?

The 17-word, aka the short answer: Company culture is the combination of all the values, social interactions, and psychological behavior in an organization.

The 340-word, aka the long answer:Company culture is hard to define in specific terms, because unlike most essential things in business, it is entirely intangible, a feeling. Branding is closely intertwined with culture in every interaction that the company makes with any of its outside stakeholders. And if you want your brand to be consistent across all channels, you have to work towards a work culture that aligns with your corporate messaging.

A brand is a reflection of your company in the minds of your stakeholders.

That is why it takes on new forms in every piece of content shared on social media, every meeting with a possible client, and every shared lunch break with Debbie from the agency next door. A brand consists of many moving parts, some tangible, some not. The tangible can be boiled down to visual identity, messaging, and imagery, if need be. These can all be changed with a new set of guidelines, a new designer, or a new marketing department, but how do you control a culture?

Culture is not just a code of conduct, communication strategy, or a list of processes. Company culture includes all the small details:

  • The tone of voice the CEO uses to address a reporter while discussing a new acquisition
  • If your employees feel comfortable to talk about non-work related issues with their manager
  • If the new sales intern feels like waking up in the morning on his second week on the jobAnd that’s why culture is one of the hardest things to get right in an agency, as it can not be acquired, mandated or forced.

Culture has to be built and continuously monitored and maintained.

You can tell a lot about an agency culture:

  • In the way, your company treats employees, customers and the surrounding community
  • In the degree that your employees are committed to the company values and goals
  • By how comfortable employees are with innovating, making decisions and expressing their opinions
  • In how information is broadcasted from one department to another and from the higher-ups to the lower-level employees

agile, BDD, Management, Specslist, Talent

Day one onboarding

There are many things a person needs to know on their first day at a company. And there are a lot of things that they will definitely not remember. To prevent information overload, it’s preferable to keep some essential things for the rest of the week so the fresh hire will pick them all up eventually. So what should they know on their first day?

  1. Give them an “onboarding buddy”. This should be someone from their team, who they can ask any and all questions to, without feeling like you are bothering them
  2. The values or the ’’WHY’’ of the company
  3. The names of their closest coworkers
  4. The tech stack your department is using
  5. Where to find the best coffee machine in the building, as well as any other refreshments they can get (fruit, cold water, etc.)
  6. How the company intranet or CMS works
  7. The most efficient way to get to their desk
  8. The information and communication flow of your company (emails, chat, phone calls, etc.)
  9. Where the bathrooms are (you’d be surprised how often this is an issue)
  10. What task management solution your team uses to keep track of tasks
  11. When lunch is
  12. Their first real work-related task

That’s about it, any other information would probably be too much, and
as we all know, if you go for a handshake tour with every department immediately, you forget the first person’s name while shaking the third one’s hand.

agile, BDD, Management, Specslist, Talent

Onboarding that rocks

Onboarding a new person to the team is a masterclass in taking your own medicine for a lot of agencies. Every good agency prides itself on an in- depth understanding of user journeys and user experience, but what is the experience of joining your agency like?

Placing someone behind a desk, giving them access to your password manager, and asking them to start developing right away is the equivalent of ordering a pizza and giving the delivery guy just your zip code. It takes so much more, and a good onboarding experience can make or break your company’s ability to foster new top talent.

agile, BDD, Management, Specslist, Talent

Interview a talent

Generally, tech companies started adopting ’’a multiple interview approach’’ that not only gives applicants a coding test or some homework, but also goes over their background and culture fit in the same depth. More and more agencies are now doing the same. This is where our hiring journey once again splits into two paths, this time, based on if you chose the internal hiring strategy or the headhunter/recruiter strategy.

The recruiter can take care of the searching, first impressions and the technical fit, but you should always have the most promising candidates meet the current team for a short and sweet meet and greet before you consider hiring them.

If the agency conducts the entire hiring process in-house, there is a lot of leeway in the process. Try new approaches and strategies, and eventually, you will find what works for you. But if you want a hint from a company that put culture first and has been doing so for 3 years, here’s how we do it at SQAEB:

  1. Collaborative effort to identify skills required. Once we are sure we need a new addition to a department, the team goes over the exact skills we are looking for. This ensures that the team knows which new skills are coming in, instead of a manager deciding it themselves.
  2. Job posting. When the manager has the final job posting ready, it is posted and shared online internally as well as externally. We know the value of a good network, so employees from all departments are asked to share it with anyone they might think is a good fit. To help gauge personality in the first screening process we usually ask for a short video introduction, along with a resumé, just to get an idea of who you are as a person even before we meet you.
  3. Screening of candidates. As soon as we have enough candidates, the first screening process starts. This consists of sorting out any that does not have the required skills or did not adequately show that they would be a good cultural fit.
  4. First interview. All candidates that pass our first screening are invited
    to a first interview. The purpose of the first interview is to get to know them as a person and figure out if they would be a good cultural fit. This includes having a current team member talk to them for 10 minutes one- on-one, without those involved with the hiring present. If the personality is a match to our culture, they are given homework and invited to a second interview.
  5. Homework. While the first interview is focused on the cultural fit, the second is about technical skills. And to judge that, each candidate is given homework to complete before the second interview. This consists of various work-related tasks where they have a chance to showcase their skills. The homework also includes writing a movie review. This is an added curveball to see how they approach problem solving of tasks they probably haven’t done since high school.
  6. Second interview. We have the second interview to go over the homework and technical questions. This is where their skills are assessed and the main goal is to ensure that the chosen candidate has the necessary skills to handle the tasks they would be given in the position.
  7. Hiring. After the second round of interviews it is often clear which candidate is the best cultural fit and whether or not they have the necessary skills.

Now that you’re done recruiting and have hired the right person, the real work starts: onboarding. Hiring the right candidate is one thing; but if you don’t manage to give them a proper onboarding experience they will not perform as well as they could. Onboarding is the first step towards nurturing top talent.
agile, BDD, Specslist, Talent

Talent, Takes one to know one

Agencies have a lot of ways to get new talent in the door. You might do all the recruitment in-house, outsource it to a headhunter/recruiter or grow to a point where a dedicated HR department or in-house recruitment person is the way to go.

But no matter which option is the most viable for you, always keep the cultural fit in mind. You might find out that the person with the most extensive resume might be too far in their career to adapt to the workflow that works for the rest of the team. There are also cases of people with less impressive qualifications, who fit in so well with the rest of the team, that they hit the ground running and start producing work way above their estimated skill-level right away.

Making your agency a cultural paradise for top talent pays off in more than one way:

On one hand, you will attract those who have already proven to be top talent, which can give the quality and speed of work an instant boost. And if they are the ones who come to you looking to join, you’ll have a much larger talent pool to choose from.

On the other hand, you will be nurturing potential top performers from their career infancy and help them grow into top talent with the right personality traits to perform at your company. That has a ROI that can only be beaten by time travelers going back in time and buying stocks in Apple.

This whole train of thought is where agencies might learn something from the world of sports, where it’s a common philosophy in some football clubs (or soccer if that’s the term you prefer to use):

”We don’t sign superstars, we make them”.
– Arsène Wenger, Manager of the Arsenal F.C.

But how do you make sure that your candidates are a cultural fit? And how can you make sure that they can do the work once they get hired?

Contrary to what you might think from our previous arguments about “personality > skills”, it’s important to start with the skills first. At the end of the day you need to know which skills you’re looking for before you can start evaluating personality and cultural fit.

When the hiring process is handled by the department or team that is looking for a new member, the senior members or managers are usually in charge of the process. If there is an obvious need for a specialist that the team doesn’t yet have, creating the requirements should be as easy as simply writing down the tasks that need to be done and translating them into skills. However, if there is just more work coming in for a specific skill set (UX, .NET Developer, etc.), the existing team members should be consulted so that the new hire can complement their skill set.

Once you are settled on the skills it’s time to consider the personality you’re looking for. Are you looking for a person with an extraordinary drive to grind it out 50 hours a week? Or maybe a true team player that makes everyone around them better? There’s no right or wrong answers here – but it’s important to have an idea of which personalities you’re looking for.

The tone of voice varies from agency to agency and even from team to team, and the structure of a job posting can vary quite a bit. But there are still some evergreen tips that could save you and potential candidates some time:

  • When a job has language or certification requirements that make or break the application, start with those
  • Don’t get caught up in the technical requirements and skills needed for the job.
  • Present the personality traits you are looking for on equal footing with skills, education and experience
  • When dealing with entry-level jobs, a portfolio of work could be supplemented with school projects that have a similar scope
  • Don’t put unnecessary year requirements on non-senior jobs
  • With software that has a steeper learning curve, ask for a specific platform that your team uses (Sketch/Adobe XD/InVision) instead of listing experience with prototyping software in general
  • Don’t ask for 8 years of experience in a language that has been around for 3 years
agile, BDD, Management, Specslist, Talent

Talent Career page

A good starting point for your ’’first point of recruitment’’ (not the first point of contact, because that’s probably your landing page) is to create a clear value proposition for the inbound job candidates. Until your agency reaches a certain size, you can’t cater to everyone’s wishes concerning work-life balance. Your hiring decisions should always be based on a cultural fit more than a technical fit.

While technical skills are clearly important, it’s much easier to improve a skill than it is to change
a personality. If we want to go into specifics, we can go back to the user experience analogy. When writing a value proposition on the careers page, you need to think about what kind of agency you really are.

’’We are looking for dedicated people to help bring the most innovative web solutions to life for our clients by day, and help us put up new shelves for all these awards by night…’’

That statement will attract a certain kind of people:

  • Fresh graduates with a lot of ambition looking for validation of their skills
  • Experienced professionals who want an environment for their talents to be utilized
  • People looking for a challenge and don’t even consider crunch time a negative word
  • Career-building professionals who are looking for a place that gets them more awards to their resume
  • People who live for their jobs and look forward to evenings and Saturdays at the office filled with pizza and fixing the kinks in the code

Then on the other side of the spectrum, you could have:

’’ You bring the talent, we bring the perks. At AUE Inc. (Agency Used as an Example), we value strategy and planning above everything else. And thanks to our in-depth research and planning, clients always get the solution they need, instead of the solution they think they want. This also means that our employees never have to worry about scope creep or staying at work past 5 PM. Oh, and did we mention possibilities of

working from home or the 4 day work week?”

A few sentences like this on your career page could go a long way towards attracting people that:

  • Love their jobs, but don’t want to sacrifice time with their family for work
  • Are perfect for the job, but would have had to relocate or travel multiple hours every day
  • Are motivated for the job, but also have other ambitions and are trying to run some sort of side-hustle or project on the side

Sections like ”International Workplace” or ”Fun Squad” shows that we care about an open and fun work environment, where your colleagues also become your friends.

agile, BDD, Management, Specslist, Talent

Personality vs. skills

Before we get any further it’s time to address the tiny elephant in the room:

What’s more important – personality or skills?

To answer that question, you only need to scroll back up a few pages to find our list of characteristics for top talent. Notice how there’s only 1 called skill, while the rest are primarily based on personality?

That’s no coincidence. While skill alone is incredibly important, it’s what makes them capable of doing their job after all, it’s not necessarily the thing that makes them top talent. If they are an amazing coder, but can’t be depended on to meet deadlines or have issues working together with their team, it’s hard to call them top talent.

At the end of the day it’s important to remember that skills can be taught and improved, but personality and culture can’t. And if you want your entire team to perform – not just the individual – it’s important to have the right mix of personalities and culture. If the right culture is there, you’ll see skills improve for everyone and soon you’ll have a team full of top talent that performs day in and day out.

SQAEB TIP

For 99 % of our job postings we use this to highlight our people- first focus:

”We care about people. That’s why the most important qualification is your personality: who you are, what values you have and how you interact with other people. We are looking for people with passion and energy to be part of something bigger than themselves and who are willing to dedicate their time and skills towards building great products and services in collaboration with talented and friendly colleagues.”

agile, BDD, Gaming, Management, Specslist, Talent

Talent RECRUITMENT

Recruitment. Love it or hate it, this is where it all starts if you want to attract top talent for your agency. But there’s so much more to recruitment than job postings and hiring recruiters. It’s in the recruitment phase that the first bit of onboarding starts. While it is 100% the candidate’s responsibility to find out as much as possible about the agency he wants to join, why not show your values and culture even at the earliest stages and make it easier for them?

We are drawn to leaders and organizations that are good at communicating what they believe. Their ability to make us feel like we belong, to make us feel special, safe and not alone is part of what gives them the ability to inspire us

– Simon Sinek, Author of ’’Start with Why’’

It’s no secret that even the most basic one-page websites have an “about us” section. But imagine being a top talent developer or specialist looking for new opportunities. They might go through 50 “about us” pages every day. Does your mission and vision statement stand out of the crowd? Do you communicate having a culture that provides a constant stream of challenging problems to solve? Do you have a hilarious video of your founder switching places with your human-sized-rabbit-office-mascot and shooting confetti at your unsuspecting support staff?

SQAEB TIP

Do you want to show your values to potential clients? Then video is the way to go. It doesn’t have to be a big production – the only thing it has to do, is to show your company values and culture.

Letting your mission, vision and culture shine through in your recruiting process helps you immensely in not only standing out from the crowd, but also in attracting the right people for your company.

agile, BDD, Gaming, Talent

What is top talent?

Before we start our deep dive into the obvious and not-so-obvious ways of attracting and retaining top talent, let’s take a moment to define:

What exactly is top talent?

Top talent is one of those terms that does not have a clear cut definition that people can point to. However, when talking about the agency world, there are certain characteristics that come up time and time again when discussing high performers:

Skill – The go-to metric for determining top talent. Whether it’s due to natural talent or 10,000 hours of practice, if someone is exceptionally skilled, they are on the best possible path to be considered top talent at any agency.

Ambition – The goal to become the top of their field. Ambition drives people to always keep up with the newest trends and developments in their field and continuously improve their skills.

Integrity – When they say something will get done, it gets done at all costs. And if both the managers and team members know they can count on someone when the going gets tough, that person becomes irreplaceable.

Communication – Knowing how to clearly communicate with managers and executives that speak the language of money on one side, while communicating with the technical team members who speak in code and high fidelity mockups on the other is a skill that should be paid in gold.

Teamwork – Everyone can excel at their individual tasks, but sharing a task or working efficiently in a team is a must-have for those that want to become the top performers in any agency

Creativity – Some creatives are a constant source of ideas during a brainstorming session. Some always see a problem from 3 more angles than everyone else. And while creativity manifests in a lot of ways, sometimes it’s the main thing behind a person’s top-talent status.

Leadership – Leadership is not just a skill for managers or team leads. People who join fresh out of college can find themselves at the top of the pyramid in any team within a few months, even with no direct effort. If an individual is approachable, facilitates a good workflow, or solves problems with a leveled head, they will soon become respected by their peers as a leader, even with no title involved.

Devotion – The green ’’you can talk to me’’-light next to the monitor turns red. The headphones go on.
6 hours, 3 cups of coffee, 1 missed lunch, and a single stretching session later, one individual just saved a 10-person project from being one week late. That’s how people become legends. And top talent.

Being considered top talent does not mean that a person has to have all of these qualities fully formed. It doesn’t even mean that top talent and top performers have to achieve all of these qualities eventually. A person who fully masters 3-4 of these qualities should quickly rise to become a prime asset to any agency. And if your agency finds itself hiring a person that displays most or all of these qualities, then you should do everything you can to keep them around until they decide it’s time to retire.

agile

Agile Testing Methods, Techniques, and Tools

Agile Testing Methods

There are certain testing practices that can be followed in every development project (agile or not) to produce quality products. These include writing tests in advance to express proper behaviour, focusing on early defect prevention, detection, and removal, and ensuring that the right test types are run at the right time and as part of the right test level. Agile practitioners aim to introduce these practices early. Testers in Agile projects play a key role in guiding the use of these testing practices throughout the lifecycle. 

Test-Driven Development, Acceptance Test-Driven Development, and Behaviour-Driven Development

Test-driven development, acceptance test-driven development, and behaviour-driven development are three complementary techniques in use among Agile teams to carry out testing across the various test levels. Each technique is an example of a fundamental principle of testing, the benefit of early testing and QA activities, since the tests are defined before the code is written. 

Test-Driven Development

Test-driven development (TDD) is used to develop code guided by automated test cases. The process for test-driven development is:

  • Add a test that captures the programmer’s concept of the desired functioning of a small piece of code
  • Run the test, which should fail since the code doesn’t exist
  • Write the code and run the test in a tight loop until the test passes
  • Refactor the code after the test is passed, re-running the test to ensure it continues to pass against the refactored code
  • Repeat this process for the next small piece of code, running the previous tests as well as the added tests

The tests written are primarily unit level and are code-focused, though tests may also be written at the integration or system levels. Test-driven development gained its popularity through Extreme Programming, but is also used in other Agile methodologies and sometimes in sequential lifecycles. It helps developers focus on clearly defined expected results. The tests are automated and are used in continuous integration.

Acceptance Test-Driven Development

Acceptance test-driven development defines acceptance criteria and tests during the creation of user stories. Acceptance test-driven development is a collaborative approach that allows every stakeholder to understand how the software component has to behave and what the developers, testers, and business representatives need to ensure this behaviour.

Acceptance test-driven development creates reusable tests for regression testing. Specific tools support creation and execution of such tests, often within the continuous integration process. These tools can connect to data and service layers of the application, which allows tests to be executed at the system or acceptance level. Acceptance test-driven development allows quick resolution of defects and validation of feature behaviour. It helps determine if the acceptance criteria are met for the feature.

Behaviour-Driven Development

Behaviour-driven development allows a developer to focus on testing the code based on the expected behaviour of the software. Because the tests are based on the exhibited behaviour from the software, the tests are generally easier for other team members and stakeholders to understand.

Specific behaviour-driven development frameworks can be used to define acceptance criteria based on the given/when/then format:

Given some initial context,

When an event occurs,

Then ensure some outcomes. 

From these requirements, the behaviour-driven development framework generates code that can be used by developers to create test cases. Behaviour-driven development helps the developer collaborate with other stakeholders, including testers, to define accurate unit tests focused on business needs. 

The Test Pyramid

A software system may be tested at different levels. Typical test levels are, from the base of the pyramid to the top, unit, integration, system, and acceptance. The test pyramid emphasises having a large number of tests at the lower levels (bottom of the pyramid) and, as development moves to the upper levels, the number of tests decreases (top of the pyramid). Usually unit and integration level tests are automated and are created using API-based tools. At the system and acceptance levels, the automated tests are created using GUI-based tools. The test pyramid concept is based on the testing principle of early QA and testing (i.e., eliminating defects as early as possible in the lifecycle). 

Testing Quadrants, Test Levels, and Testing Types

Testing quadrants, align the test levels with the appropriate test types in the Agile methodology. The testing quadrants model, and its variants, helps to ensure that all important test types and test levels are included in the development lifecycle. This model also provides a way to differentiate and describe the types of tests to all stakeholders, including developers, testers, and business representatives. 

In the testing quadrants, tests can be business (user) or technology (developer) facing. Some tests support the work done by the Agile team and confirm software behaviour. Other tests can verify the product. Tests can be fully manual, fully automated, a combination of manual and automated, or manual but supported by tools. The four quadrants are as follows: 

  • Quadrant Q1 is unit level, technology facing, and supports the developers. This quadrant contains unit tests. These tests should be automated and included in the continuous integration process.
  • Quadrant Q2 is system level, business facing, and confirms product behaviour. This quadrant contains functional tests, examples, story tests, user experience prototypes, and simulations. These tests check the acceptance criteria and can be manual or automated. They are often created during the user story development and thus improve the quality of the stories. They are useful when creating automated regression test suites.
  • Quadrant Q3 is system or user acceptance level, business facing, and contains tests that critique the product, using realistic scenarios and data. This quadrant contains exploratory testing, scenarios, process flows, usability testing, user acceptance testing, alpha testing, and beta testing. These tests are often manual and are user-oriented.
  • Quadrant Q4 is system or operational acceptance level, technology facing, and contains tests that critique the product. This quadrant contains performance, load, stress, and scalability tests, security tests, maintainability, memory management, compatibility and interoperability, data migration, infrastructure, and recovery testing. These tests are often automated.

During any given iteration, tests from any or all quadrants may be required. The testing quadrants apply to dynamic testing rather than static testing.

The Role of a Tester

Throughout this article, general reference has been made to Agile methods and techniques, and the role of a tester within various Agile lifecycles. This subsection looks specifically at the role of a tester in a project following a Scrum lifecycle. 

Teamwork 

Teamwork is a fundamental principle in Agile development. Agile emphasises the whole-team approach consisting of developers, testers, and business representatives working together. The following are organisational and behavioural best practices in Scrum teams:

  • Cross-functional: Each team member brings a different set of skills to the team. The team works together on test strategy, test planning, test specification, test execution, test evaluation, and test results reporting.
  • Self-organising: The team may consist only of developers, but, as noted before, ideally there would be one or more testers.
  • Co-located: Testers sit together with the developers and the product owner.
  • Collaborative: Testers collaborate with their team members, other teams, the stakeholders, the product owner, and the Scrum Master.
  • Empowered: Technical decisions regarding design and testing are made by the team as a whole (developers, testers, and Scrum Master), in collaboration with the product owner and other teams if needed.
  • Committed: The tester is committed to question and evaluate the product’s behaviour and characteristics with respect to the expectations and needs of the customers and users.
  • Transparent: Development and testing progress is visible on the Agile task board.
  • Credible: The tester must ensure the credibility of the strategy for testing, its implementation, and execution, otherwise the stakeholders will not trust the test results. This is often done by providing information to the stakeholders about the testing process.
  • Open to feedback: Feedback is an important aspect of being successful in any project, especially in Agile projects. Retrospectives allow teams to learn from successes and from failures.
  • Resilient: Testing must be able to respond to change, like all other activities in Agile projects.

These best practices maximise the likelihood of successful testing in Scrum projects.

Sprint Zero 

Sprint zero is the first iteration of the project where many preparation activities take place. The tester collaborates with the team on the following activities during this iteration:

  • Identify the scope of the project (i.e., the product backlog)
  • Create an initial system architecture and high-level prototypes
  • Plan, acquire, and install needed tools (e.g., for test management, defect management, test automation, and continuous integration)
  • Create an initial test strategy for all test levels, addressing (among other topics) test scope, technical risks, test types, and coverage goals
  • Perform an initial quality risk analysis
  • Define test metrics to measure the test process, the progress of testing in the project, and product quality
  • Specify the definition of “done”
  • Create the task board
  • Define when to continue or stop testing before delivering the system to the customer

Sprint zero sets the direction for what testing needs to achieve and how testing needs to achieve it throughout the sprints.

Integration 

In Agile projects, the objective is to deliver customer value on a continuous basis (preferably in every sprint). To enable this, the integration strategy should consider both design and testing. To enable a continuous testing strategy for the delivered functionality and characteristics, it is important to identify all dependencies between underlying functions and features.

Test Planning

Since testing is fully integrated into the Agile team, test planning should start during the release planning session and be updated during each sprint. Test planning for the release and each sprint should address the issues.

Sprint planning results in a set of tasks to put on the task board, where each task should have a length of one or two days of work. In addition, any testing issues should be tracked to keep a steady flow of testing.

Agile Testing Practices

Many practices may be useful for testers in a scrum team, some of which include: 

  • Pairing: Two team members (e.g., a tester and a developer, two testers, or a tester and a product owner) sit together at one workstation to perform a testing or other sprint task.
  • Incremental test design: Test cases and charters are gradually built from user stories and other test bases, starting with simple tests and moving toward more complex ones.
  • Mind mapping: Mind mapping is a useful tool when testing. For example, testers can use mind mapping to identify which test sessions to perform, to show test strategies, and to describe test data.

These practices are in addition to other practices discussed in this article and previous articles on the basics pages.

Assessing Quality Risks and Estimating Test Effort

A typical objective of testing in all projects, Agile or traditional, is to reduce the risk of product quality problems to an acceptable level prior to release. Testers in Agile projects can use the same types of techniques used in traditional projects to identify quality risks (or product risks), assess the associated level of risk, estimate the effort required to reduce those risks sufficiently, and then mitigate those risks through test design, implementation, and execution. However, given the short iterations and rate of change in Agile projects, some adaptations of those techniques are required.

Assessing Quality Risks in Agile Projects

One of the many challenges in testing is the proper selection, allocation, and prioritisation of test conditions. This includes determining the appropriate amount of effort to allocate in order to cover each condition with tests, and sequencing the resulting tests in a way that optimises the effectiveness and efficiency of the testing work to be done. Risk identification, analysis, and risk mitigation strategies can be used by the testers in Agile teams to help determine an acceptable number of test cases to execute, although many interacting constraints and variables may require compromises.

Risk is the possibility of a negative or undesirable outcome or event. The level of risk is found by assessing the likelihood of occurrence of the risk and the impact of the risk. When the primary effect of the potential problem is on product quality, potential problems are referred to as quality risks or product risks. When the primary effect of the potential problem is on project success, potential problems are referred to as project risks or planning risks.

In Agile projects, quality risk analysis takes place at two places.

  • Release planning: business representatives who know the features in the release provide a high-level overview of the risks, and the whole team, including the tester(s), may assist in the risk identification and assessment.
  • Iteration planning: the whole team identifies and assesses the quality risks.

Examples of quality risks for a system include:

  • Incorrect calculations in reports (a functional risk related to accuracy)
  • Slow response to user input (a non-functional risk related to efficiency and response time)
  • Difficulty in understanding screens and fields (a non-functional risk related to usability and understandability)

As mentioned earlier, an iteration starts with iteration planning, which culminates in estimated tasks on a task board. These tasks can be prioritised in part based on the level of quality risk associated with them. Tasks associated with higher risks should start earlier and involve more testing effort. Tasks associated with lower risks should start later and involve less testing effort.

An example of how the quality risk analysis process in an Agile project may be carried out during iteration planning is outlined in the following steps:

  1. Gather the Agile team members together, including the tester(s).
  2. List all the backlog items for the current iteration (e.g., on a task board).
  3. Identify the quality risks associated with each item, considering all relevant quality
    characteristics.
  4. Assess each identified risk, which includes two activities: categorising the risk and determining its level of risk based on the impact and the likelihood of defects.
  5. Determine the extent of testing proportional to the level of risk.
  6. Select the appropriate test technique(s) to mitigate each risk, based on the risk, the level of risk, and the relevant quality characteristic.

The tester then designs, implements, and executes tests to mitigate the risks. This includes the totality of features, behaviours, quality characteristics, and attributes that affect customer, user, and stakeholder satisfaction. 

Throughout the project, the team should remain aware of additional information that may change the set of risks and/or the level of risk associated with known quality risks. Periodic adjustment of the quality risk analysis, which results in adjustments to the tests, should occur. Adjustments include identifying new risks, re-assessing the level of existing risks, and evaluating the effectiveness of risk mitigation activities.

Quality risks can also be mitigated before test execution starts. For example, if problems with the user stories are found during risk identification, the project team can thoroughly review user stories as a mitigating strategy.

Estimating Testing Effort Based on Content and Risk

During release planning, the Agile team estimates the effort required to complete the release. The estimate addresses the testing effort as well. A common estimation technique used in Agile projects is planning poker, a consensus-based technique. The product owner or customer reads a user story to the estimators. Each estimator has a deck of cards with values similar to the Fibonacci sequence (i.e., 0, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, …), or any other progression of choice (e.g., shirt sizes ranging from extra-small to extra-extra-large). The values represent the number of story points, effort days, or other units in which the team estimates. The Fibonacci sequence is recommended because the numbers in the sequence reflect that uncertainty grows proportionally with the size of the story. A high estimate usually means that the story is not well understood or should be broken down into multiple smaller stories. 

The estimators discuss the feature, and ask questions of the product owner as needed. Aspects such as development and testing effort, complexity of the story, and scope of testing play a role in the estimation. Therefore, it is advisable to include the risk level of a backlog item, in addition to the priority specified by the product owner, before the planning poker session is initiated. When the feature has been fully discussed, each estimator privately selects one card to represent his or her estimate. All cards are then revealed at the same time. If all estimators selected the same value, that becomes the estimate. If not, the estimators discuss the differences in estimates after which the poker round is repeated until agreement is reached, either by consensus or by applying rules (e.g., use the median, use the highest score) to limit the number of poker rounds. These discussions ensure a reliable estimate of the effort needed to complete product backlog items requested by the product owner and help improve collective knowledge of what has to be done. 

Techniques in Agile Projects

Many of the test techniques and testing levels that apply to traditional projects can also be applied to Agile projects. However, for Agile projects, there are some specific considerations and variances in test techniques, terminologies, and documentation that should be considered. 

Acceptance Criteria, Adequate Coverage, and Other Information for Testing

Agile projects outline initial requirements as user stories in a prioritised backlog at the start of the project. Initial requirements are short and usually follow a predefined format. Non-functional requirements, such as usability and performance, are also important and can be specified as unique user stories or connected to other functional user stories. Non-functional requirements may follow a predefined format or standard, such as [ISO25000], or an industry specific standard.

The user stories serve as an important test basis. Other possible test bases include:

  • Experience from previous projects
  • Existing functions, features, and quality characteristics of the system
  • Code, architecture, and design
  • User profiles (context, system configurations, and user behaviour)
  • Information on defects from existing and previous projects
  • A categorisation of defects in a defect taxonomy
  • Applicable standards (e.g., [DO-178B] for avionics software)
  • Quality risks

During each iteration, developers create code which implements the functions and features described in the user stories, with the relevant quality characteristics, and this code is verified and validated via acceptance testing. To be testable, acceptance criteria should address the following topics where relevant:

  • Functional behaviour: The externally observable behaviour with user actions as input operating under certain configurations.
  • Quality characteristics: How the system performs the specified behaviour. The characteristics may also be referred to as quality attributes or non-functional requirements. Common quality characteristics are performance, reliability, usability, etc.
  • Scenarios (use cases): A sequence of actions between an external actor (often a user) and the system, in order to accomplish a specific goal or business task.
  • Business rules: Activities that can only be performed in the system under certain conditions defined by outside procedures and constraints (e.g., the procedures used by an insurance company to handle insurance claims).
  • External interfaces: Descriptions of the connections between the system to be developed and the outside world. External interfaces can be divided into different types (user interface, interface to other systems, etc.).
  • Constraints: Any design and implementation constraint that will restrict the options for the developer. Devices with embedded software must often respect physical constraints such as size, weight, and interface connections.
  • Data definitions: The customer may describe the format, data type, allowed values, and default values for a data item in the composition of a complex business data structure (e.g., the ZIP code in a US mail address).

In addition to the user stories and their associated acceptance criteria, other information is relevant for the tester, including:

  • How the system is supposed to work and be used
  • The system interfaces that can be used/accessed to test the system
  • Whether current tool support is sufficient
  • Whether the tester has enough knowledge and skill to perform the necessary tests

Testers will often discover the need for additional information (e.g., code coverage) throughout the iterations and should work collaboratively with the rest of the Agile team members to obtain that information. Relevant information plays a part in determining whether a particular activity can be considered done. This concept of the definition of done is critical in Agile projects and applies in a number of different ways as discussed in the following sub-subsections. 

Test Levels

Each test level has its own definition of done. The following list gives examples that may be relevant for the different test levels.

  • Unit testing
    • 100% decision coverage where possible, with careful reviews of any infeasible paths
    • Static analysis performed on all code
    • No unresolved major defects (ranked based on priority and severity)
    • No known unacceptable technical debt remaining in the design and the code
    • All code, unit tests, and unit test results reviewed
    • All unit tests automated
    • Important characteristics are within agreed limits (e.g., performance)
  • Integration testing
    • All functional requirements tested, including both positive and negative tests, with the number of tests based on size, complexity, and risks
    • All interfaces between units tested
    • All quality risks covered according to the agreed extent of testing
    • No unresolved major defects (prioritised according to risk and importance)
    • All defects found are reported
    • All regression tests automated, where possible, with all automated tests stored in a common repository
  • System testing
    • End-to-end tests of user stories, features, and functions
    • All user personas covered
    • The most important quality characteristics of the system covered (e.g., performance, robustness, reliability)
    • Testing done in a production-like environment(s), including all hardware and software for all supported configurations, to the extent possible
    • All quality risks covered according to the agreed extent of testing
    • All regression tests automated, where possible, with all automated tests stored in a common repository
    • All defects found are reported and possibly fixed
    • No unresolved major defects (prioritised according to risk and importance)

User Story

The definition of done for user stories may be determined by the following criteria: 

  • The user stories selected for the iteration are complete, understood by the team, and have detailed, testable acceptance criteria
  • All the elements of the user story are specified and reviewed, including the user story acceptance tests, have been completed
  • Tasks necessary to implement and test the selected user stories have been identified and estimated by the team

Feature

The definition of done for features, which may span multiple user stories or epics, may include:

  • All constituent user stories, with acceptance criteria, are defined and approved by the customer
  • The design is complete, with no known technical debt
  • The code is complete, with no known technical debt or unfinished refactoring
  • Unit tests have been performed and have achieved the defined level of coverage
  • Integration tests and system tests for the feature have been performed according to the defined coverage criteria
  • No major defects remain to be corrected
  • Feature documentation is complete, which may include release notes, user manuals, and on-line help functions

Iteration 

The definition of done for the iteration may include the following:

  • All features for the iteration are ready and individually tested according to the feature level criteria
  • Any non-critical defects that cannot be fixed within the constraints of the iteration added to the product backlog and prioritised 
  • Integration of all features for the iteration completed and tested 
  • Documentation written, reviewed, and approved 

At this point, the software is potentially releasable because the iteration has been successfully completed, but not all iterations result in a release. 

Release

The definition of done for a release, which may span multiple iterations, may include the following areas:

  • Coverage: All relevant test basis elements for all contents of the release have been covered by testing. The adequacy of the coverage is determined by what is new or changed, its complexity and size, and the associated risks of failure.
  • Quality: The defect intensity (e.g., how many defects are found per day or per transaction), the defect density (e.g., the number of defects found compared to the number of user stories, effort, and/or quality attributes), estimated number of remaining defects are within acceptable limits, the consequences of unresolved and remaining defects (e.g., the severity and priority) are understood and acceptable, the residual level of risk associated with each identified quality risk is understood and acceptable.
  • Time: If the pre-determined delivery date has been reached, the business considerations associated with releasing and not releasing need to be considered.
  • Cost: The estimated lifecycle cost should be used to calculate the return on investment for the delivered system (i.e., the calculated development and maintenance cost should be considerably lower than the expected total sales of the product). The main part of the lifecycle cost often comes from maintenance after the product has been released, due to the number of defects escaping to production. 

Applying Acceptance Test-Driven Development

Acceptance test-driven development is a test-first approach. Test cases are created prior to implementing the user story. The test cases are created by the Agile team, including the developer, the tester, and the business representatives and may be manual or automated. The first step is a specification workshop where the user story is analysed, discussed, and written by developers, testers, and business representatives. Any incompleteness, ambiguities, or errors in the user story are fixed during this process. 

The next step is to create the tests. This can be done by the team together or by the tester individually. In any case, an independent person such as a business representative validates the tests. The tests are examples that describe the specific characteristics of the user story. These examples will help the team implement the user story correctly. Since examples and tests are the same, these terms are often used interchangeably. The work starts with basic examples and open questions. 

Typically, the first tests are the positive tests, confirming the correct behaviour without exception or error conditions, comprising the sequence of activities executed if everything goes as expected. After the positive path tests are done, the team should write negative path tests and cover non-functional attributes as well (e.g., performance, usability). Tests are expressed in a way that every stakeholder is able to understand, containing sentences in natural language involving the necessary preconditions, if any, the inputs, and the related outputs. 

The examples must cover all the characteristics of the user story and should not add to the story. This means that an example should not exist which describes an aspect of the user story not documented in the story itself. In addition, no two examples should describe the same characteristics of the user story. 

Functional and Non-Functional Black Box Test Design

In Agile testing, many tests are created by testers concurrently with the developers’ programming activities. Just as the developers are programming based on the user stories and acceptance criteria, so are the testers creating tests based on user stories and their acceptance criteria. (Some tests, such as exploratory tests and some other experience-based tests, are created later, during test execution) Testers can apply traditional black box test design techniques such as equivalence partitioning, boundary value analysis, decision tables, and state transition testing to create these tests. For example, boundary value analysis could be used to select test values when a customer is limited in the number of items they may select for purchase. 

In many situations, non-functional requirements can be documented as user stories. Black box test design techniques (such as boundary value analysis) can also be used to create tests for non-functional quality characteristics. The user story might contain performance or reliability requirements. For example, a given execution cannot exceed a time limit or a number of operations may fail less than a certain number of times. 

Exploratory Testing and Agile Testing

Exploratory testing is important in Agile projects due to the limited time available for test analysis and the limited details of the user stories. In order to achieve the best results, exploratory testing should be combined with other experience-based techniques as part of a reactive testing strategy, blended with other testing strategies such as analytical risk-based testing, analytical requirements-based testing, model-based testing, and regression-averse testing. Test strategies and test strategy blending is discussed in the basics Level pages. 

In exploratory testing, test design and test execution occur at the same time, guided by a prepared test charter. A test charter provides the test conditions to cover during a time-boxed testing session. During exploratory testing, the results of the most recent tests guide the next test. The same white box and black box techniques can be used to design the tests as when performing pre-designed testing. 

A test charter may include the following information: 

  • Actor: intended user of the system
  • Purpose: the theme of the charter including what particular objective the actor wants to achieve, i.e., the test conditions
  • Setup: what needs to be in place in order to start the test execution
  • Priority: relative importance of this charter, based on the priority of the associated user story or the risk level
  • Reference: specifications (e.g., user story), risks, or other information sources
  • Data: whatever data is needed to carry out the charter
  • Activities: a list of ideas of what the actor may want to do with the system (e.g., “Log on to the system as a super user”) and what would be interesting to test (both positive and negative tests)
  • Oracle notes: how to evaluate the product to determine correct results (e.g., to capture what happens on the screen and compare to what is written in the user’s manual)
  • Variations: alternative actions and evaluations to complement the ideas described under activities

To manage exploratory testing, a method called session-based test management can be used. A session is defined as an uninterrupted period of testing which could last from 60 to 120 minutes. Test sessions include the following:

  • Survey session (to learn how it works)
  • Analysis session (evaluation of the functionality or characteristics)
  • Deep coverage (corner cases, scenarios, interactions)

The quality of the tests depends on the testers’ ability to ask relevant questions about what to test. Examples include the following:

  • What is most important to find out about the system?
  • In what way may the system fail?
  • What happens if…..?
  • What should happen when…..?
  • Are customer needs, requirements, and expectations fulfilled?
  • Is the system possible to install (and remove if necessary) in all supported upgrade paths?

During test execution, the tester uses creativity, intuition, cognition, and skill to find possible problems with the product. The tester also needs to have good knowledge and understanding of the software under test, the business domain, how the software is used, and how to determine when the system fails.

A set of heuristics can be applied when testing. A heuristic can guide the tester in how to perform the testing and to evaluate the results [Hendrickson]. Examples include:

  • Boundaries
  • CRUD (Create, Read, Update, Delete)
  • Configuration variations
  • Interruptions (e.g., log off, shut down, or reboot)

It is important for the tester to document the process as much as possible. Otherwise, it would be difficult to go back and see how a problem in the system was discovered. The following list provides examples of information that may be useful to document:

  • Test coverage: what input data have been used, how much has been covered, and how much remains to be tested
  • Evaluation notes: observations during testing, do the system and feature under test seem to be stable, were any defects found, what is planned as the next step according to the current observations, and any other list of ideas
  • Risk/strategy list: which risks have been covered and which ones remain among the most important ones, will the initial strategy be followed, does it need any changes
  • Issues, questions, and anomalies: any unexpected behaviour, any questions regarding the efficiency of the approach, any concerns about the ideas/test attempts, test environment, test data, misunderstanding of the function, test script or the system under test
  • Actual behaviour: recording of actual behaviour of the system that needs to be saved (e.g., video, screen captures, output data files)

The information logged should be captured and/or summarised into some form of status management tools (e.g., test management tools, task management tools, the task board), in a way that makes it easy for stakeholders to understand the current status for all testing that was performed.

Tools in Agile Projects

Tools described in the basics Level pages are relevant and used by testers on Agile teams. Not all tools are used the same way and some tools have more relevance for Agile projects than they have in traditional projects. For example, although the test management tools, requirements management tools, and incident management tools (defect tracking tools) can be used by Agile teams, some Agile teams opt for an all-inclusive tool (e.g., application lifecycle management or task management) that provides features relevant to Agile development, such as task boards, burn-down charts, and user stories. Configuration management tools are important to testers in Agile teams due to the high number of automated tests at all levels and the need to store and manage the associated automated test artefacts.

In addition to the tools described in the basic Level pages, testers on Agile projects may also utilise the tools described in the following subsections. These tools are used by the whole team to ensure team collaboration and information sharing, which are key to Agile practices.

Task Management and Tracking Tools

In some cases, Agile teams use physical story/task boards (e.g., whiteboard, cork-board) to manage and track user stories, tests, and other tasks throughout each sprint. Other teams will use application lifecycle management and task management software, including electronic task boards. These tools serve the following purposes:

  • Record stories and their relevant development and test tasks, to ensure that nothing gets lost during a sprint
  • Capture team members’ estimates on their tasks and automatically calculate the effort required to implement a story, to support efficient iteration planning sessions
  • Associate development tasks and test tasks with the same story, to provide a complete picture of the team’s effort required to implement the story
  • Aggregate developer and tester updates to the task status as they complete their work, automatically providing a current calculated snapshot of the status of each story, the iteration, and the overall release
  • Provide a visual representation (via metrics, charts, and dashboards) of the current state of each user story, the iteration, and the release, allowing all stakeholders, including people on geographically distributed teams, to quickly check status
  • Integrate with configuration management tools, which can allow automated recording of code check-ins and builds against tasks, and, in some cases, automated status updates for tasks

Communication and Information Sharing Tools

In addition to e-mail, documents, and spoken communication, Agile teams often use three additional types of tools to support communication and information sharing: wikis, instant messaging, and desktop sharing.

Wikis allow teams to build and share an online knowledge base on various aspects of the project, including the following:

  • Product feature diagrams, feature discussions, prototype diagrams, photos of whiteboard discussions, and other information
  • Tools and/or techniques for developing and testing found to be useful by other members of the team
  • Metrics, charts, and dashboards on product status, which is especially useful when the wiki is integrated with other tools such as the build server and task management system, since the tool can update product status automatically
  • Conversations between team members, similar to instant messaging and email, but in a way that is shared with everyone else on the team

Instant messaging, audio teleconferencing, and video chat tools provide the following benefits:

  • Allow real time direct communication between team members, especially distributed teams
  • Involve distributed teams in standup meetings
  • Reduce telephone bills by use of voice-over-IP technology, removing cost constraints that could reduce team member communication in distributed settings

Desktop sharing and capturing tools provide the following benefits:

  • In distributed teams, product demonstrations, code reviews, and even pairing can occur
  • Capturing product demonstrations at the end of each iteration, which can be posted to the team’s wiki

These tools should be used to complement and extend, not replace, face-to-face communication in Agile teams.

Software Build and Distribution Tools

As discussed earlier in this article, daily build and deployment of software is a key practice in Agile teams. This requires the use of continuous integration tools and build distribution tools. The uses, benefits, and risks of these tools was described earlier on the basics of agile page. 

Configuration Management Tools

On Agile teams, configuration management tools may be used not only to store source code and automated tests, but manual tests and other test work products are often stored in the same repository as the product source code. This provides traceability between which versions of the software were tested with which particular versions of the tests, and allows for rapid change without losing historical information. The main types of version control systems include centralised source control systems and distributed version control systems. The team size, structure, location, and requirements to integrate with other tools will determine which version control system is right for a particular Agile project.

Test Design, Implementation, and Execution Tools

Some tools are useful to Agile testers at specific points in the software testing process. While most of these tools are not new or specific to Agile, they provide important capabilities given the rapid change of Agile projects.

  • Test design tools: Use of tools such as mind maps have become more popular to quickly design and define tests for a new feature.
  • Test case management tools: The type of test case management tools used in Agile may be part of the whole team’s application lifecycle management or task management tool.
  • Test data preparation and generation tools: Tools that generate data to populate an application’s database are very beneficial when a lot of data and combinations of data are necessary to test the application. These tools can also help re-define the database structure as the product undergoes changes during an Agile project and refactor the scripts to generate the data. This allows quick updating of test data as changes occur. Some test data preparation tools use production data sources as a raw material and use scripts to remove or anonymise sensitive data. Other test data preparation tools can help with validating large data inputs or outputs.
  • Test data load tools: After data has been generated for testing, it needs to be loaded into the application. Manual data entry is often time consuming and error prone, but data load tools are available to make the process reliable and efficient. In fact, many of the data generator tools include an integrated data load component. In other cases, bulk-loading using the database management systems is also possible.
  • Automated test execution tools: There are test execution tools which are more aligned to Agile testing. Specific tools are available via both commercial and open source avenues to support test first approaches, such as behaviour-driven development, test-driven development, and acceptance test-driven development. These tools allow testers and business staff to express the expected system behaviour in tables or natural language using keywords.
  • Exploratory test tools: Tools that capture and log activities performed on an application during an exploratory test session are beneficial to the tester and developer, as they record the actions taken. This is useful when a defect is found, as the actions taken before the failure occurred have been captured and can be used to report the defect to the developers. Logging steps performed in an exploratory test session may prove to be beneficial if the test is ultimately included in the automated regression test suite.

Cloud Computing and Virtualisation Tools

Virtualisation allows a single physical resource (server) to operate as many separate, smaller resources. When virtual machines or cloud instances are used, teams have a greater number of servers available to them for development and testing. This can help to avoid delays associated with waiting for physical servers. Provisioning a new server or restoring a server is more efficient with snapshot capabilities built into most virtualisation tools. Some test management tools now utilise virtualisation technologies to snapshot servers at the point when a fault is detected, allowing testers to share the snapshot with the developers investigating the fault.

agile

Basics of Agile Testing Practices, Principles, and Processes

The Differences between Testing in Traditional and Agile Approaches

As described in the basics pages, test activities are related to development activities, and thus testing varies in different lifecycles. Testers must understand the differences between testing in traditional lifecycle models (e.g., sequential such as the V-model or iterative such as RUP) and Agile lifecycles in order to work effectively and efficiently. The Agile models differ in terms of the way testing and development activities are integrated, the project work products, the names, entry and exit criteria used for various levels of testing, the use of tools, and how independent testing can be effectively utilised. 

Testers should remember that organisations vary considerably in their implementation of lifecycles. Deviation from the ideals of Agile lifecycles may represent intelligent customisation and adaptation of the practices. The ability to adapt to the context of a given project, including the software development practices actually followed, is a key success factor for testers. 

Testing and Development Activities

One of the main differences between traditional lifecycles and Agile lifecycles is the idea of very short iterations, each iteration resulting in working software that delivers features of value to business stakeholders. At the beginning of the project, there is a release planning period. This is followed by a sequence of iterations. At the beginning of each iteration, there is an iteration planning period. Once iteration scope is established, the selected user stories are developed, integrated with the system, and tested. These iterations are highly dynamic, with development, integration, and testing activities taking place throughout each iteration, and with considerable parallelism and overlap. Testing activities occur throughout the iteration, not as a final activity. 

Testers, developers, and business stakeholders all have a role in testing, as with traditional lifecycles. Developers perform unit tests as they develop features from the user stories. Testers then test those features. Business stakeholders also test the stories during implementation. Business stakeholders might use written test cases, but they also might simply experiment with and use the feature in order to provide fast feedback to the development team. 

In some cases, hardening or stabilisation iterations occur periodically to resolve any lingering defects and other forms of technical debt. However, the best practice is that no feature is considered done until it has been integrated and tested with the system. Another good practice is to address defects remaining from the previous iteration at the beginning of the next iteration, as part of the backlog for that iteration (referred to as “fix bugs first”). However, some complain that this practice results in a situation where the total work to be done in the iteration is unknown and it will be more difficult to estimate when the remaining features can be done. At the end of the sequence of iterations, there can be a set of release activities to get the software ready for delivery, though in some cases delivery occurs at the end of each iteration. 

When risk-based testing is used as one of the test strategies, a high-level risk analysis occurs during release planning, with testers often driving that analysis. However, the specific quality risks associated with each iteration are identified and assessed in iteration planning. This risk analysis can influence the sequence of development as well as the priority and depth of testing for the features. It also influences the estimation of the test effort required for each feature. 

In some Agile practices (e.g., Extreme Programming), pairing is used. Pairing can involve testers working together in twos to test a feature. Pairing can also involve a tester working collaboratively with a developer to develop and test a feature. Pairing can be difficult when the test team is distributed, but processes and tools can help enable distributed pairing. 

Testers may also serve as testing and quality coaches within the team, sharing testing knowledge and supporting quality assurance work within the team. This promotes a sense of collective ownership of quality of the product. 

Test automation at all levels of testing occurs in many Agile teams, and this can mean that testers spend time creating, executing, monitoring, and maintaining automated tests and results. Because of the heavy use of test automation, a higher percentage of the manual testing on Agile projects tends to be done using experience-based and defect-based techniques such as software attacks, exploratory testing, and error guessing. While developers will focus on creating unit tests, testers should focus on creating automated integration, system, and system integration tests. This leads to a tendency for Agile teams to favour testers with a strong technical and test automation background. 

One core Agile principle is that change may occur throughout the project. Therefore, lightweight work product documentation is favoured in Agile projects. Changes to existing features have testing implications, especially regression testing implications. The use of automated testing is one way of managing the amount of test effort associated with change. However, it’s important that the rate of change not exceed the project team’s ability to deal with the risks associated with those changes. 

Project Work Products

Project work products of immediate interest to Agile testers typically fall into three categories: 

  1. Business-oriented work products that describe what is needed (e.g., requirements specifications) and how to use it (e.g., user documentation)
  2. Development work products that describe how the system is built (e.g., database entity-relationship diagrams), that actually implement the system (e.g., code), or that evaluate individual pieces of code (e.g., automated unit tests)
  3. Test work products that describe how the system is tested (e.g., test strategies and plans), that actually test the system (e.g., manual and automated tests), or that present test results (e.g., test dashboards)

In a typical Agile project, it is a common practice to avoid producing vast amounts of documentation. Instead, focus is more on having working software, together with automated tests that demonstrate conformance to requirements. This encouragement to reduce documentation applies only to documentation that does not deliver value to the customer. In a successful Agile project, a balance is struck between increasing efficiency by reducing documentation and providing sufficient documentation to support business, testing, development, and maintenance activities. The team must make a decision during release planning about which work products are required and what level of work product documentation is needed. 

Typical business-oriented work products on Agile projects include user stories and acceptance criteria. User stories are the Agile form of requirements specifications, and should explain how the system should behave with respect to a single, coherent feature or function. A user story should define a feature small enough to be completed in a single iteration. Larger collections of related features, or a collection of sub-features that make up a single complex feature, may be referred to as “epics”. Epics may include user stories for different development teams. For example, one user story can describe what is required at the API-level (middleware) while another story describes what is needed at the UI-level (application). These collections may be developed over a series of sprints. Each epic and its user stories should have associated acceptance criteria. 

Typical developer work products on Agile projects include code. Agile developers also often create automated unit tests. These tests might be created after the development of code. In some cases, though, developers create tests incrementally, before each portion of the code is written, in order to provide a way of verifying, once that portion of code is written, whether it works as expected. While this approach is referred to as test first or test-driven development, in reality the tests are more a form of executable low-level design specifications rather than tests. 

Typical tester work products on Agile projects include automated tests, as well as documents such as test plans, quality risk catalogs, manual tests, defect reports, and test results logs. The documents are captured in as lightweight a fashion as possible, which is often also true of these documents in traditional lifecycles. Testers will also produce test metrics from defect reports and test results logs, and again there is an emphasis on a lightweight approach. 

In some Agile implementations, especially regulated, safety critical, distributed, or highly complex projects and products, further formalisation of these work products is required. For example, some teams transform user stories and acceptance criteria into more formal requirements specifications. Vertical and horizontal traceability reports may be prepared to satisfy auditors, regulations, and other requirements. 

Test Levels

Test levels are test activities that are logically related, often by the maturity or completeness of the item under test. 

In sequential lifecycle models, the test levels are often defined such that the exit criteria of one level are part of the entry criteria for the next level. In some iterative models, this rule does not apply. Test levels overlap. Requirement specification, design specification, and development activities may overlap with test levels. 

In some Agile lifecycles, overlap occurs because changes to requirements, design, and code can happen at any point in an iteration. While Scrum, in theory, does not allow changes to the user stories after iteration planning, in practice such changes sometimes occur. During an iteration, any given user story will typically progress sequentially through the following test activities: 

  • Unit testing, typically done by the developer
  • Feature acceptance testing, which is sometimes broken into two activities: 
  • Feature verification testing, which is often automated, may be done by developers or testers, and involves testing against the user story’s acceptance criteria
  • Feature validation testing, which is usually manual and can involve developers, testers, and business stakeholders working collaboratively to determine whether the feature is fit for use, to improve visibility of the progress made, and to receive real feedback from the business stakeholders

In addition, there is often a parallel process of regression testing occurring throughout the iteration. This involves re-running the automated unit tests and feature verification tests from the current iteration and previous iterations, usually via a continuous integration framework.

In some Agile projects, there may be a system test level, which starts once the first user story is ready for such testing. This can involve executing functional tests, as well as non-functional tests for performance, reliability, usability, and other relevant test types.

Agile teams can employ various forms of acceptance testing. Internal alpha tests and external beta tests may occur, either at the close of each iteration, after the completion of each iteration, or after a series of iterations. User acceptance tests, operational acceptance tests, regulatory acceptance tests, and contract acceptance tests also may occur, either at the close of each iteration, after the completion of each iteration, or after a series of iterations.

Testing and Configuration Management

Agile projects often involve heavy use of automated tools to develop, test, and manage software development. Developers use tools for static analysis, unit testing, and code coverage. Developers continuously check the code and unit tests into a configuration management system, using automated build and test frameworks. These frameworks allow the continuous integration of new software with the system, with the static analysis and unit tests run repeatedly as new software is checked in. 

These automated tests can also include functional tests at the integration and system levels. Such functional automated tests may be created using functional testing harnesses, open-source user interface functional test tools, or commercial tools, and can be integrated with the automated tests run as part of the continuous integration framework. In some cases, due to the duration of the functional tests, the functional tests are separated from the unit tests and run less frequently. For example, unit tests may be run each time new software is checked in, while the longer functional tests are run only every few days. 

One goal of the automated tests is to confirm that the build is functioning and installable. If any automated test fails, the team should fix the underlying defect in time for the next code check-in. This requires an investment in real-time test reporting to provide good visibility into test results. This approach helps reduce expensive and inefficient cycles of “build-install-fail-rebuild-reinstall” that can occur in many traditional projects, since changes that break the build or cause software to fail to install are detected quickly. 

Automated testing and build tools help to manage the regression risk associated with the frequent change that often occurs in Agile projects. However, over-reliance on automated unit testing alone to manage these risks can be a problem, as unit testing often has limited defect detection effectiveness. Automated tests at the integration and system levels are also required. 

Organisational Options for Independent Testing

As discussed in the basics section, independent testers are often more effective at finding defects. In some Agile teams, developers create many of the tests in the form of automated tests. One or more testers may be embedded within the team, performing many of the testing tasks. However, given those testers’ position within the team, there is a risk of loss of independence and objective evaluation.

Other Agile teams retain fully independent, separate test teams, and assign testers on-demand during the final days of each sprint. This can preserve independence, and these testers can provide an objective, unbiased evaluation of the software. However, time pressures, lack of understanding of the new features in the product, and relationship issues with business stakeholders and developers often lead to problems with this approach. 

A third option is to have an independent, separate test team where testers are assigned to Agile teams on a long-term basis, at the beginning of the project, allowing them to maintain their independence while gaining a good understanding of the product and strong relationships with other team members. In addition, the independent test team can have specialised testers outside of the Agile teams to work on long-term and/or iteration-independent activities, such as developing automated test tools, carrying out non-functional testing, creating and supporting test environments and data, and carrying out test levels that might not fit well within a sprint (e.g., system integration testing). 

Status of Testing in Agile Projects

Change takes place rapidly in Agile projects. This change means that test status, test progress, and product quality constantly evolve, and testers must devise ways to get that information to the team so that they can make decisions to stay on track for successful completion of each iteration. In addition, change can affect existing features from previous iterations. Therefore, manual and automated tests must be updated to deal effectively with regression risk. 

Communicating Test Status, Progress, and Product Quality

Agile teams progress by having working software at the end of each iteration. To determine when the team will have working software, they need to monitor the progress of all work items in the iteration and release. Testers in Agile teams utilise various methods to record test progress and status, including test automation results, progression of test tasks and stories on the Agile task board, and burn-down charts showing the team’s progress. These can then be communicated to the rest of the team using media such as wiki dashboards and dashboard-style emails, as well as orally during stand-up meetings. Agile teams may use tools that automatically generate status reports based on test results and task progress, which in turn update wiki-style dashboards and emails. This method of communication also gathers metrics from the testing process, which can be used in process improvement. Communicating test status in such an automated manner also frees testers’ time to focus on designing and executing more test cases.

Teams may use burn-down charts to track progress across the entire release and within each iteration. A burn-down chart represents the amount of work left to be done against time allocated to the release or iteration.

To provide an instant, detailed visual representation of the whole team’s current status, including the status of testing, teams may use Agile task boards. The story cards, development tasks, test tasks, and other tasks created during iteration planning are captured on the task board, often using colour-coordinated cards to determine the task type. During the iteration, progress is managed via the movement of these tasks across the task board into columns such as to do, work in progress, verify, and done. Agile teams may use tools to maintain their story cards and Agile task boards, which can automate dashboards and status updates. 

Testing tasks on the task board relate to the acceptance criteria defined for the user stories. As test automation scripts, manual tests, and exploratory tests for a test task achieve a passing status, the task moves into the done column of the task board. The whole team reviews the status of the task board regularly, often during the daily stand-up meetings, to ensure tasks are moving across the board at an acceptable rate. If any tasks (including testing tasks) are not moving or are moving too slowly, the team reviews and addresses any issues that may be blocking the progress of those tasks. 

The daily stand-up meeting includes all members of the Agile team including testers. At this meeting, they communicate their current status. The agenda for each member is: 

  • What have you completed since the last meeting?
  • What do you plan to complete by the next meeting?
  • What is getting in your way?

Any issues that may block test progress are communicated during the daily stand-up meetings, so the whole team is aware of the issues and can resolve them accordingly.

To improve the overall product quality, many Agile teams perform customer satisfaction surveys to receive feedback on whether the product meets customer expectations. Teams may use other metrics similar to those captured in traditional development methodologies, such as test pass/fail rates, defect discovery rates, confirmation and regression test results, defect density, defects found and fixed, requirements coverage, risk coverage, code coverage, and code churn to improve the product quality.

As with any lifecycle, the metrics captured and reported should be relevant and aid decision-making. Metrics should not be used to reward, punish, or isolate any team members. 

Managing Regression Risk with Evolving Manual and Automated Test Cases

In an Agile project, as each iteration completes, the product grows. Therefore, the scope of testing also increases. Along with testing the code changes made in the current iteration, testers also need to verify no regression has been introduced on features that were developed and tested in previous iterations. The risk of introducing regression in Agile development is high due to extensive code churn (lines of code added, modified, or deleted from one version to another). Since responding to change is a key Agile principle, changes can also be made to previously delivered features to meet business needs. In order to maintain velocity without incurring a large amount of technical debt, it is critical that teams invest in test automation at all test levels as early as possible. It is also critical that all test assets such as automated tests, manual test cases, test data, and other testing artefacts are kept up-to-date with each iteration. It is highly recommended that all test assets be maintained in a configuration management tool in order to enable version control, to ensure ease of access by all team members, and to support making changes as required due to changing functionality while still preserving the historic information of the test assets. 

Because complete repetition of all tests is seldom possible, especially in tight-timeline Agile projects, testers need to allocate time in each iteration to review manual and automated test cases from previous and current iterations to select test cases that may be candidates for the regression test suite, and to retire test cases that are no longer relevant. Tests written in earlier iterations to verify specific features may have little value in later iterations due to feature changes or new features which alter the way those earlier features behave. 

While reviewing test cases, testers should consider suitability for automation. The team needs to automate as many tests as possible from previous and current iterations. This allows automated regression tests to reduce regression risk with less effort than manual regression testing would require. This reduced regression test effort frees the testers to more thoroughly test new features and functions in the current iteration. 

It is critical that testers have the ability to quickly identify and update test cases from previous iterations and/or releases that are affected by the changes made in the current iteration. Defining how the team designs, writes, and stores test cases should occur during release planning. Good practices for test design and implementation need to be adopted early and applied consistently. The shorter timeframes for testing and the constant change in each iteration will increase the impact of poor test design and implementation practices. 

Use of test automation, at all test levels, allows Agile teams to provide rapid feedback on product quality. Well-written automated tests provide a living document of system functionality. By checking the automated tests and their corresponding test results into the configuration management system, aligned with the versioning of the product builds, Agile teams can review the functionality tested and the test results for any given build at any given point in time.

Automated unit tests are run before source code is checked into the mainline of the configuration management system to ensure the code changes do not break the software build. To reduce build breaks, which can slow down the progress of the whole team, code should not be checked in unless all automated unit tests pass. Automated unit test results provide immediate feedback on code and build quality, but not on product quality. 

Automated acceptance tests are run regularly as part of the continuous integration full system build. These tests are run against a complete system build at least daily, but are generally not run with each code check-in as they take longer to run than automated unit tests and could slow down code check-ins. The test results from automated acceptance tests provide feedback on product quality with respect to regression since the last build, but they do not provide status of overall product quality. 

Automated tests can be run continuously against the system. An initial subset of automated tests to cover critical system functionality and integration points should be created immediately after a new build is deployed into the test environment. These tests are commonly known as build verification tests. Results from the build verification tests will provide instant feedback on the software after deployment, so teams don’t waste time testing an unstable build. 

Automated tests contained in the regression test set are generally run as part of the daily main build in the continuous integration environment, and again when a new build is deployed into the test environment. As soon as an automated regression test fails, the team stops and investigates the reasons for the failing test. The test may have failed due to legitimate functional changes in the current iteration, in which case the test and/or user story may need to be updated to reflect the new acceptance criteria. Alternatively, the test may need to be retired if another test has been built to cover the changes. However, if the test failed due to a defect, it is a good practice for the team to fix the defect prior to progressing with new features. 

In addition to test automation, the following testing tasks may also be automated: 

  • Test data generation
  • Loading testing data into systems
  • Deployment of builds into the test environments
  • Restoration of a test environment (e.g., the database or website data files) to a baseline
  • Comparison of data outputs

Automation of these tasks reduces the overhead and allows the team to spend time developing and testing new features.

Role and Skills of a Tester in an Agile Team

In an Agile team, testers must closely collaborate with all other team members and with business stakeholders. This has a number of implications in terms of the skills a tester must have and the activities they perform within an Agile team.

Agile Tester Skills

Agile testers should have all the skills mentioned in the basics section. In addition to these skills, a tester in an Agile team should be competent in test automation, test-driven development, acceptance test-driven development, white-box, black-box, and experience-based testing.

As Agile methodologies depend heavily on collaboration, communication, and interaction between the team members as well as stakeholders outside the team, testers in an Agile team should have good interpersonal skills. Testers in Agile teams should:

  • Be positive and solution-oriented with team members and stakeholders
  • Display critical, quality-oriented, skeptical thinking about the product
  • Actively acquire information from stakeholders (rather than relying entirely on written specifications)
  • Accurately evaluate and report test results, test progress, and product quality
  • Work effectively to define testable user stories, especially acceptance criteria, with customer representatives and stakeholders
  • Collaborate within the team, working in pairs with programmers and other team members
  • Respond to change quickly, including changing, adding, or improving test cases
  • Plan and organise their own work

Continuous skills growth, including interpersonal skills growth, is essential for all testers, including those on Agile teams. 

The Role of a Tester in an Agile Team

The role of a tester in an Agile team includes activities that generate and provide feedback not only on test status, test progress, and product quality, but also on process quality. In addition to the activities described elsewhere in this article, these activities include: 

  • Understanding, implementing, and updating the test strategy
  • Measuring and reporting test coverage across all applicable coverage dimensions
  • Ensuring proper use of testing tools
  • Configuring, using, and managing test environments and test data
  • Reporting defects and working with the team to resolve them
  • Coaching other of the team members in relevant aspects of testing 
  • Ensuring the appropriate testing tasks are scheduled during release and iteration planning
  • Actively collaborating with developers and business stakeholders to clarify requirements, especially in terms of testability, consistency, and completeness
  • Participating proactively in the team retrospectives, suggesting and implementing improvements

Within an Agile team, each team member is responsible for product quality and plays a role in performing test-related tasks.

Agile organisations may encounter some test-related organisational risks:

  • Testers work so closely to developers that they lose the appropriate tester mindset
  • Testers become tolerant of or silent about inefficient, ineffective, or low-quality practices within the team
  • Testers cannot keep pace with the incoming changes in time-constrained iterations

To mitigate these risks, organisations may consider variations for preserving independence discussed in this article.

agile

Agile Software Development

Basics of Agile Software Development

A tester on an Agile project will work differently than one working on a traditional project. Testers must understand the values and principles that underpin Agile projects, and how testers are an integral part of a whole-team approach together with developers and business representatives. The members in an Agile project communicate with each other early and frequently, which helps with removing defects early and developing a quality product. 

Agile Software Development and the Agile Manifesto 

In 2001, a group of individuals, representing the most widely used lightweight software development methodologies, agreed on a common set of values and principles which became known as the Manifesto for Agile Software Development or the Agile Manifesto [Agile-manifesto]. The Agile Manifesto contains four statements of values:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

The Agile Manifesto argues that although the concepts on the right have value, those on the left have greater value.

Individuals and Interactions

Agile development is very people-centred. Teams of people build software, and it is through continuous communication and interaction, rather than a reliance on tools or processes, that teams can work most effectively.

Working Software

From a customer perspective, working software is much more useful and valuable than overly detailed documentation and it provides an opportunity to give the development team rapid feedback. In addition, because working software, albeit with reduced functionality, is available much earlier in the development lifecycle, Agile development can confer significant time-to-market advantage. Agile development is, therefore, especially useful in rapidly changing business environments where the problems and/or solutions are unclear or where the business wishes to innovate in new problem domains.

Customer Collaboration

Customers often find great difficulty in specifying the system that they require. Collaborating directly with the customer improves the likelihood of understanding exactly what the customer requires. While having contracts with customers may be important, working in regular and close collaboration with them is likely to bring more success to the project.

Responding to Change 

Change is inevitable in software projects. The environment in which the business operates, legislation, competitor activity, technology advances, and other factors can have major influences on the project and its objectives. These factors must be accommodated by the development process. As such, having flexibility in work practices to embrace change is more important than simply adhering rigidly to a plan.

Principles 

The core Agile Manifesto values are captured in twelve principles

  • Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
  • Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
  • Deliver working software frequently, at intervals of between a few weeks to a few months, with a preference to the shorter timescale.
  • Business people and developers must work together daily throughout the project.
  • Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
  • The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
  • Working software is the primary measure of progress.
  • Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
  • Continuous attention to technical excellence and good design enhances agility.
  • Simplicity—the art of maximising the amount of work not done—is essential.
  • The best architectures, requirements, and designs emerge from self-organising teams.
  • At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly.

The different Agile methodologies provide prescriptive practices to put these values and principles into action.

Whole-Team Approach

The whole-team approach means involving everyone with the knowledge and skills necessary to ensure project success. The team includes representatives from the customer and other business stakeholders who determine product features. The team should be relatively small; successful teams have been observed with as few as three people and as many as nine. Ideally, the whole team shares the same workspace, as co-location strongly facilitates communication and interaction. The whole-team approach is supported through the daily stand-up meetings involving all members of the team, where work progress is communicated and any impediments to progress are highlighted. The whole-team approach promotes more effective and efficient team dynamics.

The use of a whole-team approach to product development is one of the main benefits of Agile development. Its benefits include:

  • Enhancing communication and collaboration within the team
  • Enabling the various skill sets within the team to be leveraged to the benefit of the project
  • Making quality everyone’s responsibility

The whole team is responsible for quality in Agile projects. The essence of the whole-team approach lies in the testers, developers, and the business representatives working together in every step of the development process. Testers will work closely with both developers and business representatives to ensure that the desired quality levels are achieved. This includes supporting and collaborating with business representatives to help them create suitable acceptance tests, working with developers to agree on the testing strategy, and deciding on test automation approaches. Testers can thus transfer and extend testing knowledge to other team members and influence the development of the product.

The whole team is involved in any consultations or meetings in which product features are presented, analysed, or estimated. The concept of involving testers, developers, and business representatives in all feature discussions is known as the power of three.

Early and Frequent Feedback

Agile projects have short iterations enabling the project team to receive early and continuous feedback on product quality throughout the development lifecycle. One way to provide rapid feedback is by continuous integration.

When sequential development approaches are used, the customer often does not see the product until the project is nearly completed. At that point, it is often too late for the development team to effectively address any issues the customer may have. By getting frequent customer feedback as the project progresses, Agile teams can incorporate most new changes into the product development process. Early and frequent feedback helps the team focus on the features with the highest business value, or associated risk, and these are delivered to the customer first. It also helps manage the team better since the capability of the team is transparent to everyone. For example, how much work can we do in a sprint or iteration? What could help us go faster? What is preventing us from doing so? 

The benefits of early and frequent feedback include:

  • Avoiding requirements misunderstandings, which may not have been detected until later in the development cycle when they are more expensive to fix.
  • Clarifying customer feature requests, making them available for customer use early. This way, the product better reflects what the customer wants. 
  • Discovering (via continuous integration), isolating, and resolving quality problems early.
  • Providing information to the Agile team regarding its productivity and ability to deliver.
  • Promoting consistent project momentum.

Aspects of Agile Approaches

There are a number of Agile approaches in use by organisations. Common practices across most Agile organisations include collaborative user story creation, retrospectives, continuous integration, and planning for each iteration as well as for overall release. This subsection describes some of the Agile approaches.

Agile Software Development Approaches

There are several Agile approaches, each of which implements the values and principles of the Agile Manifesto in different ways. In this article , three representatives of Agile approaches are considered: Extreme Programming (XP), Scrum, and Kanban.

Extreme Programming

Extreme Programming (XP), is an Agile approach to software development described by certain values, principles, and development practices.

XP embraces five values to guide development: communication, simplicity, feedback, courage, and respect.

XP describes a set of principles as additional guidelines: humanity, economics, mutual benefit, self-similarity, improvement, diversity, reflection, flow, opportunity, redundancy, failure, quality, baby steps, and accepted responsibility.

XP describes thirteen primary practices: sit together, whole team, informative workspace, energised work, pair programming, stories, weekly cycle, quarterly cycle, slack, ten-minute build, continuous integration, test first programming, and incremental design. 

Many of the Agile software development approaches in use today are influenced by XP and its values and principles. For example, Agile teams following Scrum often incorporate XP practices.

Scrum 

Scrum is an Agile management framework which contains the following constituent instruments and practices: 

  • Sprint: Scrum divides a project into iterations (called sprints) of fixed length (usually two to four weeks).
  • Product Increment: Each sprint results in a potentially releasable/shippable product (called an increment).
  • Product Backlog: The product owner manages a prioritised list of planned product items (called the product backlog). The product backlog evolves from sprint to sprint (called backlog refinement).
  • Sprint Backlog: At the start of each sprint, the Scrum team selects a set of highest priority items (called the sprint backlog) from the product backlog. Since the Scrum team, not the product owner, selects the items to be realised within the sprint, the selection is referred to as being on the pull principle rather than the push principle.
  • Definition of Done: To make sure that there is a potentially releasable product at each sprint’s end, the Scrum team discusses and defines appropriate criteria for sprint completion. The discussion deepens the team’s understanding of the backlog items and the product requirements.
  • Time-boxing: Only those tasks, requirements, or features that the team expects to finish within the sprint are part of the sprint backlog. If the development team cannot finish a task within a sprint, the associated product features are removed from the sprint and the task is moved back into the product backlog. Time-boxing applies not only to tasks, but in other situations (e.g., enforcing meeting start and end times).
  • Transparency: The development team reports and updates sprint status on a daily basis at a meeting called the daily scrum. This makes the content and progress of the current sprint, including test results, visible to the team, management, and all interested parties. For example, the development team can show sprint status on a whiteboard.

Scrum defines three roles:

  • Scrum Master: ensures that Scrum practices and rules are implemented and followed, and resolves any violations, resource issues, or other impediments that could prevent the team from following the practices and rules. This person is not the team lead, but a coach.
  • Product Owner: represents the customer, and generates, maintains, and priorities the product backlog. This person is not the team lead.
  • Development Team: develops and test the product. The team is self-organised: There is no team lead, so the team makes the decisions. The team is also cross-functional.

Scrum (as opposed to XP) does not dictate specific software development techniques (e.g., test first programming). In addition, Scrum does not provide guidance on how testing has to be done in a Scrum project.

Kanban

Kanban is a management approach that is sometimes used in Agile projects. The general objective is to visualise and optimise the flow of work within a value-added chain. Kanban utilises three instruments:

  • Kanban Board: The value chain to be managed is visualised by a Kanban board. Each column shows a station, which is a set of related activities, e.g., development or testing. The items to be produced or tasks to be processed are symbolised by tickets moving from left to right across the board through the stations.
  • Work-in-Progress Limit: The amount of parallel active tasks is strictly limited. This is controlled by the maximum number of tickets allowed for a station and/or globally for the board. Whenever a station has free capacity, the worker pulls a ticket from the predecessor station.
  • Lead Time: Kanban is used to optimise the continuous flow of tasks by minimising the (average) lead time for the complete value stream.

Kanban features some similarities to Scrum. In both frameworks, visualising the active tasks (e.g., on a public whiteboard) provides transparency of content and progress of tasks. Tasks not yet scheduled are waiting in a backlog and moved onto the Kanban board as soon as there is new space (production capacity) available.

Iterations or sprints are optional in Kanban. The Kanban process allows releasing its deliverables item by item, rather than as part of a release. Time-boxing as a synchronising mechanism, therefore, is optional, unlike in Scrum, which synchronies all tasks within a sprint.

Collaborative User Story Creation

Poor specifications are often a major reason for project failure. Specification problems can result from the users’ lack of insight into their true needs, absence of a global vision for the system, redundant or contradictory features, and other miscommunications. In Agile development, user stories are written to capture requirements from the perspectives of developers, testers, and business representatives. In sequential development, this shared vision of a feature is accomplished through formal reviews after requirements are written; in Agile development, this shared vision is accomplished through frequent informal reviews while the requirements are being written

The user stories must address both functional and non-functional characteristics. Each story includes acceptance criteria for these characteristics. These criteria should be defined in collaboration between business representatives, developers, and testers. They provide developers and testers with an extended vision of the feature that business representatives will validate. An Agile team considers a task finished when a set of acceptance criteria have been satisfied.

Typically, the tester’s unique perspective will improve the user story by identifying missing details or non-functional requirements. A tester can contribute by asking business representatives open-ended questions about the user story, proposing ways to test the user story, and confirming the acceptance criteria.

The collaborative authorship of the user story can use techniques such as brainstorming and mind mapping. The tester may use the INVEST technique [INVEST]:

  • Independent
  • Negotiable
  • Valuable
  • Estimable
  • Small
  • Testable

According to the 3C concept, a user story is the conjunction of three elements:

  • Card: The card is the physical media describing a user story. It identifies the requirement, its criticality, expected development and test duration, and the acceptance criteria for that story.
    The description has to be accurate, as it will be used in the product backlog.
  • Conversation: The conversation explains how the software will be used. The conversation can be documented or verbal. Testers, having a different point of view than developers and business representatives, bring valuable input to the exchange of thoughts, opinions, and experiences. Conversation begins during the release-planning phase and continues when the story is scheduled.
  • Confirmation: The acceptance criteria, discussed in the conversation, are used to confirm that the story is done. These acceptance criteria may span multiple user stories. Both positive and negative tests should be used to cover the criteria. During confirmation, various participants play the role of a tester. These can include developers as well as specialists focused on performance, security, interoperability, and other quality characteristics. To confirm a story as done, the defined acceptance criteria should be tested and shown to be satisfied.

Agile teams vary in terms of how they document user stories. Regardless of the approach taken to document user stories, documentation should be concise, sufficient, and necessary.

Retrospectives

In Agile development, a retrospective is a meeting held at the end of each iteration to discuss what was successful, what could be improved, and how to incorporate the improvements and retain the successes in future iterations. Retrospectives cover topics such as the process, people, organisations, relationships, and tools. Regularly conducted retrospective meetings, when appropriate follow up activities occur, are critical to self-organisation and continual improvement of development and testing.

Retrospectives can result in test-related improvement decisions focused on test effectiveness, test productivity, test case quality, and team satisfaction. They may also address the testability of the applications, user stories, features, or system interfaces. Root cause analysis of defects can drive testing and development improvements. In general, teams should implement only a few improvements per iteration. This allows for continuous improvement at a sustained pace.

The timing and organisation of the retrospective depends on the particular Agile method followed. Business representatives and the team attend each retrospective as participants while the facilitator organises and runs the meeting. In some cases, the teams may invite other participants to the meeting.

Testers should play an important role in the retrospectives. Testers are part of the team and bring their unique perspective. Testing occurs in each sprint and vitally contributes to success. All team members, testers and non-testers, can provide input on both testing and non-testing activities.

Retrospectives must occur within a professional environment characterised by mutual trust. The attributes of a successful retrospective are the same as those for any other review as is discussed in previous articles.

Continuous Integration

Delivery of a product increment requires reliable, working, integrated software at the end of every sprint. Continuous integration addresses this challenge by merging all changes made to the software and integrating all changed components regularly, at least once a day. Configuration management, compilation, software build, deployment, and testing are wrapped into a single, automated, repeatable process. Since developers integrate their work constantly, build constantly, and test constantly, defects in code are detected more quickly.

Following the developers’ coding, debugging, and check-in of code into a shared source code repository, a continuous integration process consists of the following automated activities:

  • Static code analysis: executing static code analysis and reporting results
  • Compile: compiling and linking the code, generating the executable files
  • Unit test: executing the unit tests, checking code coverage and reporting test results
  • Deploy: installing the build into a test environment
  • Integration test: executing the integration tests and reporting results
  • Report (dashboard): posting the status of all these activities to a publicly visible location or e-mailing status to the team

An automated build and test process takes place on a daily basis and detects integration errors early and quickly. Continuous integration allows Agile testers to run automated tests regularly, in some cases as part of the continuous integration process itself, and send quick feedback to the team on the quality of the code. These test results are visible to all team members, especially when automated reports are integrated into the process. Automated regression testing can be continuous throughout the iteration. Good automated regression tests cover as much functionality as possible, including user stories delivered in the previous iterations. Good coverage in the automated regression tests helps support building (and testing) large integrated systems. When the regression testing is automated, the Agile testers are freed to concentrate their manual testing on new features, implemented changes, and confirmation testing of defect fixes.

In addition to automated tests, organisations using continuous integration typically use build tools to implement continuous quality control. In addition to running unit and integration tests, such tools can run additional static and dynamic tests, measure and profile performance, extract and format documentation from the source code, and facilitate manual quality assurance processes. This continuous application of quality control aims to improve the quality of the product as well as reduce the time taken to deliver it by replacing the traditional practice of applying quality control after completing all development.

Build tools can be linked to automatic deployment tools, which can fetch the appropriate build from the continuous integration or build server and deploy it into one or more development, test, staging, or even production environments. This reduces the errors and delays associated with relying on specialised staff or programmers to install releases in these environments.

Continuous integration can provide the following benefits:

  • Allows earlier detection and easier root cause analysis of integration problems and conflicting changes
  • Gives the development team regular feedback on whether the code is working
  • Keeps the version of the software being tested within a day of the version being developed
  • Reduces regression risk associated with developer code refactoring due to rapid re-testing of the code base after each small set of changes
  • Provides confidence that each day’s development work is based on a solid foundation
  • Makes progress toward the completion of the product increment visible, encouraging developers and testers
  • Eliminates the schedule risks associated with big-bang integration
  • Provides constant availability of executable software throughout the sprint for testing, demonstration, or education purposes
  • Reduces repetitive manual testing activities
  • Provides quick feedback on decisions made to improve quality and tests

However, continuous integration is not without its risks and challenges:

  • Continuous integration tools have to be introduced and maintained
  • The continuous integration process must be defined and established
  • Test automation requires additional resources and can be complex to establish
  • Thorough test coverage is essential to achieve automated testing advantages
  • Teams sometimes over-rely on unit tests and perform too little system and acceptance testing

Continuous integration requires the use of tools, including tools for testing, tools for automating the build process, and tools for version control.

Release and Iteration Planning 

As mentioned in this article, planning is an on-going activity, and this is the case in Agile lifecycles as well. For Agile lifecycles, two kinds of planning occur, release planning and iteration planning. 

Release planning looks ahead to the release of a product, often a few months ahead of the start of a project. Release planning defines and re-defines the product backlog, and may involve refining larger user stories into a collection of smaller stories. Release planning provides the basis for a test approach and test plan spanning all iterations. Release plans are high-level. 

In release planning, business representatives establish and prioritise the user stories for the release, in collaboration with the team. Based on these user stories, project and quality risks are identified and a high-level effort estimation is performed.

Testers are involved in release planning and especially add value in the following activities:

  • Defining testable user stories, including acceptance criteria
  • Participating in project and quality risk analyses
  • Estimating testing effort associated with the user stories
  • Defining the necessary test levels
  • Planning the testing for the release

After release planning is done, iteration planning for the first iteration starts. Iteration planning looks ahead to the end of a single iteration and is concerned with the iteration backlog.

In iteration planning, the team selects user stories from the prioritised release backlog, elaborates the user stories, performs a risk analysis for the user stories, and estimates the work needed for each user story. If a user story is too vague and attempts to clarify it have failed, the team can refuse to accept it and use the next user story based on priority. The business representatives must answer the team’s questions about each story so the team can understand what they should implement and how to test each story.

The number of stories selected is based on established team velocity and the estimated size of the selected user stories. After the contents of the iteration are finalised, the user stories are broken into tasks, which will be carried out by the appropriate team members.

Testers are involved in iteration planning and especially add value in the following activities:

  • Participating in the detailed risk analysis of user stories
  • Determining the testability of the user stories
  • Creating acceptance tests for the user stories
  • Breaking down user stories into tasks (particularly testing tasks)
  • Estimating testing effort for all testing tasks
  • Identifying functional and non-functional aspects of the system to be tested
  • Supporting and participating in test automation at multiple levels of testing

Release plans may change as the project proceeds, including changes to individual user stories in the product backlog. These changes may be triggered by internal or external factors. Internal factors include delivery capabilities, velocity, and technical issues. External factors include the discovery of new markets and opportunities, new competitors, or business threats that may change release objectives and/or target dates. In addition, iteration plans may change during an iteration. For example, a particular user story that was considered relatively simple during estimation might prove more complex than expected.

These changes can be challenging for testers. Testers must understand the big picture of the release for test planning purposes, and they must have an adequate test basis and test oracle in each iteration for test development purposes as discussed in earlier articles. The required information must be available to the tester early, and yet change must be embraced according to Agile principles. This dilemma requires careful decisions about test strategies and test documentation.

Release and iteration planning should address test planning as well as planning for development activities. Particular test-related issues to address include:

  • The scope of testing, the extent of testing for those areas in scope, the test goals, and the reasons for these decisions.
  • The team members who will carry out the test activities.
  • The test environment and test data needed, when they are needed, and whether any additions or changes to the test environment and/or data will occur prior to or during the project.
  • The timing, sequencing, dependencies, and prerequisites for the functional and non-functional test activities (e.g., how frequently to run regression tests, which features depend on other features or test data, etc.), including how the test activities relate to and depend on development activities.
  • The project and quality risks to be addressed.

In addition, the larger team estimation effort should include consideration of the time and effort needed to complete the required testing activities.

agile, Management

Test management

Test Organisation

Independent Testing

Testing tasks may be done by people in a specific testing role, or by people in another role (e.g., customers). A certain degree of independence often makes the tester more effective at finding defects due to differences between the author’s and the tester’s cognitive biases. Independence is not, however, a replacement for familiarity, and developers can efficiently find many defects in their own code. 

Degrees of independence in testing include the following (from low level of independence to high level):

  • No independent testers; the only form of testing available is developers testing their own code 
  • Independent developers or testers within the development teams or the project team; this could be developers testing their colleagues’ products 
  • Independent test team or group within the organisation, reporting to project management or executive management 
  • Independent testers from the business organisation or user community, or with specialisations in specific test types such as usability, security, performance, regulatory/compliance, or portability 
  • Independent testers external to the organisation, either working on-site (in-house) or off-site (outsourcing)

For most types of projects, it is usually best to have multiple test levels, with some of these levels handled by independent testers. Developers should participate in testing, especially at the lower levels, so as to exercise control over the quality of their own work.

The way in which independence of testing is implemented varies depending on the software development lifecycle model. For example, in Agile development, testers may be part of a development team. In some organisations using Agile methods, these testers may be considered part of a larger independent test team as well. In addition, in such organisations, product owners may perform acceptance testing to validate user stories at the end of each iteration.

Potential benefits of test independence include:

  • Isolation from the development team, may lead to a lack of collaboration, delays in providing feedback to the development team, or an adversarial relationship with the development team
  • Developers may lose a sense of responsibility for quality
  • Independent testers may be seen as a bottleneck
  • Independent testers may lack some important information (e.g., about the test object)

Many organisations are able to successfully achieve the benefits of test independence while avoiding the drawbacks.

Tasks of a Test Manager and Tester 

In this article, two test roles are covered, test managers and testers. The activities and tasks performed by these two roles depend on the project and product context, the skills of the people in the roles, and the organisation.

The test manager is tasked with overall responsibility for the test process and successful leadership of the test activities. The test management role might be performed by a professional test manager, or by a project manager, a development manager, or a quality assurance manager. In larger projects or organisations, several test teams may report to a test manager, test coach, or test coordinator, each team being headed by a test leader or lead tester.

Typical test manager tasks may include:

  • Develop or review a test policy and test strategy for the organisation 
  • Plan the test activities by considering the context, and understanding the test objectives and risks. This may include selecting test approaches, estimating test time, effort and cost, acquiring resources, defining test levels and test cycles, and planning defect management
  • Write and update the test plan(s) 
  • Coordinate the test plan(s) with project managers, product owners, and others 
  • Share testing perspectives with other project activities, such as integration planning 
  • Initiate the analysis, design, implementation, and execution of tests, monitor test progress and results, and check the status of exit criteria (or definition of done) and facilitate test completion activities 
  • Prepare and deliver test progress reports and test summary reports based on the information gathered 
  • Adapt planning based on test results and progress (sometimes documented in test progress reports, and/or in test summary reports for other testing already completed on the project) and take any actions necessary for test control 
  • Support setting up the defect management system and adequate configuration management of test-ware 
  • Introduce suitable metrics for measuring test progress and evaluating the quality of the testing and the product
  • Support the selection and implementation of tools to support the test process, including recommending the budget for tool selection (and possibly purchase and/or support), allocating time and effort for pilot projects, and providing continuing support in the use of the tool(s) 
  • Decide about the implementation of test environment(s) 
  • Promote and advocate the testers, the test team, and the test profession within the organisation 
  • Develop the skills and careers of testers (e.g., through training plans, performance evaluations, coaching, etc.)

The way in which the test manager role is carried out varies depending on the software development lifecycle. For example, in Agile development, some of the tasks mentioned above are handled by the Agile team, especially those tasks concerned with the day-to-day testing done within the team, often by a tester working within the team. Some of the tasks that span multiple teams or the entire organisation, or that have to do with personnel management, may be done by test managers outside of the development team, who are sometimes called test coaches.

Typical tester tasks may include:

  • Review and contribute to test plans 
  • Analyse, review, and assess requirements, user stories and acceptance criteria, specifications, and models for testability (i.e., the test basis) 
  • Identify and document test conditions, and capture traceability between test cases, test conditions, and the test basis 
  • Design, set up, and verify test environment(s), often coordinating with system administration and network management 
  • Design and implement test cases and test procedures 
  • Prepare and acquire test data
  • Create the detailed test execution schedule 
  • Execute tests, evaluate the results, and document deviations from expected results 
  • Use appropriate tools to facilitate the test process 
  • Automate tests as needed (may be supported by a developer or a test automation expert)
  • Evaluate non-functional characteristics such as performance efficiency, reliability, usability, security, compatibility, and portability 
  • Review tests developed by others

People who work on test analysis, test design, specific test types, or test automation may be specialists in these roles. Depending on the risks related to the product and the project, and the software development lifecycle model selected, different people may take over the role of tester at different test levels. For example, at the component testing level and the component integration testing level, the role of a tester is often done by developers. At the acceptance test level, the role of a tester is often done by business analysts, subject matter experts, and users. At the system test level and the system integration test level, the role of a tester is often done by an independent test team. At the operational acceptance test level, the role of a tester is often done by operations and/or systems administration staff.

Test Planning and Estimation

Purpose and Content of a Test Plan

A test plan outlines test activities for development and maintenance projects. Planning is influenced by the test policy and test strategy of the organisation, the development lifecycles and methods being used, the scope of testing, objectives, risks, constraints, criticality, testability, and the availability of resources. 

As the project and test planning progress, more information becomes available and more detail can be included in the test plan. Test planning is a continuous activity and is performed throughout the product’s lifecycle. (Note that the product’s lifecycle may extend beyond a project’s scope to include the maintenance phase.) Feedback from test activities should be used to recognise changing risks so that planning can be adjusted. Planning may be documented in a master test plan and in separate test plans for test levels, such as system testing and acceptance testing, or for separate test types, such as usability testing and performance testing. Test planning activities may include the following and some of these may be documented in a test plan:

  • Determining the scope, objectives, and risks of testing
  • Defining the overall approach of testing
  • Integrating and coordinating the test activities into the software lifecycle activities
  • Making decisions about what to test, the people and other resources required to perform the various test activities, and how test activities will be carried out
  • Scheduling of test analysis, design, implementation, execution, and evaluation activities, either on particular dates (e.g., in sequential development) or in the context of each iteration (e.g., in iterative development)
  • Selecting metrics for test monitoring and control
  • Budgeting for the test activities
  • Determining the level of detail and structure for test documentation (e.g., by providing templates or example documents)

The content of test plans vary, and can extend beyond the topics identified above.

Test Strategy and Test Approach

A test strategy provides a generalised description of the test process, usually at the product or organisational level. Common types of test strategies include:

  • Analytical: This type of test strategy is based on an analysis of some factor (e.g., requirement or risk). Risk-based testing is an example of an analytical approach, where tests are designed and prioritised based on the level of risk.
  • Model-Based: In this type of test strategy, tests are designed based on some model of some required aspect of the product, such as a function, a business process, an internal structure, or a non-functional characteristic (e.g., reliability). Examples of such models include business process models, state models, and reliability growth models.
  • Methodical: This type of test strategy relies on making systematic use of some predefined set of tests or test conditions, such as a taxonomy of common or likely types of failures, a list of important quality characteristics, or company-wide look-and-feel standards for mobile apps or web pages. 
  • Process-compliant (or standard-compliant): This type of test strategy involves analysing, designing, and implementing tests based on external rules and standards, such as those specified by industry-specific standards, by process documentation, by the rigorous identification and use of the test basis, or by any process or standard imposed on or by the organisation. 
  • Directed (or consultative): This type of test strategy is driven primarily by the advice, guidance, or instructions of stakeholders, business domain experts, or technology experts, who may be outside the test team or outside the organisation itself.
  • Regression-averse: This type of test strategy is motivated by a desire to avoid regression of existing capabilities. This test strategy includes reuse of existing testware (especially test cases and test data), extensive automation of regression tests, and standard test suites.
  • Reactive: In this type of test strategy, testing is reactive to the component or system being tested, and the events occurring during test execution, rather than being pre-planned (as the preceding strategies are). Tests are designed and implemented, and may immediately be executed in response to knowledge gained from prior test results. Exploratory testing is a common technique employed in reactive strategies.

An appropriate test strategy is often created by combining several of these types of test strategies. For example, risk-based testing (an analytical strategy) can be combined with exploratory testing (a reactive strategy); they complement each other and may achieve more effective testing when used together.

While the test strategy provides a generalised description of the test process, the test approach tailors the test strategy for a particular project or release. The test approach is the starting point for selecting the test techniques, test levels, and test types, and for defining the entry criteria and exit criteria (or definition of ready and definition of done, respectively). The tailoring of the strategy is based on decisions made in relation to the complexity and goals of the project, the type of product being developed, and product risk analysis. The selected approach depends on the context and may consider factors such as risks, safety, available resources and skills, technology, the nature of the system (e.g., custom-built versus COTS), test objectives, and regulations.

Entry Criteria and Exit Criteria (Definition of Ready and Definition of Done)

In order to exercise effective control over the quality of the software, and of the testing, it is advisable to have criteria which define when a given test activity should start and when the activity is complete. Entry criteria (more typically called definition of ready in Agile development) define the preconditions for undertaking a given test activity. If entry criteria are not met, it is likely that the activity will prove more difficult, more time-consuming, more costly, and more risky. Exit criteria (more typically called definition of done in Agile development) define what conditions must be achieved in order to declare a test level or a set of tests completed. Entry and exit criteria should be defined for each test level and test type, and will differ based on the test objectives.

Typical entry criteria include: 

  • Availability of testable requirements, user stories, and/or models (e.g., when following a model-based testing strategy)
  • Availability of test items that have met the exit criteria for any prior test levels
  • Availability of test environment
  • Availability of necessary test tools
  • Availability of test data and other necessary resources

Typical exit criteria include:

  • Planned tests have been executed
  • A defined level of coverage (e.g., of requirements, user stories, acceptance criteria, risks, code) has been achieved 
  • The number of unresolved defects is within an agreed limit 
  • The number of estimated remaining defects is sufficiently low
  • The evaluated levels of reliability, performance efficiency, usability, security, and other relevant quality characteristics are sufficient

Even without exit criteria being satisfied, it is also common for test activities to be curtailed due to the budget being expended, the scheduled time being completed, and/or pressure to bring the product to market. It can be acceptable to end testing under such circumstances, if the project stakeholders and business owners have reviewed and accepted the risk to go live without further testing.

Test Execution Schedule

Once the various test cases and test procedures are produced (with some test procedures potentially automated) and assembled into test suites, the test suites can be arranged in a test execution schedule that defines the order in which they are to be run. The test execution schedule should take into account such factors as prioritizations, dependencies, confirmation tests, regression tests, and the most efficient sequence for executing the tests.

Ideally, test cases would be ordered to run based on their priority levels, usually by executing the test cases with the highest priority first. However, this practice may not work if the test cases have dependencies or the features being tested have dependencies. If a test case with a higher priority is dependent on a test case with a lower priority, the lower priority test case must be executed first. Similarly, if there are dependencies across test cases, they must be ordered appropriately regardless of their relative priorities. Confirmation and regression tests must be prioritised as well, based on the importance of rapid feedback on changes, but here again dependencies may apply.

In some cases, various sequences of tests are possible, with differing levels of efficiency associated with those sequences. In such cases, trade-offs between efficiency of test execution versus adherence to prioritisation must be made.

Factors Influencing the Test Effort

Test effort estimation involves predicting the amount of test-related work that will be needed in order to meet the objectives of the testing for a particular project, release, or iteration. Factors influencing the test effort may include characteristics of the product, characteristics of the development process, characteristics of the people, and the test results, as shown below.

Product characteristics

  • The risks associated with the product
  • The quality of the test basis
  • The size of the product
  • The complexity of the product domain
  • The requirements for quality characteristics (e.g., security, reliability) 
  • The required level of detail for test documentation 
  • Requirements for legal and regulatory compliance

Development characteristics process

  • The stability and maturity of the organisation
  • The development model in use
  • The approach to test
  • The tools used
  • The test to process 
  • Time pressure

People characteristics

  • The skills and experience of the people involved, especially with similar projects and products (e.g., domain knowledge)
  • Team cohesion and leadership

Test results

  • The number and severity of defects found
  • The amount of re-work required

Test Estimation Techniques

There are a number of estimation techniques used to determine the effort required for adequate testing. Two of the most commonly used techniques are:

  • The metrics-based technique: estimating the test effort based on metrics of former similar projects, or based on typical values
  • The expert-based technique: estimating the test effort based on the experience of the owners of the testing tasks or by experts

For example, in Agile development, burn-down charts are examples of the metrics-based approach as effort remaining is being captured and reported, and is then used to feed into the team’s velocity to determine the amount of work the team can do in the next iteration; whereas planning poker, also called scrum poker, is an example of the expert-based approach, as team members are estimating the effort to deliver a feature based on their experience.

Within sequential projects, defect removal models are examples of the metrics-based approach, where volumes of defects and time to remove them are captured and reported, which then provides a basis for estimating future projects of a similar nature; whereas the Wideband Delphi estimation technique is an example of the expert-based approach in which a group of experts provides estimates based on their experience.

Test Monitoring and Control

The purpose of test monitoring is to gather information and provide feedback and visibility about test activities. Information to be monitored may be collected manually or automatically and should be used to assess test progress and to measure whether the test exit criteria, or the testing tasks associated with an Agile project’s definition of done, are satisfied, such as meeting the targets for coverage of product risks, requirements, or acceptance criteria.

Test control describes any guiding or corrective actions taken as a result of information and metrics gathered and (possibly) reported. Actions may cover any test activity and may affect any other software lifecycle activity.

Examples of test control actions include: 

  • Re-prioritising tests when an identified risk occurs (e.g., software delivered late)
  • Changing the test schedule due to availability or unavailability of a test environment or other resources
  • Re-evaluating whether a test item meets an entry or exit criterion due to rework

Metrics Used in Testing

Metrics can be collected during and at the end of test activities in order to assess:

  • Progress against the planned schedule and budget
  • Current quality of the test object
  • Adequacy of the test approach
  • Effectiveness of the test activities with respect to the objectives

Common test metrics include:

  • Percentage of planned work done in test case preparation (or percentage of planned test cases implemented)
  • Percentage of planned work done in test environment preparation
  • Test case execution (e.g., number of test cases run/not run, test cases passed/failed, and/or test conditions passed/failed)
  • Defect information (e.g., defect density, defects found and fixed, failure rate, and confirmation test results)
  • Test coverage of requirements, user stories, acceptance criteria, risks, or code
  • Task completion, resource allocation and usage, and effort
  • Cost of testing, including the cost compared to the benefit of finding the next defect or the cost compared to the benefit of running the next test

Audiences, Contents, and Purposes for Test Reports

The purpose of test reporting is to summarise and communicate test activity information, both during and at the end of a test activity (e.g., a test level). The test report prepared during a test activity may be referred to as a test progress report, while a test report prepared at the end of a test activity may be referred to as a test summary report.

During test monitoring and control, the test manager regularly issues test progress reports for stakeholders. In addition to content common to test progress reports and test summary reports, typical test progress reports may also include:

  • The status of the test activities and progress against the test plan
  • Factors impeding progress
  • Testing planned for the next reporting period
  • The quality of the test objects

When exit criteria are reached, the test manager issues the test summary report. This report provides a summary of the testing performed, based on the latest test progress report and any other relevant information.

Typical test summary reports may include:

  • Summary of testing performed
  • Information on what occurred during a test period
  • Deviations from plan, including deviations in schedule, duration, or effort of test activities
  • Status of testing and product quality with respect to the exit criteria or definition of done
  • Factors that have blocked or continue to block progress
  • Metrics of defects, test cases, test coverage, activity progress, and resource consumption.
  • Residual risks
  • Reusable test work products produced

The contents of a test report will vary depending on the project, the organisational requirements, and the software development lifecycle. For example, a complex project with many stakeholders or a regulated project may require more detailed and rigorous reporting than a quick software update. As another example, in Agile development, test progress reporting may be incorporated into task boards, defect summaries, and burn-down charts, which may be discussed during a daily stand-up meeting.

In addition to tailoring test reports based on the context of the project, test reports should be tailored based on the report’s audience. The type and amount of information that should be included for a technical audience or a test team may be different from what would be included in an executive summary report. In the first case, detailed information on defect types and trends may be important. In the latter case, a high-level report (e.g., a status summary of defects by priority, budget, schedule, and test conditions passed/failed/not tested) may be more appropriate.

Configuration Management

The purpose of configuration management is to establish and maintain the integrity of the component or system, the test-ware, and their relationships to one another through the project and product lifecycle. 

To properly support testing, configuration management may involve ensuring the following:

  • All test items are uniquely identified, version controlled, tracked for changes, and related to each other
  • All items of test-ware are uniquely identified, version controlled, tracked for changes, related to each other and related to versions of the test item(s) so that traceability can be maintained throughout the test process
  • All identified documents and software items are referenced unambiguously in test documentation

During test planning, configuration management procedures and infrastructure (tools) should be identified and implemented.

Risks and Testing

Definition of Risk

Risk involves the possibility of an event in the future which has negative consequences. The level of risk is determined by the likelihood of the event and the impact (the harm) from that event.

Product and Project Risks

Product risk involves the possibility that a work product (e.g., a specification, component, system, or test) may fail to satisfy the legitimate needs of its users and/or stakeholders. When the product risks are associated with specific quality characteristics of a product (e.g., functional suitability, reliability, performance efficiency, usability, security, compatibility, maintainability, and portability), product risks are also called quality risks. Examples of product risks include:

  • Software might not perform its intended functions according to the specification
  • Software might not perform its intended functions according to user, customer, and/or stakeholder needs
  • A system architecture may not adequately support some non-functional requirement(s)
  • A particular computation may be performed incorrectly in some circumstances
  • A loop control structure may be coded incorrectly
  • Response-times may be inadequate for a high-performance transaction processing system
  • User experience (UX) feedback might not meet product expectations

Project risk involves situations that, should they occur, may have a negative effect on a project’s ability to achieve its objectives. Examples of project risks include:

  • Project issues:
    • Delays may occur in delivery, task completion, or satisfaction of exit criteria or definition of done 
    • Inaccurate estimates, reallocation of funds to higher priority projects, or general cost-cutting across the organisation may result in inadequate funding 
    • Late changes may result in substantial re-work
  • Organisational issues: 
    • Skills, training, and staff may not be sufficient 
    • Personnel issues may cause conflict and problems 
    • Users, business staff, or subject matter experts may not be available due to conflicting business priorities
  • Political issues:
    • Testers may not communicate their needs and/or the test results adequately
    • Developers and/or testers may fail to follow up on information found in testing and reviews (e.g., not improving development and testing practices)
    • There may be an improper attitude toward, or expectations of, testing (e.g., not appreciating the value of finding defects during testing)
  • Technical issues: 
    • Requirements may not be defined well enough 
    • The requirements may not be met, given existing constraints 
    • The test environment may not be ready on time 
    • Data conversion, migration planning, and their tool support may be late 
    • Weaknesses in the development process may impact the consistency or quality of project work products such as design, code, configuration, test data, and test cases
    • Poor defect management and similar problems may result in accumulated defects and other technical debt
  • Supplier issues:
    • A third party may fail to deliver a necessary product or service, or go bankrupt
    • Contractual issues may cause problems to the project

Project risks may affect both development activities and test activities. In some cases, project managers are responsible for handling all project risks, but it is not unusual for test managers to have responsibility for test-related project risks.

Product Quality and Risk-based Testing

Risk is used to focus the effort required during testing. It is used to decide where and when to start testing and to identify areas that need more attention. Testing is used to reduce the probability of an adverse event occurring, or to reduce the impact of an adverse event. Testing is used as a risk mitigation activity, to provide information about identified risks, as well as providing information on residual (unresolved) risks. 

A risk-based approach to testing provides proactive opportunities to reduce the levels of product risk. It involves product risk analysis, which includes the identification of product risks and the assessment of each risk’s likelihood and impact. The resulting product risk information is used to guide test planning, the specification, preparation and execution of test cases, and test monitoring and control. Analysing product risks early contributes to the success of a project. 

In a risk-based approach, the results of product risk analysis are used to:

  • Determine the test techniques to be employed
  • Determine the particular levels and types of testing to be performed (e.g., security testing, accessibility testing)
  • Determine the extent of testing to be carried out
  • Prioritise testing in an attempt to find the critical defects as early as possible 
  • Determine whether any activities in addition to testing could be employed to reduce risk (e.g., providing training to inexperienced designers)

Risk-based testing draws on the collective knowledge and insight of the project stakeholders to carry out product risk analysis. To ensure that the likelihood of a product failure is minimised, risk management activities provide a disciplined approach to:

  • Analyse (and re-evaluate on a regular basis) what can go wrong (risks)
  • Determine which risks are important to deal with
  • Implement actions to mitigate those risks
  • Make contingency plans to deal with the risks should they become actual events

In addition, testing may identify new risks, help to determine what risks should be mitigated, and lower uncertainty about risks.

Defect Management

Since one of the objectives of testing is to find defects, defects found during testing should be logged. The way in which defects are logged may vary, depending on the context of the component or system being tested, the test level, and the software development lifecycle model. Any defects identified should be investigated and should be tracked from discovery and classification to their resolution (e.g., correction of the defects and successful confirmation testing of the solution, deferral to a subsequent release, acceptance as a permanent product limitation, etc.). In order to manage all defects to resolution, an organisation should establish a defect management process which includes a workflow and rules for classification. This process must be agreed with all those participating in defect management, including architects, designers, developers, testers, and product owners. In some organisations, defect logging and tracking may be very informal. 

During the defect management process, some of the reports may turn out to describe false positives, not actual failures due to defects. For example, a test may fail when a network connection is broken or times out. This behaviour does not result from a defect in the test object, but is an anomaly that needs to be investigated. Testers should attempt to minimise the number of false positives reported as defects. 

Defects may be reported during coding, static analysis, reviews, or during dynamic testing, or use of a software product. Defects may be reported for issues in code or working systems, or in any type of documentation including requirements, user stories and acceptance criteria, development documents, test documents, user manuals, or installation guides. In order to have an effective and efficient defect management process, organisations may define standards for the attributes, classification, and workflow of defects.

Typical defect reports have the following objectives: 

  • Provide developers and other parties with information about any adverse event that occurred, to enable them to identify specific effects, to isolate the problem with a minimal reproducing test, and to correct the potential defect(s), as needed or to otherwise resolve the problem
  • Provide test managers a means of tracking the quality of the work product and the impact on the testing (e.g., if a lot of defects are reported, the testers will have spent a lot of time reporting them instead of running tests, and there will be more confirmation testing needed)
  • Provide ideas for development and test process improvement

A defect report filed during dynamic testing typically includes:

  • An identifier
  • A title and a short summary of the defect being reported
  • Date of the defect report, issuing organization, and author
  • Identification of the test item (configuration item being tested) and environment
  • The development lifecycle phase(s) in which the defect was observed
  • A description of the defect to enable reproduction and resolution, including logs, database dumps, screenshots, or recordings (if found during test execution)
  • Expected and actual results
  • Scope or degree of impact (severity) of the defect on the interests of stakeholder(s)
  • Urgency/priority to fix
  • State of the defect report (e.g., open, deferred, duplicate, waiting to be fixed, awaiting confirmation testing, re-opened, closed)
  • Conclusions, recommendations and approvals
  • Global issues, such as other areas that may be affected by a change resulting from the defect
  • Change history, such as the sequence of actions taken by project team members with respect to the defect to isolate, repair, and confirm it as fixed
  • References, including the test case that revealed the problem

Some of these details may be automatically included and/or managed when using defect management tools, e.g., automatic assignment of an identifier, assignment and update of the defect report state during the workflow, etc. Defects found during static testing, particularly reviews, will normally be documented in a different way, e.g., in review meeting notes.

agile

Agile software development

The fundamentals of agile software development

A tester on an Agile project will work differently than one working on a traditional project. Testers must understand the values and principles that underpin Agile projects, and how testers are an integral part of a whole-team approach together with developers and business representatives. The members in an Agile project communicate with each other early and frequently, which helps with removing defects early and developing a quality product.

Agile software development and the agile manifesto

In 2001, a group of individuals, representing the most widely used lightweight software development methodologies, agreed on a common set of values and principles which became known as the Manifesto for Agile Software Development or the Agile Manifesto [Agilemanifesto]. The Agile Manifesto contains four statements of values:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

The Agile Manifesto argues that although the concepts on the right have value, those on the left have greater value.

Individuals and interactions

Agile development is very people-centered. Teams of people build software, and it is through continuous communication and interaction, rather than a reliance on tools or processes, that teams can work most effectively.

Working software

From a customer perspective, working software is much more useful and valuable than overly detailed documentation and it provides an opportunity to give the development team rapid feedback. In addition, because working software, albeit with reduced functionality, is available much earlier in the development lifecycle, Agile development can confer significant time-to-market advantage. Agile development is, therefore, especially useful in rapidly changing business environments where the problems and/or solutions are unclear or where the business wishes to innovate in new problem domains.

Customer collaboration

Customers often find great difficulty in specifying the system that they require. Collaborating directly with the customer improves the likelihood of understanding exactly what the customer requires. While having contracts with customers may be important, working in regular and close collaboration with them is likely to bring more success to the project.

Responding to change

Change is inevitable in software projects. The environment in which the business operates, legislation, competitor activity, technology advances, and other factors can have major influences on the project and its objectives. These factors must be accommodated by the development process. As such, having flexibility in work practices to embrace change is more important than simply adhering rigidly to a plan.

Agile principles

The core Agile Manifesto values are captured in twelve principles:

  • Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
  • Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
  • Deliver working software frequently, at intervals of between a few weeks to a few months, with a preference to the shorter timescale.
  • Business people and developers must work together daily throughout the project.
  • Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
  • The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
  • Working software is the primary measure of progress.
  • Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
  • Continuous attention to technical excellence and good design enhances agility.
  • Simplicity—the art of maximizing the amount of work not done—is essential.
  • The best architectures, requirements, and designs emerge from self-organizing teams.
  • At regular intervals, the team reflects on how to become more effective, then tunes andadjusts its behavior accordingly.

The different Agile methodologies provide prescriptive practices to put these values and principles into action.