QA Testing

Accessibility testing isn't the end, it starts at the beginning with test criteria

Why this matters

Apart from using assistive technology this process shouldn’t look different from other QA testing processes.

Team commitments

What requirements must the team agree to meet if your organization is going to be successful?

Testers need to be capable of manually testing and describing the problems they find, not just running automated scans and listing WCAG criterion failures.

Can explain why accessibility is a requirement

Team members can possess varying levels of technical understanding, but all should be alert to the basic reasons accessibility is a requirement.

Accessibility target policies

At a minimum, any team needs to understand accessibility targets exist, are defined by enterprise policy and enforced by leadership.

This bare minimum adoption achieves some level of compliance, even when it’s begrudging.

However, if the reasons for accessibility policies are taken to heart in service of the customer, products can dramatically improve.

Living your organization’s values

Accessibility isn’t just the right thing to do, it’s the smartest thing to do.

Every organization has a set of values, often including core ethical tenets like treating people with respect and doing the right thing.

How does accessibility fit those values? How does ignoring accessibility breach them?

A tool for innovation

Accessible design and development builds better products for everyone. When teams put accessibility at the beginning of their processes, they create more valuable products for your enterprise.

Competitive advantage

26% of the US population has a disability that requires accommodation, making people with disabilities the largest minority in the United States. This adds up to billions of dollars in combined purchasing power.

Accessibility is the law. Designing and building accessibility into products also helps the enterprise avoid legal risk and liability due to a customer complaint.

Can characterize automated and manual testing

Manual testing

Manual testing is precisely that: a human actually testing the experience using the screenreader and browser combinations you need to support.

Experts can deliver an organized report of defects by severity. This is a necessary tool for improving the customer experience.

Limits of manual testing

A manual test isn’t the same as a usability study, but it is effective in uncovering the issues your customers experience.

Manual testing is performed by people, and perception of what constitutes a defect can vary slightly from one tester to the next. It will be helpful for your testers to reference your severity definitions, and use uniform testing acceptance criteria like MagentaA11y.com.

Automated testing

Automated tests find programmatic errors, but can’t describe actual customer experiences.

How to use automated scans effectively

Scanning tools quickly pinpoint syntax defects in code. Some flagged issues won’t affect the customer experience, but you should exercise scrutiny and manual testing if a web page is riddled with invalid code and errors.

Limitations of automated scans

Testing tools have value. But it’s important to understand their drawbacks. Even the most robust tools can identify less than half of the potential defects on a page.

Code can be inaccessible for a person using their keyboard or screen reader without being flagged as invalid markups by an automated tools.

Practical examples
  • Automated scans can instantaneously test checkboxes for properly associated labels and other code attributes, but can’t tell you if the labels make sense.
  • Automation tools can flag an image for missing alt text, but can’t tell you if it would be better for the screen reader to ignore that particular icon.
  • Custom components, like an accordion expander, could be inaccessible with the keyboard and yet be formed of valid code that won’t raise an error.

Can effectively use assistive technology

Testers need a solid understanding of how to use assistive technology and what expectations to have from components.

When testers don’t understand how to use the keyboard or screenreader on desktop and mobile, they will create invalid defect tickets. This will waste time as developers review and may even try to solve a problem that isn’t there, potentially creating a real problem.

Can understand accessibility acceptance criteria

Acceptance criteria have to be specific enough to cover core functionality, including every quirk or difference between the five screen reader and browser combinations, but broad enough to not become overly verbose. MagentaA11y.com is one way to generate acceptance criteria.

What are the components of acceptance criteria

Understand each HTML element and how it fits into a group.

Name

This is how the element will be read to the screen reader. For example, the name of a link is typically the inner text of the link. The name of an input is typically the label.

Role

Any element has a role. It could be an input’s role is “radio” or a submit button’s role is “button.”

State

Many controls have a state. For instance, a checkbox input can be “checked” or “unchecked.”

Group

Nearly all components work as part of a bigger context or group of elements.

For instance, radio inputs need a name. Headings should exist in an ordered logical way, starting with an H1, that is the page title, with major sections using an H2 at the beginning.

Can interpret accessibility assessments

Teams will need help and training on how to interpret, prioritize and act on information from full assessments. The content is often based around WCAG criteria and may or may not offer techniques for remediation.

Definition of ready requires accessibility acceptance criteria

Any QA process defines what is required for testing to begin. This is called a “definition of ready.” Without a definition of ready a team cannot estimate the amount of work required.

Acceptance criteria from MagentaA11y can used to clearly define test instructions.

Assistive technology test suite

Prioritize your test suite by what devices and browser combinations your customers are using.

Setup of these tools can vary. For instance, if your team is already using Macs then they already have VoiceOver and can install NVDA using a virtual machine environment, without having to set up a separate physical PC.

Desktop

Keyboard

Successful keyboard interaction is a prerequisite for testing with a screen reader.

PC + NVDA + Chrome

If you’re only going to test with one screen reader, it should be NVDA. It is free and it is very demanding of compliant code.

Mac + VoiceOver + Safari

If you’re testing with two screen readers, VoiceOver should be the second. VoiceOver is built into MacOS and pairs with Safari.

PC + JAWS + Chrome

Most of your customers with vision disabilities in the U.S. will be using JAWS because it’s subsidized by the federal government. JAWS is more forgiving of non-compliant code than NVDA or VoiceOver, so despite its market share, it is not always ideal as your sole testing platform.

Mobile

iOS device + VoiceOver + Bluetooth keyboard

VoiceOver is built into iOS and pairs with Safari.

Android device + Talkback + Bluetooth keyboard

Talkback is a free screen reader for Android and pairs with Chrome.

Uniform automated testing tools

There are a multitude of free automated testing tools available. Simplify compliance by using one uniform tool used for development, QA testing and pipeline gating.

Your checklist

Download checklist
Core responsibilities
Testing for accessibility
Download checklist