Workshop –
Testing AI-Based Systems
The testing of traditional systems is well-understood, but AI-Based systems, which are becoming more prevalent and critical to our daily lives, introduce new challenges. This course introduces the key concepts of Artificial Intelligence (AI), how we decide acceptance criteria and how we test AI-based systems. These systems are typically complex (e.g. deep neural nets), based on big data, poorly specified and non-deterministic, which creates many new challenges and opportunities for testing them.
The course introduces the variety of types of AI-based systems in use today and explains machine-learning, which is often a key part of these systems. We will look at how the setting of acceptance criteria needs to change for these systems, and also show how the characteristics of AI-based systems make testing more difficult than for traditional systems.
The unique development process used for machine learning systems introduces a number of potential areas where mistakes can be made, and defects introduced. The course provides insights into how an independent perspective can help contribute to the most effective implementation of this process and prevent problems such as data bias, overfitting, underfitting and misclassification of input data.
The inclusion of AI makes the testing of AI-Based Systems more challenging from several perspectives. Their probabilistic and non-deterministic nature and inherent complexity makes the derivation of expected results particularly difficult. This creates a test oracle problem that leads to the need for new testing approaches to complement more traditional test techniques.
The course explains how some traditional black box testing techniques, such as combinatorial testing, back-to-back testing and A/B testing, are especially useful for these systems. It then introduces some approaches, specifically focussed on AI-Based Systems, such as adversarial testing and metamorphic testing in more detail.
The need for white box testing of traditional systems has long been accepted for critical systems. The use of AI-Based systems in critical situations, such as controlling self-driving cars or providing medical advice suggests that white box approaches will also be needed for these systems. Measures of white box coverage for neural networks are introduced, along with tools to support them.
The need for virtual test environments is demonstrated using the case of autonomous cars as an example. The corresponding requirements for test scenarios (for both defect-finding and regulation) are considered.
Finally, the use of AI as the basis of tools to support testing are briefly introduced by looking at examples of the successful application of AI to common testing problems.
Training Expectation
At the end of this course candidates will be able to:
- Understand the current state of AI and expected advances in the near future;
- Interpret and provide guidance on the specification of acceptance criteria for AI-Based Systems;
- Contribute to the development process for machine learning systems and suggest opportunities for influencing their quality;
- Understand the new challenges of testing AI-Based Systems, such as their complexity and non-determinism;
- Contribute to the test strategy for an AI-Based Systems;
- Apply black box and white box test design techniques to generate test suites for AI-Based Systems;
- Recognize the need for virtual test environments to support the release of complex AI-Based Systems;
- Understand the current state of testing supported by AI.
Target Audience
This course is focused on individuals with an interest in, or a need to perform, the testing of AI-Based Systems, especially those working in areas such as autonomous systems, big data, retail, finance, engineering and IT services. This includes people in roles such as system testers, test analysts, test engineers, test consultants, test managers, user acceptance testers, business analysts and systems developers.
Course length/numbers
No limit on numbers, but with smaller groups the tutorial will be more interactive and allow delegates’ specific questions to be addressed.
This can be delivered as either a two-hour, half-day, full day or two-day workshop.