Framework for Qualifying AI Systems in Industrial Quality Inspection (AIQualify)

The manufacturing industry and its end customers are placing ever higher quality demands. Instead of manual quality inspection, which is sometimes inefficient and prone to errors, individual industries are increasingly relying on automated inspections, for example by means of optical inspection systems and subsequent image processing. However, traditional image processing reaches its limits especially when there is a high variability of defects and/or parts. Machine learning (ML) methods, a subfield of artificial intelligence (AI), are increasingly making it possible to overcome these limitations. These approaches are trained entirely on data without the need for explicit rules provided by humans. Features of ML that differ from traditional software/AI approaches and make qualification difficult include limited explainability of decision making, insufficient robustness to small changes in input data, or the lack of meaningful criteria that can be used as proof of qualification. There is also a lack of appropriate standards and development methods to demonstrate the suitability/qualification of an ML-based AI system. Thus, concerns about reliability or accuracy of ML-based AI systems have so far prevented their widespread industrial use.

The goal of the project is the development of a software framework for the qualification and auditing of ML-based AI systems in industrial quality inspection. The framework consists of a procedure model including software-supported methods and tools, which allows (a) the determination and formulation of test and evaluation criteria as well as (b) the acceptance of the AI system along these criteria. The framework should be modular in design, allowing easy integration and extension of testing or auditing modules. The framework is intended to enable small and medium-sized enterprises (SMEs) in particular to qualify third-party AI systems so that they can evaluate the performance of the AI system even without their own AI experts. The framework is also intended to be applicable for AI service providers for self-disclosure or for independent testing bodies, for example for certification.

In this project, a systematic identification and formulation of application-specific and AI-specific testing and evaluation criteria are proposed for the different stages of AI system development based on the framework of assurance cases. The core element of such assurance cases is the so-called primary claim, which specifies the goal to be achieved by the AI system, for example, that a certain pseudo error rate does not fall below a defined threshold. For further structuring, the primary claim can be broken down into sub-claims or criteria to be met. In addition, all assumptions made and contextual information must be integrated into the assurance cases. In order to provide a structured assurance case argument to meet the claim, any system-related or process-related proof required is to be captured. These assurance cases are to be used in auditing the AI system along the different stages of development. Therefore, this project aims at developing a computer-aided assistance system as part of a software framework for the supportive creation of assurance cases, which is based on prefabricated templates corresponding to the application context of the quality inspections and centrally manages the assurance cases created. For this purpose, a computer-aided auditing platform is to be developed for the framework.

Developing high quality AI systems and auditing them requires professionals with profound AI knowledge. However, numerous studies show that the low use of AI systems, especially in small and medium-sized enterprises (SMEs), is due to a lack of know-how and skilled workers. As a result, SMEs rely on service providers to develop custom-fit, high-performance AI systems. However, these systems need to be audited to ensure that the requirements are met, which in turn requires AI expertise. In order to break this vicious circle, this project aims to provide a solution, especially for SMEs, which allows the qualification of AI systems even without in-house AI experts. Significant economic advantages result for all industries that manufacture technical components and use AI systems for quality inspection or plan to use them. Important economic sectors include mechanical engineering, process technology, medical technology, microsystems technology and automotive engineering. Furthermore, there are advantages especially in areas where high quantities are produced and thus manual quality inspection is impossible or uneconomical, where high quality requirements apply or where production processes are not sufficiently stable, so that deviations in product quality can occur more frequently. Furthermore, ML-based quality inspection is suitable for multi-variant production and/or production with small batch sizes, since rule-based algorithms for quality inspection are too rigid here and cause recurring high setup efforts (e.g. for rule definition, parameterization). Beyond manufacturing companies, the project offers innovation potential for AI and testing service providers to self-audit AI systems for customers or to use the research results in the development process to be able to test the fit of the system to customer requirements at an early stage.

Manual visual inspection is still widely used in industry, especially for surface inspection of components. Automated solutions based on camera images and classic rule-based inspection methods often fail due to the variance of the defect characteristics in the image caused by the optical properties of the components to be inspected and environmental influences on the inspection system. AI-based optical inspection systems cope much better with these changes, but usually require sufficient image data, especially images with the annotated defects that occur, for training the AI to reliably detect and classify defects. However, usually only a few defective components are available during the development of the inspection system, which can be used for the design and parameterization of the optical inspection system with the image processing software.

In order to create a broad database for AI-based inspection solutions and to drastically reduce or completely avoid the time-consuming labeling of defect images, sensor-realistic images of the component with the different defects are generated synthetically. For the use case of surface inspection of perforated discs, synthetic defect images were generated in large numbers and variations for the different defect types of dents, dings, scratches and edge chipping, and additionally some real camera images of the perforated discs with different defects were also captured.

Based on the synthetically generated images, a robust AI-based defect detection and classification could be implemented. A demonstrator setup for surface inspection of the perforated discs with camera, illumination and AI-based evaluation software as well as the associated extensive database of real and synthetic images is available and can be investigated within the AIQualify project with respect to the qualification of AI-based inspection applications.