Detail of YY/T 1833.3-2022

Introduction of YY/T 1833.3-2022

YY/T 1833.3-2022 Artificial intelligence medical device - Quality requirements and evaluation - Part 3: General requirement for data annotation 1 Scope This document specifies the general requirements and evaluation methods for data annotation of artificial intelligence medical device. It is applicable to data annotation activities of artificial intelligence medical device. 2 Normative references The following documents contain provisions which, through reference in this text, constitute provisions of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies. YY/T 1833.1 Artificial intelligence medical device - Quality requirements and evaluation - Part 1: Terminology YY/T 1833.2 Artificial intelligence medical device - Quality requirements and evaluation - Part 2: General requirements for datasets 3 Terms and definitions For the purposes of this document, the terms and definitions given in YY/T 1833.1 and YY/T 1833.2 as well as the following apply. 3.1 annotation task activity for purposefully analyzing a set of data to add external knowledge 3.2 annotation object specific information such as data type, features and attributes analyzed in an annotation task 3.3 structured annotation annotation task that records results by fixed formats and rules 3.4 non-structured annotation annotation task that records results by unfixed formats and rules 3.5 semi-structured annotation annotation task that records results by fixed formats and unfixed rules 3.6 manual annotation annotation task performed entirely in manual way 3.7 automatic annotation annotation task performed entirely by machines, and reviewed manually after the annotation is completed 3.8 semi-automatic annotation annotation task performed jointly by human and machine 3.9 semantic annotation annotation task that takes the meaning and relationship represented by data as the annotation object 3.10 annotator personnel who have the ability to complete specific annotation task and meet quality requirements, perform annotation task, and directly contribute to the annotation task Note: Including initial annotator, annotation reviewer, arbitrator, etc. 3.11 initial annotator person who performs the annotation task and gives the preliminary annotation results 3.12 annotation reviewer person responsible for the review and quality control of the preliminary annotation results 3.13 arbitrator person responsible for giving the final result when many annotators have inconsistent annotation results for the same data Note: Under normal circumstances, the qualification requirements of arbitrators > annotation reviewers ≥ initial annotators. 3.14 annotator performance representational ability of the annotator to perform the annotation task 3.15 annotation responsible organization entity that organizes and carries out the annotation task and is directly responsible for the annotation quality 4 Documents describing the annotation task 4.1 Annotation task classification Before the annotation task starts, the annotation responsible organization shall clearly define the classification of the annotation task, including data mode, execution subject, annotation result format, annotation result property, annotation result form and other dimensions. The data modes of the annotation task may be divided into image, signal, video, text, etc. According to the execution subject, annotation tasks may be divided into manual annotation, automatic annotation, semi-automatic annotation, etc. According to the annotation result format, annotation tasks may be divided into structured annotation, non-structured annotation, semi-structured annotation, etc. The annotation result property may be divided into GT value, reference standard, gold standard and other types. The annotation result form may be divided into detection, classification, segmentation, semantics and other types. Note: Semantic annotation is often used to describe relationships or connections between objects, such as the relative positions of muscle and fat on ultrasound images. 4.2 Annotation task description 4.2.1 Annotation rules

Contents of YY/T 1833.3-2022

Foreword i Introduction ii 1 Scope 2 Normative references 3 Terms and definitions 4 Documents describing the annotation task 5 Data annotation quality features 6 Annotation and quality control process 7 Annotation tools 8 Evaluation method Annex A (Informative) Examples of annotation task description Annex B (Informative) Examples of business architecture (pulmonary nodules in thoracic CT ) Annex C (Informative) Evaluation of AI-assisted annotation performance Bibliography