AESIA Guidelines on the AI Act – Guide No. 9: Accuracy

The Spanish Agency for the Supervision of Artificial Intelligence (AESIA) has published a set of guidelines developed under the Spanish AI regulatory sandbox. Guide No. 9 is dedicated to the accuracy requirements established by Article 15 of the AI Act. It follows Guide No. 8 on Transparency and focuses on one of the three technical requirements addressed by Article 15: the accuracy of high-risk AI systems.
The Guide is non-binding. It provides practical measures to help providers ensure that AI systems achieve an appropriate level of accuracy and maintain that level throughout their lifecycle. AESIA does not treat accuracy as a single performance test carried out before deployment. It treats it as a requirement that must be defined, measured, monitored and maintained for as long as the system operates.
What Article 15 requires
Article 15 requires high-risk AI systems to be designed and developed so that they achieve an appropriate level of accuracy, robustness and cybersecurity, and perform consistently in these respects throughout their lifecycle. The Guide focuses on accuracy and explains how providers can determine what an appropriate level means for a particular system.
That level cannot be set in the abstract. It must be connected to the intended purpose of the system, the risks identified during risk management and the consequences of inaccurate outputs.
Defining accuracy for the specific system
The first step is to define what accuracy means for the specific AI system. Accuracy is not a generic percentage that can be applied to every model. The appropriate level depends on:
- the task the system performs;
- the type of output it produces;
- the context in which it is used;
- the potential consequences of errors.
A system that classifies low-risk content may tolerate a different error rate from a system that supports decisions in healthcare, employment or access to essential services. The provider must therefore set accuracy criteria that fit the intended purpose and the risks associated with the system.
The Guide connects this assessment directly with the risk management system under Article 9, which we examined in AESIA Guidelines on the AI Act – Guide No. 5: Risk Management. The risks identified during risk management should inform the accuracy requirements. Where an incorrect prediction could cause serious harm to individuals or affect their fundamental rights, the provider may need stricter accuracy criteria and stronger controls. Accuracy is therefore not only a technical performance characteristic. It is part of the wider risk management of the AI system.
Choosing the right metrics
Once the appropriate level of accuracy has been defined, the provider must select metrics that measure it effectively. The choice of metrics must match the characteristics and intended purpose of the system. Different types of AI systems may need different measures, and the provider should be able to justify why the metrics chosen represent the system’s performance adequately. The goal is not the most sophisticated metric available. It is the metric, or combination of metrics, that shows whether the system performs as required.
The choice must also take account of the type of errors the system produces. A simple overall accuracy rate may hide important differences between false positives and false negatives, or between groups of users or categories of inputs. Depending on the use case, measures such as precision, recall, sensitivity, specificity or error rates may be more appropriate. The metric should reflect the task and the risks associated with inaccurate outputs.
Accuracy by design
Accuracy requirements must be set during the design phase. Providers should decide in advance which accuracy criteria the system must meet and how they will be measured. This includes defining the metrics and, where machine-learning techniques are used, choosing an appropriate target or objective function for training the model. The objective function should be aligned with the intended purpose and the identified risks, not chosen only for technical convenience.
The Guide therefore treats accuracy as something to be engineered into the system, not checked only once development is complete. The design should include mechanisms that support the required level of accuracy and provide a basis for testing whether the system meets the provider’s criteria. Evidence produced during development should be kept, so that the organisation can show how the accuracy requirements were defined and validated.
Testing and validation
Testing and validation are central to the accuracy framework. The provider should evaluate the system with appropriate datasets and testing procedures and check whether the results meet the predefined criteria. The evaluation should be sufficiently representative of the conditions in which the system will operate.
This links directly to the data governance requirements of Article 10, examined in Guide No. 7: Data and Data Governance. The quality and representativeness of validation data can directly affect how reliable the accuracy assessment is.
Accuracy after deployment
Accuracy can change after deployment. A system may perform differently in real-world conditions, when it meets new types of input or when it is integrated into a different technical or organisational environment. Updates, modifications or changes in the data used by the system can also affect performance. The provider must therefore monitor accuracy throughout the lifecycle and identify when performance falls below the required level.
Monitoring should not be limited to detecting technical failures. The organisation should set indicators and thresholds that reveal a decline in accuracy. When performance falls outside the established parameters, corrective measures should follow. Depending on the circumstances, these may include:
- reviewing the underlying data;
- adjusting or retraining the model;
- changing the relevant parameters or the operating conditions;
- temporarily restricting the use of the system.
AESIA addresses post-market monitoring in more detail in its Guide No. 13.
Consistency over time
The Guide stresses consistency over time. Article 15 does not ask for an appropriate level of accuracy only at the first assessment. The system must perform consistently throughout its lifecycle. Providers should be able to show that the mechanisms set up during design and validation keep working after deployment, and that changes in performance are identified and addressed.
This matters especially for AI systems that continue to learn after being placed on the market or put into service. Continuous learning can change the behaviour and performance of the system and introduce new sources of error. The provider must ensure that learning does not undermine the accuracy requirements, and that appropriate controls apply when the system changes as a result of new data.
Feedback loops
The Guide also highlights the link between accuracy and feedback loops. When a system continues to learn, its outputs may influence the data used in future operations. If inaccurate or biased outputs become future input data, the feedback loop can progressively reinforce errors. Article 15 requires appropriate measures to reduce this risk. Accuracy monitoring must therefore consider not only the current performance of the model, but also how the system’s own outputs may affect its future performance.
Accuracy and data
Accuracy is closely connected with data quality and representativeness. A system may meet an accuracy threshold during testing and still perform worse on data that differ significantly from those used in development. Providers should consider whether the testing environment reflects the operating environment and whether the relevant populations, circumstances and input characteristics are represented. Article 10 and Article 15 cannot be treated as separate technical exercises.
Acceptable error
Providers must also distinguish between accuracy requirements and acceptable error. Every AI system operates with some uncertainty or error, and the AI Act does not require perfect accuracy in every circumstance. The question is whether the level of accuracy is appropriate for the intended purpose and whether the remaining risks are adequately addressed. Explicit thresholds allow the organisation to decide when performance is still acceptable and when corrective action is needed.
Documentation and quality management
Documentation is an essential part of the process. Providers should document:
- the accuracy criteria set for the system;
- the reasons for choosing particular metrics;
- the target or objective function;
- the results of testing and validation;
- the monitoring mechanisms used after deployment;
- the corrective measures taken when accuracy declines.
The documentation should be enough to reconstruct how the organisation concluded that the system met the accuracy requirements. AESIA’s Guide No. 15 sets out what the technical documentation should contain.
The Guide also connects accuracy with the quality management system under Article 17. Accuracy criteria should be built into the organisation’s quality controls, verified during development and monitored after deployment. The evidence from accuracy assessments must also be consistent with the technical documentation and with the information given to deployers under Article 13, discussed in Guide No. 8: Transparency. This ensures that the declared performance of the system matches the performance actually observed. Accuracy also supports effective human oversight, because the people overseeing a system need to know how reliable its outputs are. On this topic, see Guide No. 6: Human Oversight.
A practical sequence
In practice, accuracy should be managed as a continuous governance process. For providers working through Article 15, the sequence is:
- Define the accuracy requirements according to the intended purpose and the identified risks.
- Set appropriate performance thresholds.
- Select suitable accuracy metrics.
- Align the training or objective function with the intended purpose.
- Test and validate the system with appropriate data.
- Set up mechanisms to monitor accuracy throughout the lifecycle.
- Detect declines or significant changes in performance.
- Address errors and feedback loops where relevant.
- Take corrective measures when accuracy falls below the established level.
- Document the criteria, measurements, results and actions taken.
The central question under Article 15 is not only whether an AI system is accurate when it is tested. It is whether the provider can show that the system was designed around appropriate accuracy requirements, measured with meaningful metrics, validated against its intended purpose and continuously monitored so that its performance remains consistent throughout its lifecycle.
Accuracy under the AI Act is therefore a dynamic requirement. It connects technical design, risk management, data governance, quality management and post-deployment monitoring. Providers must treat performance as an ongoing compliance responsibility, not a one-time technical specification.
Are you ready to transform the Data in value for your business?

