Validation is a chain, not a certificate
A digital product can be technically stable yet clinically misleading, clinically accurate yet unusable, or effective in a trial yet unsafe after an update. The IMDRF separates clinical evaluation into a valid clinical association, analytical or technical validation, and clinical validation. These questions ask whether an output has a sound relationship to the condition, whether software processes inputs correctly, and whether it achieves its purpose in the intended population and setting. [1]
Evidence should match the claim. A diary that stores symptoms needs usability, reliability, privacy and accessibility evidence. A model that recommends urgent assessment needs diagnostic performance, workflow and safety evidence. A therapeutic intervention needs comparative evidence on symptoms, function or another patient-important outcome. A broad phrase such as clinically validated is not meaningful without the intended use, comparator and endpoint.
- Write the intended-use statement before choosing the study design.
- Define the target population, user, setting, input and action.
- List foreseeable wrong outputs and their consequences.
Risk determines the evidence threshold
The NICE Evidence Standards Framework classifies digital technologies and sets proportionate standards. Its 21 standards cover design factors, describing value, demonstrating performance, delivering value and deployment. Higher-risk tools require stronger clinical-effectiveness evidence, while every tier must address relevant safety and quality requirements. [2]
Risk includes more than regulatory class. A cycle tracker may appear low risk but can create harm if it presents an estimated fertile window as contraception-grade certainty. A mental health chatbot may create risk if it fails to recognize crisis language. A remote blood pressure pathway can fail through a bad cuff, missing data, delayed escalation or unclear responsibility. Evaluation should study the complete sociotechnical pathway.
- Assess severity, reversibility and detectability of harm.
- Include non-use, dropout and digital exclusion as outcomes.
- Use the current standard of care as the comparator.
Prospective studies reveal workflow effects
Retrospective testing is useful for early model development but cannot show how clinicians and patients respond to an output. DECIDE-AI was developed for early-stage live clinical evaluation of AI decision support and includes 17 AI-specific reporting items with 28 subitems, plus generic items. It emphasizes actual clinical performance, human factors and safety at small scale before larger trials. [4]
When a product is ready for a randomized trial, CONSORT-AI adds 14 reporting items to standard trial reporting. Investigators should describe the intervention, required user skills, setting, input and output handling, human-AI interaction and error cases. [3] These details make replication possible and expose whether apparent benefit came from the algorithm, added staffing or a redesigned workflow.
- Prespecify intervention fidelity and contamination.
- Measure alert response, override, delay and workload.
- Report technical failures and missing outputs by group.
- Register trials and publish the protocol and analysis plan.
Women's health requires life-stage and equity testing
Validation populations should reflect the intended users and the biological or social variables that can affect performance. Depending on the use case, this may include age, cycle stage, pregnancy or postpartum timing, menopause, medication, comorbidity, language, skin tone for optical sensors, disability and access to a compatible device. Subgroup selection should be clinically justified and prespecified.
Overall accuracy can conceal poor sensitivity in a smaller high-risk group. Report confusion matrices, discrimination, calibration and confidence intervals, then test whether the recommended action improves outcomes. FDA transparency principles advise describing training and testing data, known biases, failure modes, gaps in population representation and circumstances where inputs differ from development data. [5]
- Do not treat race as a biological shortcut without examining structural explanations.
- Test translations and literacy demands, not just the English interface.
- Assess performance with real consumer devices and connectivity conditions.
Evidence continues after launch
Deployment should begin with acceptance testing, staff training, a fallback process and clear accountability. Monitoring should cover effectiveness, safety, equity, usability, privacy, cybersecurity and service impact. WHO's monitoring and evaluation guide for digital health interventions emphasizes defining the intervention, maturity, indicators and evaluation design rather than assuming digital delivery is beneficial. [6]
Software updates can alter risk. Each change needs classification, verification and, when clinically meaningful, renewed validation. Teams should define drift thresholds, review cadence, incident reporting, rollback authority and communication with patients and clinicians. The most trustworthy product statement is version-specific and says what has been shown, what remains uncertain and what is being monitored.
- Maintain a versioned evidence dossier and change log.
- Review subgroup performance and adverse events on a set schedule.
- Investigate near misses as well as confirmed harm.
- Retire functions that no longer meet the evidence threshold.
What the evidence cannot yet answer
- Reporting guidelines improve transparency but do not make a weak study valid.
- A regulatory authorization and a health technology assessment answer different questions.
- Short trials may miss rare harms, behavior change and performance drift.
- Local validation cannot rescue a product whose clinical association is unsound.
Questions worth taking into care
- What exact claim is supported by the available evidence?
- Was the product evaluated prospectively in its intended workflow?
- Are outcomes patient-important and compared with current care?
- Does subgroup reporting include confidence intervals and missingness?
- What monitoring and rollback plan applies to each software version?
Source record
Evidence used in this review
Sources were selected for clinical authority, methodological relevance and traceability. Links open the original guidance, public-health record or research publication.
- [1]Software as a Medical Device
U.S. Food and Drug Administration · 2025
- [2]Evidence standards framework for digital health technologies
National Institute for Health and Care Excellence · 2022
- [3]CONSORT-AI extension
Nature Medicine · 2020
- [4]DECIDE-AI reporting guideline
Nature Medicine · 2022
- [5]Transparency for Machine Learning-Enabled Medical Devices: Guiding Principles
US Food and Drug Administration, Health Canada and MHRA · 2024
- [6]Monitoring and Evaluating Digital Health Interventions
World Health Organization · 2016
This evidence synthesis is for general information. It does not diagnose a condition or replace care from a qualified health professional. Treatment choices depend on individual history, examination, local guidance and informed preference. Emergency or rapidly worsening symptoms need urgent local medical assessment.