Artificial intelligence is moving rapidly into NHS clinical workflows. Ambient voice technology, AI-generated consultation notes, automated triage and diagnostic support may reduce workload and improve access, but they also introduce clinical risks that traditional IT assurance may not fully address.

Recent developments reinforce one central message: approving an AI system for deployment is not the end of clinical-safety assurance. AI must be monitored throughout its working life.

1. UK regulation moves towards lifecycle assurance

On 10 September 2026, the National Commission into the Regulation of AI in Healthcare recommended:
• staged or provisional authorisation
• controlled deployment while real-world safety is demonstrated
• continuous lifecycle monitoring
• greater transparency when AI influences care
• public access to safety information
• stronger MHRA enforcement powers
• clearer accountability across manufacturers, providers and clinicians

These are recommendations rather than binding requirements, but they show the likely direction of UK regulation. Local DCB0160 approval should not be treated as permanent: assurance needs planned review dates and explicit reassessment triggers.

Source: https://www.gov.uk/government/publications/national-commission-into-the-regulation-of-ai-in-healthcare-recommendations-for-a-future-regulatory-framework

2. HSSIB investigation into Ambient Voice Technology

HSSIB has launched an investigation into Ambient Voice Technology. Although its investigation focuses on hospitals, the same technologies are already being used in primary care. GP practices, PCNs and ICBs should preserve hazard logs, incident records, implementation decisions and monitoring evidence. The investigation emphasises human factors, clear accountability, usable reporting routes, shared responsibility and continuous learning.

Source: https://www.hssib.org.uk/news-events-blog/our-response-to-the-national-commission-recommendations-on-safe-adoption-of-ai-in-healthcare/

3. Risks emerging from AI-generated clinical notes

Recent safety reporting describes AI-generated notes that omitted important information, invented medicines or diagnoses, reversed clinical negation, confused speakers and struggled with accents or speech patterns. A change from “demyelination” to “no demyelination” illustrates how a small transcription error can materially alter meaning. Long or repetitive notes can also make important errors harder to spot and may propagate into referrals, prescriptions or patient communications.

Source: https://www.theguardian.com/society/2026/aug/31/doctors-ai-scribes-get-names-of-drugs-and-diagnoses-wrong-nhs-watchdog-warns

Emerging research also suggests that an automated AI reviewer may detect added or altered information better than information that has been omitted. This preliminary evidence challenges the use of a second large language model as the principal quality-control mechanism for AI-generated clinical notes. Clinician review remains essential.

Research: https://arxiv.org/abs/2608.31016

4. Human oversight must be meaningful

Simply placing a “human in the loop” does not guarantee safety. Review can weaken under time pressure, when notes are long, when clinicians become accustomed to accepting AI output, or when the system fails by omission rather than by inserting an obvious error. Users need enough time and information to compare, edit and stop unsafe output before it reaches EMIS, prescribing, referrals or patient communications.

Public support is also conditional on accuracy, meaningful human oversight, accountability, transparency and assurance that AI will not worsen care for particular groups. Deployment controls should include a clear explanation for patients, an alternative route where appropriate, accessible error reporting and equality-focused monitoring.

Source: https://www.health.org.uk/reports-and-analysis/reports/the-publics-views-on-the-regulation-of-ai-in-health-care

5. Practical actions for GP practices, PCNs and ICBs

1) Obtain the supplier’s DCB0129 clinical safety case and current hazard log.
2) Complete a local DCB0160 assessment based on actual workflows, staff, patients and connected systems.
3) Include hazards for omissions, hallucinations, incorrect negation, medicines, speaker confusion, language, accent and accessibility.
4) Use a controlled pilot with success criteria, exit criteria and named authority to pause deployment.
5) Require notification of model, functionality, hosting and workflow changes.
6) Retain clinician review before content enters the clinical record, a referral, prescription or patient communication.
7) Monitor clinically significant omissions separately from general accuracy or hallucination rates.
8) Stratify monitoring by patient group, language, accent, disability and consultation type.
9) Provide rapid routes for patients and staff to report and correct errors, including downstream consequences.
10) Define thresholds for local incident escalation, LFPSE reporting and MHRA Yellow Card reporting.
11) Maintain an AI register recording purpose, owner, supplier, version and review dates.
12) Reassess after significant updates, incidents, changes in intended use or changes to the surrounding workflow.

The key message

AI can support safer and more efficient healthcare, but the speed of adoption must not outpace assurance. The safest approach is to treat AI as a continuously managed clinical risk rather than a product that becomes “safe” once deployed.

RJS Care Ltd supports GP practices, PCNs and healthcare organisations with DCB0160 assurance, hazard identification, implementation controls and post-deployment monitoring. To discuss support, contact Ryan at ryan.smith7@nhs.net.