AI Assistants Are Changing What Counts as Clinical Judgment

Key Takeaways

• AI-scribe education can no longer stop at adoption, consent, or documentation efficiency. It needs to teach verification, supervision, and role boundaries.
• Oncology examples this week were specialty-specific, but the learning problem is broader: clinicians are using AI to manage information overload and need disciplined habits for checking outputs.
• ACGME’s faculty-development messaging points to a parallel need in GME: assessment quality is being translated into short, observable, scenario-based training.

AI scribes were described this week less as note-takers and more as clinical assistants that can synthesize chart context, literature, guidelines, and possible next steps. The signal is narrow but useful: when tools begin to shape judgment, CME has to teach how clinicians verify, supervise, and explain that judgment.

AI scribes are becoming assistants, not just note-takers

The JAMA discussion starts from a familiar premise: “This technology was developed to overcome the crushing burden of documentation.” But the more important point came later. The conversation described ambient tools moving from clean, documentation-focused apps into integrated EHR assistants that can pull from patient history, medication profiles, guidelines, literature, and institutional protocols to support synthesis and differential diagnosis (JAMA+ AI Conversations).

That changes the CME job. A course on AI scribes that only covers consent language, note review, and documentation efficiency is now incomplete. The harder learning task is helping clinicians decide when an AI output is merely administrative support, when it is functioning like clinical decision support, and what verification steps should happen before the clinician signs, orders, or communicates a plan.

We saw a related pattern in an earlier brief on clinicians preferring curated AI over general LLMs; this week’s difference is that the tool is moving into the encounter and the chart workflow, not just the search box. The JAMA discussion also raised the regulatory gray zone: because scribes are classified as administrative software, they fall outside FDA device review under that exclusion, even though tools that cross into clinical decision support, diagnosis, or treatment recommendation can come under FDA review and raise automation-bias concerns.

The oncology conversation added a ground-level example. Trainees and clinicians described using AI to triage a flood of papers, identify recent practice-changing studies, and help patients ask better AI questions (Treating Together). Those examples are oncology-led, but the provider implication travels: CME teams should teach a repeatable verification workflow—source check, guideline check, patient-context check, and human escalation—rather than treating AI literacy as tool familiarity. AMA Ed Hub’s own post about a structured decision simulator for ambient scribes points in the same direction, though it is provider-owned educational content rather than independent clinician demand (AMA Ed Hub post).

Faculty assessment is getting packaged as observable practice

The second signal was not clinician conversation; it was an ACGME organizational signal. ACGME spotlighted a faculty-development microlearning module on reliability and validity, noting: “This module uses high-quality video vignettes to teach the causes of low inter-rater reliability in workplace-based assessment through experiential learning.” (ACGME post)

That matters because assessment quality is being translated from abstract competency language into short, observable faculty behaviors. A separate ACGME post promoted a six-day course for GME leaders focused on strengthening assessment practices and building effective, learner-centered programs (ACGME course post). This is a policy and infrastructure signal, not proof of broad clinician demand.

For CME providers serving health systems, academic centers, or specialty societies, the lesson is format as much as topic. Faculty-development content that explains validity in a lecture is weaker than content that lets faculty watch a vignette, rate the learner, compare reasoning, and identify where bias or poor observation entered the process. The question for CME teams is whether their faculty-development portfolio teaches assessment as a visible practice, or mainly as vocabulary.

What CME Providers Should Do Now

• Audit AI-related activities for a required verification step before learners accept, sign, message, or act on an AI output.
• Build AI cases around role boundaries: what the tool may suggest, what the clinician must confirm, and when human consultation is required.
• For GME-facing work, convert assessment concepts into short rater exercises using vignettes, comparison ratings, and debriefs on rater cognition.

What CME teams should reconsider

The common thread is not AI versus accreditation. It is the need to make professional judgment inspectable. If a clinician uses an AI assistant, what exactly must be checked before the output becomes care? If a faculty member rates a trainee, what exactly supports that judgment beyond impression?

CME teams that answer those questions in the activity design—not just in the learning objectives—will be better positioned for the next version of both clinical AI education and faculty development.