When an AI model risk scores an email for potential compliance violations, regulators expect extensive validation including test datasets, performance metrics, bias testing, ongoing monitoring and detailed documentation. The model’s decision-making process must be transparent, measurable and continuously validated.
When a human analyst risk scores that same email, making the same compliance assessment with the same potential impact, what validation is required? Much less.
It’s worth considering why the same compliance outcome faces completely different validation standards depending, on whether AI or a human makes the decision.
The Same Decision, Different Standards
Consider email surveillance in a typical financial institution. An AI model analyses an email and assigns a risk score based on content, context, and learned patterns.
Regulatory expectations for this AI model include:
- Performance testing against representative datasets
- Accuracy metrics measuring correct vs incorrect risk assessments
- Bias testing to ensure consistent treatment across different scenarios
- Ongoing monitoring for model drift and performance degradation
- Documentation of training data
- Model architecture
- Validation results
- etc, etc, etc.
Now consider a human analyst who reviews that same email and assigns a risk score. The analyst makes a subjective judgment about compliance risk based on experience, training, and intuition. What validation is required for this human decision-making process? Where is the etc, etc, etc?
Typically, the analyst received initial training on compliance policies and some ongoing education, or past work experience. Beyond that, validation of decision quality takes different forms.
Some institutions conduct random sampling of closed alerts to review analyst decisions, but this approach is quite different from the systematic validation required of AI models. The human’s risk scoring ability is often assumed rather than systematically validated.
Both the AI model and the human analyst are performing the same fundamental task of assessing compliance risk, yet only one faces rigorous validation requirements.
The Reliability Assumption
The double standard rests on an assumption that human judgment is inherently more reliable than AI decision-making, or at least more acceptable when it fails, but what evidence supports this assumption?
While human analysts bring valuable contextual understanding and can adapt to novel situations that AI models might miss, they also bring factors like inconsistency, fatigue, cognitive bias and subjective interpretation that can affect decision quality.
If AI models and human analysts are making assessments with similar consequences, should both be subject to the same validation requirements?
Consistency : The Complexity of Human vs AI Performance
AI models face extensive testing for consistency, though many modern AI systems are non-deterministic by design. LLMs can produce different outputs for identical inputs, making traditional consistency testing problematic. Model validation attempts to test whether outputs remain stable, but this becomes complex when the system is designed to vary its responses.
Many current implementations use LLMs for risk assessment and summarisation – for example, determining a risk score as a recommendation – while humans make the actual decisions. It is notable that there has been recent development in autonomous AI decision-making with agentic AI, but this is directly impacted by the quality of the GenAI output.
Human analysts also make assessments that vary significantly, but for different reasons. At small scale, perhaps 30 traders, analysts can build useful mental profiles and contextual understanding that actually improves decision quality, but beyond 50 monitored users, human memory becomes unreliable.
Human decisions can also vary significantly based on factors having nothing to do with actual risk. Unlike AI models where performance variables are technical and measurable, the human performance factors are more complex and numerous…is below a glimpse into the future of analyst performance monitoring?
Consider an analyst presented with 200 alerts to review. If the 6th alert contained a message identical to the 89th alert and the 192nd alert, the 192nd alert would likely be reviewed more quickly, either because it’s getting to the end of the alert queue, or an assumption that they’ve “seen this before”, which introduces bias.
While this variability is acknowledged in the industry, systematic measurement is less common, and whereas AI non-determinism creates variation, human decision fatigue can create dramatically different levels of care and attention to identical cases.
The Experience Illusion
Experience in surveillance work is valuable, but it raises questions about how we measure the quality of that experience. When analysts have “5 years of ecomms surveillance experience,” this represents significant expertise. However, if those years involved reviewing predominantly false positives, what does this mean for validation approaches?
The challenge is that different types of experience may require different validation methods, as an analyst who has reviewed many true violations may develop different skills than one who has primarily seen false alerts. Without systematic measurement, it becomes difficult to assess what specific expertise different analysts have developed.
This connects directly to model performance, since analyst decisions on alert reviews typically drive model improvements and updates. Any variance in analyst performance or consistency affects the quality of feedback used to enhance the AI models. This creates a cycle where the quality of human decision impacts AI model training.
Continuous Monitoring : Essential for AI, Absent for Humans
The industry has an emphasis on perfecting alert generation through continuous monitoring of AI model performance.
AI gets tracked continuously through accuracy rates, false positive percentages and alert quality metrics. When performance degrades, it triggers investigation and potential model updates. This ongoing monitoring is considered essential for responsible AI deployment because models can drift over time.
Human analyst performance requires the same continuous monitoring principle. Analyst decision-making can drift due to alert fatigue, affecting attention to detail over time, false positive exposure gradually skewing judgment and workload impacts creating inconsistent review quality.
Systematic tracking of analyst decision-quality is less common across the industry, with productivity metrics like alert/case closure rates being standard. How often are analysts’ risk assessments validated against eventual outcomes? When analysts miss significant risks or incorrectly dismiss genuine violations, systematic tracking of these patterns will help.
WAIT…no, is this even possible? How can you validate if an analyst has missed a significant risk, or has failed to escalate a legitimate concern, short of double checking each alert?
The solution for validation is more about preventing the problem rather than trying to detect misses after they happen.
The industry generally assumes that initial training and occasional updates will maintain analyst performance. But if AI models require continuous performance monitoring and training to ensure reliability in use, the same principle could apply to human analysts making similar assessments or decisions.
Bias : Tested vs Ignored
AI model validation includes bias testing to ensure consistent treatment across different scenarios. Models are tested for disparate impact, unfair advantage, and systematic errors that could affect particular groups or situations unfairly.
While human decision-making bias is acknowledged, it’s rarely tested systematically, though practical biases can significantly affect the quality of risk assessment:
- An analyst might judge all subsequent alerts as more suspicious after reviewing a clear violation earlier that day (anchoring bias).
- They might focus on email phrases that support their initial impression while overlooking details that suggest the communication is actually routine (confirmation bias).
- A recent memorable case involving a particular type of misconduct might make an analyst overestimate how common that violation type is (availability heuristic).
These judgments can be skewed by recent cases or memorable incidents in ways that have nothing to do with actual risk.
While AI bias testing has become standard practice, systematic human bias testing in compliance decision-making is less common. The industry generally relies on awareness training to manage human bias, though research shows that bias awareness alone doesn’t eliminate biased decision-making.
The next section discusses continuous assessment approaches. Before we explore those, take a moment to assess these communications using your normal surveillance process.
Rate each by risk level on a scale of 1-10:
- Sarah : “Don’t mention this to compliance, keep between us”
- Dave : “Yes mate. If we do this proper, there’s 300k in it”
- Sally : “Better to discuss offline, it’s no longer a rumour”
- Karl : “Let drip this in slowly, won’t get noticed”
- William : “Certainly. If we do this appropriately, there is 10k in it”
The Benefits of Continuous Human Assessment
Continuous assessment of human analysts could provide the same performance assurance that we require from AI models.
Detecting Performance Drift: Measuring whether decision quality degrades over time due to fatigue, overconfidence, or false positive conditioning.
Validating Experience Claims: Determining whether years of surveillance experience actually improved decision-making ability or developed different skills than expected.
Identifying Bias Patterns: Systematic analysis of decision patterns to detect experience-based bias, fatigue effects, or workload impacts.
Measuring Training Effectiveness: Tracking whether training translates to better real-world decisions, not just completed courses.
Ensuring Consistency: Testing whether analysts with different backgrounds reach comparable conclusions on similar cases.
Calibrating Expertise: Validating that “expert” analysts actually perform better than newer staff, rather than just being more confident.
Let’s go back to the communications in the section above – how did you score each?
If you scored Dave and William differently, you’ve just demonstrated how subtle bias can influence risk assessment.
Subconsciously, does ‘Dave’ seem more dodgy than ‘William’?
Their communications were essentially identical except for the amounts mentioned (300k vs 10k) and language style, yet many readers will have given them different risk scores.
This highlights why systematic bias testing for human analysts is so challenging yet necessary.
The Uncomfortable Questions
This validation double standard raises fundamental questions about how institutions approach decision-making oversight.
- Why are human decisions assumed to be more reliable? If both AI and human analysts can make errors with compliance consequences, shouldn’t both face similar validation requirements?
- What constitutes adequate human decision validation? If AI models need test datasets and performance metrics, what would equivalent human validation look like?
- Should decision validation scale with impact? High-stakes compliance decisions might warrant more rigorous validation regardless of whether AI or humans make them.
- How should institutions measure decision quality? If analyst accuracy isn’t measured systematically, how do institutions know whether their compliance decisions are reliable?
- What about hybrid decisions? When humans review AI recommendations and make final decisions, who’s responsible for validation? How should combined human-AI decision-making be validated?
Toward Consistent Standards
If the goal is reliable compliance decisions – an outcomes-based approach – it shouldn’t matter whether AI or humans make them, both should be validated equally.
Some institutions are beginning to address this gap through approaches like decision auditing, where they systematically review analyst decisions against eventual outcomes:
- Conduct inter-rater reliability testing, having multiple analysts review the same cases to measure consistency.
- Run calibration exercises, regularly testing analyst decision-making against known scenarios.
- Validate competency continuously, conducting ongoing training assessment to ensure experience is improving rather than degrading performance.
These approaches don’t eliminate human judgment but do provide validation that human decision-making meets quality standards equivalent to what we require from AI models.
Conclusion : Bridge The Validation Gap
The validation standards for AI surveillance models and human decision-makers follow different approaches. Extensive validation standards exist for AI models that transcribe calls, detect languages, generate alerts, score and summarise content, while validation approaches for compliance analysts making compliance decisions tend to be less systematic.
In the same way that transcription models require continuous accuracy monitoring and language detection models need performance validation, humans making compliance decisions could benefit from equivalent attention. AI model performance can drift over time, and human decision-making quality can also vary, yet validation approaches differ significantly between the two.
For institutions evaluating their approach to validation, the questions become:
- How do you know your analysts are making good decisions?
- What standards exist to measure human decision quality?
- Is there equal emphasis on AI effectiveness and human effectiveness?
“If our strategy isn’t right, if we don’t use the right tools, if we just spray water around in a disordered fashion, we might dampen the surroundings, but the fire will continue to burn.“
— Sarah Pritchard, UK FCA
What do you think – should validation standards be consistent regardless of whether decisions are made by AI models or human analysts?
If you’re looking to enhance your surveillance training and assessment processes, we’d welcome the opportunity to discuss how we can help.
