Communications surveillance technology relies on three main approaches to detect potential abuse – Lexicons, NLP, and Large Language Models (LLMs). Each comes with its own strengths and limitations, and understanding what each can realistically handle will help you design controls that match your population.
Lexicons / Key Words
Lexicons work by matching exact phrases, so “place a low bid” gets flagged as expected. The trouble is that “place a cheap bid” or “place a small bid” carry the same intent but use different words, and unless you’ve manually added every variation, those won’t match.
Synonyms multiply quickly, and maintaining a comprehensive lexicon for even a single language demands constant effort that simply isn’t realistic for most teams. It’s a starting point.
NLP
NLP addresses the synonym problem by understanding intent rather than relying on exact matches, which means “low bid,” “cheap bid,” and “small bid” can all be recognised as carrying the same meaning.
This represents a significant step forward from lexicon-based detection, though it raises an obvious question – are there models for each language?
NLP models need training data, and creating effective training data for surveillance requires people who understand both the language and market abuse patterns. That skill set is rare, so while robust out-of-the-box NLP models exist for English, coverage for other languages is often thin or non-existent out-of-the-box.
Cross-language models that connect intent across language boundaries don’t exist out-of-the-box at all.
LLMs
LLMs can work across multiple languages in a way that NLP cannot as they function in languages where NLP models are thin or non-existent, and they can recognise that a phrase mixing Japanese and English carries the same meaning as the English-only version.
Out of the box, LLMs have their own challenges as they weren’t trained on market abuse, so they need carefully designed prompts that inject surveillance expertise. This requires vendors to have multi-language surveillance experience to configure these prompts effectively.
Running LLM inference across all communications is also both expensive and ESG-unfriendly given the energy consumption of running inference at scale, making it impractical as a blanket solution.
The simple solution is not “run everything” through an LLM.
The Multi-Language Challenge
Consider a straightforward coordination phrase “Can you place a low bid, then I’ll take the other side.” In a multi-lingual environment, words like “bid,” “place,” or “I’ll take” might be substituted with equivalents in another language. But how realistic is language switching in written and voice channels?
For written communications like email and chat, language switching is a conscious act. You toggle your keyboard, type a word/sentence/characters or two, toggle back. It happens, but it’s not automatic unlike speech, where people switch languages mid-sentence without thinking about it.
Before treating written language switching as a significant problem, you first need to identify what languages are being used in your communications – whether that’s a single language or multiple languages within a message – and then understand how much switching is actually occurring. The catch is that you can’t get those accurate statistics until you’ve set up language detection.
How your comms solution handles and outputs language detection is (almost) a prerequisite – the stats drives which detection model(s) are appropriate for your population.
Voice calls are different because language switching in speech happens naturally and unconsciously, making it the channel where multi-lingual detection genuinely matters. Voice also has an upstream dependency on accurate transcription before any detection technology can even attempt to analyse content, and this applies to single-language transcription as much as it does to language switching.
If your transcription is producing errors in straightforward Japanese or Cantonese, you have a foundational problem before you even consider the complexity of mixed-language speech. Not a topic for today as voice transcription and language detection for spoken communications deserve their own discussion, which we’ll cover in a follow-up post.
Prove Your Assumptions
Most firms operate on assumptions about their language profile that have never been validated:
- Your business language is English, or Mandarin, or Japanese – but is it?
- External conversations probably have more language diversity than internal ones – but do they?
- Language switching is more common on calls than in written communications – but is it actually?
Language detection gives you the data to prove or disprove these assumptions. Without it, you’re designing surveillance controls based on what you think is happening rather than what’s actually happening. With it, you can answer these questions with evidence and build controls that match your real risk profile rather than an imagined one.
You may have selected a vendor based on demo data and presentations showing a united nations of flags, but what sits behind those flags?
- Are they claiming to have lexicons for those languages, or AI models?
- Do they have the language skill set in-house to have trained, tested, and validated those capabilities?
- Can they provide evidence that would satisfy your model validation and audit teams?
If the answer is a slide deck and reassurances rather than documentation and test results, you’re buying a futures contract rather than spot delivery.
Before You Panic
If you’re monitoring 1,000 staff, all of this might sound alarming, but it helps to ask the question differently – how many of those 1,000 actually communicate in multiple languages, and how much of their communication is in each language?
If the majority communicate in English, and those who use other languages do so for only a small portion of their traffic, then the multi-lingual gap affects a subset of your population rather than everyone.
The issue isn’t that surveillance technology can’t do the job – it’s considering having a mix of approaches to meet the population.
The Foundation : Language Detection
To segment your population effectively, you need to know who actually communicates in multiple languages, and HR data won’t tell you this.
A “Singapore employee” could be communicating in English, Mandarin, Tamil, Malay, or all four, and you’ll have no HR data at all on external counterparties.
The only way to build an accurate multi-lingual population profile is from the communications themselves. Language detection reveals observed behaviour rather than assumed attributes:
- this trader communicates in Japanese and English;
- this counterparty responds in Cantonese;
- this desk has 40% non-English traffic.
If your language detection is weak, you can’t build the profile, you can’t segment, and you can’t right-size your controls.
One Approach – Vendor Feedback Welcome!
Language detection becomes your segmentation layer:
- routing single-language traffic to lexicon and NLP approaches that are cheap(ish), fast, and well understood, while;
- directing multi-lingual traffic to the LLM.
Prompt management becomes your governance framework, allowing you to document exactly what the LLM was instructed to look for, how it was framed, and to maintain version history.
The prompt is where your surveillance expertise lives, and that’s what you govern. Meanwhile, your “Know Your Trader” profile gets built from observed language patterns rather than HR assumptions, giving you an accurate picture of who your multi-lingual population actually is.
One important consideration: the language detection components and LLMs that underpin this approach are rarely controlled by surveillance vendors themselves. They’re typically third-party services or open-source models that the vendor has integrated. This makes it critical to understand:
- How much testing and validation your vendor has actually performed on these components for your specific languages.
- What language skills exist within the vendor organisation to support ongoing quality assurance.
The technology stack may look impressive on a diagram, but the vendor’s ability to stand behind it depends on work they’ve done, not work someone else has done.
Conclusion : CONsCIOUS ACCEPTANCE
Lexicons work for known phrases in a single language, NLP handles synonyms and intent within a single language, and LLMs can bridge languages but require genuine investment in prompts, expertise, and selective deployment.
Language detection gives you the data. With that data you can decide what to deploy, what gaps remain, and consciously accept (or not) those gaps. Without it, you don’t know what you’re exposed to.
If this has triggered questions about your own language coverage and you’d like some independent advice or just chat through the considerations, please get in touch.
