How to evaluate an Arabic-native AI agent
Almost every vendor selling into the GCC now claims Arabic support. Very few of them mean the same thing by it. This guide gives you the eight tests we would run if we were on your side of the table — each one designed to fail a translated chatbot inside two minutes.
Published
Why "supports Arabic" is not a specification
A model that was trained overwhelmingly on English and then pointed at an Arabic prompt will produce grammatical Modern Standard Arabic. It will also sound like a government circular. Your customers do not write to you in MSA — they write in Saudi, Emirati, or Khaleeji dialect, half in Latin script, with English product names dropped mid-sentence.
The gap between those two things is where evaluations go wrong. The demo is scripted in clean MSA, everything works, and the pilot then meets a real WhatsApp inbox. So push the demo off its script early and often.
Test 1 — Dialect, not just Arabic
Send the agent three messages written the way your customers actually write: one in Saudi dialect, one in Egyptian, one in Levantine. A dialect-competent agent answers all three without switching the customer into MSA and without asking them to rephrase.
Watch the register of the reply, not only its accuracy. An agent that answers a casual dialect question in formal fus-ha has understood the words and missed the person.
Test 2 — Code-switching inside one session
Start in Arabic, ask a follow-up in English, then return to Arabic. The agent must carry the full context across all three turns without resetting, re-asking, or answering the second question in the first question's language.
This is the single most common failure in production, because most stacks implement language as a per-session setting rather than a per-message property.
Test 3 — Arabic numerals, dates, and money
Ask for an order placed "last Thursday" and see whether the agent resolves the date against the Hijri or Gregorian calendar — and whether it says which. Ask for a price and check that SAR is rendered correctly in an RTL sentence, with the digits in the direction your customers expect.
Bidirectional text is where thin Arabic support visibly breaks: prices reverse, order numbers split, and a customer loses trust before they have finished reading.
Test 4 — Names and transliteration
Give the agent a name written both ways — عبدالرحمن and Abdulrahman — in the same conversation, and check that it treats them as one person. Then ask it to write that name back into a CRM field and see which form it picks.
Identity resolution across scripts is unglamorous and decides whether your CRM ends up with one record or two.
Test 5 — Grounding and refusal
Ask about a product tier, discount, or policy you do not offer. The correct answer is a clear "we do not offer that", optionally with the nearest real alternative. An agent that invents a plausible Arabic answer here will invent one in front of a customer.
Ask the same invented question twice, in Arabic and in English. Some systems ground well in English and hallucinate in Arabic, because the retrieval index was only ever built over English documents.
Test 6 — Escalation with context
Force an escalation and then look at what the human receives. A full bilingual transcript, the resolved intent, and the customer's actual question is a working handoff. A ticket that says "customer needs help" is a queue with extra steps.
Ask specifically whether the summary handed to the agent is written in the human agent's working language, not the customer's.
Test 7 — Voice, if voice is in scope
Text quality tells you nothing about call quality. Have the agent call a mobile and listen for three things: latency before the first word, how it handles being interrupted mid-sentence, and whether the Arabic prosody sounds like a person or a navigation system.
Then ask what happens when the line is noisy, because a real follow-up call happens in a car.
Test 8 — What it does after the conversation
The conversation is the visible half. Ask the vendor to show the write-back: the CRM record, the ERP field, the refund actually issued. Ask to see the audit trail for that action, in a system you already run.
An agent that talks well and writes nothing back has moved your problem from the inbox to the queue, and left the manual work exactly where it was.