A fine-tuned model can read a customer’s angry paragraph and pick the right department in about 40 milliseconds. That’s on a single T4 GPU, with a model you can download for free. It used to mean a pile of rules. Now it feels like magic, and the instinct is to point it at the work and wait for the wonders.
I’ve watched this story develop from the other end. When I work on older Service Cloud orgs — upgrades, migrations — routing lives in rules. I have seen it many times. Then Einstein arrived with a gate and a confidence score. Now there are small open decision models like Laya, which you run yourself.
So I built a test to see what each one actually does. What I found had little to do with the models. It had much more to do with the data feeding them, and that data decides whether your implementation turns out intelligent or just confident.
How I Built the Test
The job is classification: read a support case and pick one of five departments (Billing, Technical, Sales, Operations or Other). All cases are synthetic.
| Approach | What it is | How I ran it |
|---|---|---|
| Keyword rules | The old way | Built from the training cases. A script’s idea of rules, not a business’s own |
| Einstein Case Classification | Salesforce’s own model | Trained in a sandbox, Automate on at 90% |
| Laya | Open-source decision model by NandhaKishorM (Apache 2.0) | Fine-tuned on a GPU |
I also ran TF-IDF, a basic model that counts words, as a sanity check.
The Data
| Training | 800 closed cases, right department known. Narrow: about 132 distinct descriptions, roughly 20 themes with 40 near-copies each |
|---|---|
| Test | 200 new cases, department hidden. 10 themes that don’t appear in the training |
Every approach trained on the same 800 cases and took the same 200-case test.
Worth being precise about scoring, because it’s easy to get wrong. Einstein declined to route some cases. Count those as wrong and you punish the one system that knew when to stop. I show both: the score over all 200, and for Einstein, how many it routed.
What Came Back
| Approach | Score on the test |
|---|---|
| Keyword rules | 41.0% |
| Einstein | 20.0% |
| Laya, fine-tuned (average of 3 runs) | 98.7% |
| TF-IDF | 30.5% |
| TF-IDF, trained on a broader set of 800 cases | 96.0% |
_Einstein: 40 of 200 right. The 53 cases it left unrouted count as not right. The last row is the same TF-IDF model on the same 200 test cases. Only its training data changed._
If I had stopped at the first four rows, I’d have written a post about Laya winning. The last row changed that.
The Number That Stopped Me
Einstein routed 147 of the 200 cases. Every one went to Billing. It got all 40 real Billing cases right — and sent 107 others there too.
I never saw a confidence score. What I know is that Automate was set to act at 90%, and Einstein acted on every one of those 147. It treated its wrong answers exactly as it treated its right ones.
I can’t see inside Einstein, so why it did this is a guess. My guess is the narrow training data: 800 cases, but only 132 distinct descriptions. Which brings me to the last row of the table.
Same Model. 96% or 30.5%.
Look at the two TF-IDF rows. Same cheap model, same 200 test cases. Trained on the narrow 800, it scores 30.5%. Trained on a broader set of 800, it scores 96%.
So the test is not hard for every model. It’s hard for models trained on narrow data. The training set is a component you design, and in this test it decided more than the model did.
Where Laya Fits
As shipped, Laya got about half right (46% to 54%), so it’s not a drop-in. Fine-tuned on a GPU, it averaged 98.7% across three runs (97%, 100%, 99%). Its confidence held up too: at 0.90 and above, it routed about 85% of cases, and about 99.8% of those were right.
I would not read that as Laya being the smartest model. A basic TF-IDF model reached 96% once its training data was broader. And the same fine-tuned Laya models scored 45.5% on a different test from an earlier run of mine, so 98.7% is not a general number. I did not train Laya on broader data, so I can’t say how it would do.
What You Can Tweak
With Laya, I could change what I wanted. I fine-tuned it on my own cases. I ran three seeds to see how much the result moved. I read its confidence on every case and checked how often it was right at each level. The numbers were sitting in a file.
With Einstein, I had my training cases, my fields and a threshold. Salesforce does let you pick input fields, split cases into segments and filter the training examples. It does not let you pick the model or look inside. The platform tests several and chooses for you. So when it sent 147 cases to Billing, I could see what happened. I couldn’t see why.
That’s a trade, not a complaint. Einstein takes the work off you and keeps the control. A local model hands you the control and the work. When the gate fails, that one is yours.
The Architect’s Note
This was one run, in one sandbox, on synthetic data. With 200 test cases, even Einstein’s 20% could sit anywhere from 15% to 26%. Real case history is far more varied than anything I generated. So I’m not telling you which system wins. Here’s the path I’d follow instead.
- Data first. Count the kinds of cases, not the rows. 800 rows with 132 distinct descriptions is a small training set wearing a large one’s clothes. Check the labels too. Salesforce’s admin guidance warns that wrong labels teach the model the same mistakes, and it describes one customer with about 30% incorrect data. If routing has lived in rules for years, ask what your closed-case labels are made of.
- Keep the model swappable. Treat it as a component behind a clean interface. Einstein today, a local model tomorrow. The part you can’t swap is your data.
- Own the gate. Set the threshold from accuracy you tested on topics the model never saw, not from a number that felt safe.
- Start in recommendation mode. The same guidance says to monitor accuracy before turning on auto-triage. After this test, I’d treat that as a rule.
- Close the loop. Every case a human reroutes should become a training case. That’s my own advice and I follow it every time!
💬 Let’s Argue
The models are getting cheaper, faster and easier to swap every year. Your data is the part that’s yours.
If you would like to review, build The code, the data and every per-case result are in the repo. Thanks to NandhaKishorM for building Laya and releasing it openly.
What would your model learn from your last 800 closed cases?