A real local encoder. 3,080 labelled banking messages. The accepted decisions, the ones sent for review, and the mistakes—all from the same frozen test run.
70.5% fewer LLM calls. Accuracy changed from 69.5% to 83.4%, below our preset 90% floor. This configuration did not pass.
The frozen sample contains 770 public BANKING77 messages, ten per category. Direct Qwen classifies every message; the hybrid runs MemCat first and makes a fresh Qwen request on review or failure. Both return an exact label or review. These are classification calls; no answers are generated for a customer.
| Measured outcome | Qwen directly | MemCat + Qwen |
|---|---|---|
| Correct / 770 | 535 / 770 (69.48%) | 642 / 770 (83.38%) |
| Wrong / review / failed | 218 / 17 / 0 | 123 / 5 / 0 |
| Actual LLM calls | 770 | 227 |
| Reported prompt + output tokens | 2,103,931 | 620,541 |
| Sum of measured query times | 299.67 s | 87.72 s |
| Per-message p50 / p95 | 353.57 ms / 794.05 ms | 8.41 ms / 436.41 ms |
MemCat accepted 543 messages locally: 527 were correct and 16 were wrong (97.05% accepted precision). On the remaining 227, the actual Qwen fallback returned 115 correct labels (50.66%), 107 wrong labels and 5 reviews. Most final errors came from the fallback route.
The paired accuracy difference was 13.90 percentage points; the one-sided 95% lower bound was 11.95 points. The interval uses 20,000 predeclared stratified paired bootstrap samples. Review and failure count as incorrect on this in-scope sample. This is an internal tolerance, not customer approval.
| Target frozen before inference | Result |
|---|---|
| All 770 cases recorded on both routes | Met |
| At least 90% correct across all hybrid inputs | Not met |
| Paired 95% lower bound no worse than −2 percentage points | Met |
| At most 1% hybrid operational failures | Met |
| At least 50% fewer actual LLM requests | Met |
| At least 50% fewer returned prompt + output tokens | Met |
| At least 30% less measured query time | Met |
Qwen2.5 7B Q4_K_M ran locally through Ollama 0.35.1 on an Apple M4 Max with an 8,192-token context. It received the 77 labels, descriptions and one training example per category. MemCat used 32 examples plus a description per category. This comparison does not establish the best possible Qwen configuration.
Category descriptions were literal names. The upstream label get_physical_card is attached to PIN-related examples, which can mislead a prompted model. An independent source check verified every imported label against the original CSV files. The frozen prompt and this failed result are preserved; future prompt changes require separate evaluation.
Qwen ran first, then the hybrid, with the target model unloaded and the same three excluded warmups repeated between routes. Prefix caching remained active. Query time includes local classification and fallback when used; setup, warmups and receipt writing are excluded and recorded separately. Fixed route order may affect timing. The sample was previously evaluated by MemCat; it is not a fresh held-out test or customer traffic. Known mixed-intent failures below remain unresolved.
582a276a3995f5c10be8553b632fe60a9b82462a108fdaeb0bde82f5c5dbd858Public BANKING77 data: Casanueva et al. / PolyAI, CC BY 4.0. Dollar savings, total compute cost and customer adoption remain unmeasured. Sources and licenses.
All 3,080 official BANKING77 test rows, across 77 categories. Thresholds and prototypes were frozen before this test. No test rows were removed.
2,092 correct / 2,146 accepted
2,146 accepted / 3,080 total
1.75% of all 3,080 inputs
95% Wilson interval for correctness among accepted decisions: 96.73%–98.07%. Review means the classifier declined to choose; no fallback model ran, and those messages are not counted as correct answers.
Without abstention, its top-choice accuracy is 88.25% and macro-F1 is 0.8821. The higher accepted precision comes from sending harder messages to review.
The same test inputs and labels. Each baseline used only training data; rejection thresholds were selected on calibration data. The local text baselines require no neural encoder.
| Method | Top-choice accuracy | Accepted precision | Coverage | Wrong routes |
|---|---|---|---|---|
| MemCat + MiniLMExample similarity + abstention | 88.25% | 97.48% | 69.68% | 54 |
| Exact training lookup | 0.13% | — | 0.00% | 0 |
| Category word overlap | 36.66% | — | 0.00% | 0 |
| TF-IDF nearest centroid | 81.17% | 96.93% | 46.46% | 44 |
| Always review | — | — | 0.00% | 0 |
“—” means no automatic decisions, so precision is undefined. Exact lookup and category overlap did not meet the calibration target and accepted no test messages under their selected rejection policy. Always choosing the most common training label achieved 1.30% accuracy.
Individual, sequential queries on Apple M4 Max, Node 22.22.3. These measurements include token validation, tokenization, q8 encoding, category matching, and the review decision.
Runtime: ONNX CPU; individual sequential queries. Browser WASM uses a different execution environment; the live demo reports its own loading and inference times. These figures do not measure an LLM workflow’s end-to-end latency or cost savings.
The Node starter includes the encoder adapter, pinned dependencies, and a replay command. We verified a fresh installation outside this repository using the published SDK: it downloaded the assets, ran local inference, and wrote inspectable results without an API key or source edits.
This is an internal integration check on Node 22 and Apple silicon. Its four authored examples are separate from the accuracy benchmark above. Setup time, first inference, asset requests, and failures are logged separately; no LLM cost savings were measured.
Passing an in-scope benchmark does not make every decision safe. We also ran a separate unknown-topic diagnostic and an authored stress suite.
Accepted as a banking category. All official CLINC test rows from eight predeclared tasks, including weather, recipes, and music. The 95% Wilson upper bound is 1.58%.
This does not establish rejection of unfamiliar requests close to banking topics.
Authored cases missed their expected handling. One mixed request was routed automatically; two other cases were deferred instead of receiving the expected category.
These are regression cases, not a representative accuracy sample.
“Why was I charged twice, and how do I close my account?”
Reviewtransaction_charged_twice“I do not want to close my account. Where is my card?”
card_arrivalReview“my caard got stoln”
lost_or_stolen_cardReviewThe first 10 of 54 errors in source order. Related banking categories are sometimes confused. The full report contains every result, expected label, similarity decision, and timing.
“Is there a way to know when my card will arrive?”
card_arrivalcard_delivery_estimate“How long does a card delivery take?”
card_arrivalcard_delivery_estimate“How do I change currencies to euros?”
fiat_currency_supportexchange_via_app“Can I get my card expedited?”
card_delivery_estimatecard_arrival“How do I unblock my card using the app?”
card_not_workingpin_blocked“Can I change from one currency to another?”
exchange_via_appfiat_currency_support“What all currencies can be exchanged?”
exchange_via_appfiat_currency_support“Is it possible to change to another currency?”
exchange_via_appfiat_currency_support“Can I change to another currency?”
exchange_via_appfiat_currency_support“Change currency”
exchange_via_appfiat_currency_supportEnglish, single-intent banking messages with 77 published labels. Public benchmark performance does not establish accuracy on your support queue, language, or category definitions.
7 official test messages match earlier source rows after normalization. All test rows are retained. Excluding those overlaps gives 97.48% accepted precision at 69.61% coverage across 3,073 messages. Public data may also have appeared in encoder pretraining.
MemCat uses MiniLM embeddings and nearest-example cosine similarity. It adds explicit thresholds, serializable indexes, evaluation, tracing, and shadow mode. This is not a newly trained foundation model.
This local run made 0 hosted classification calls. Local compute and review work still have costs. The separate local Qwen comparison above measures real classification requests and latency. Customer savings, paid-provider bills and independent adoption remain unmeasured.
Training, development, and calibration contained 7,963, 965, and 1,071 rows respectively. The selected model and thresholds were frozen before testing. The report records dataset hashes, source-code hashes, outcomes, baselines, and limitations.
8d29ebcbfa71e7000b095c983fe74b5cc4e133cd4c8aaacf19844ead5e115c41BANKING77: Casanueva et al. / PolyAI, CC BY 4.0. CLINC150: Larson et al. / Clinc, CC BY 3.0. MiniLM encoder: Apache 2.0. Full attribution and immutable source revisions are in the linked notice.