Independent. Not affiliated with TypeSafe AI, the maker of Jev. Disclosures

Decision LayerUnofficial

Try “routing”, “confidence”, “leads” or “noul”.

    Tests

    We run it before we recommend it

    TypeSafe itself tells users to “run your own private evals, treat public ones with a grain of salt” (TypeSafe blog). That is what this page is for.

    Published tests

    None yet

    Our first runs are being set up. Until a test is published here with its data and code, nothing on this site says “we tested it”.

    First in the queue

    1. 01Support ticket routingJev vs a small LLM with structured output, on a public ticket dataset.
    2. 02Insurance lead scoringSynthetic agency leads with known labels, so no real person’s data is used.
    3. 03Does confidence match accuracy?A calibration check: when Jev says 0.9, is it right about 90% of the time?

    Every test report includes

    • DataWhat it is, where it came from, how many examples, how it was labeled.
    • BaselineWhat we compared against: an LLM, a classifier or a simple rule.
    • AccuracyOverall, and on the hard cases, with the mistakes shown.
    • CalibrationHow often answers were right at each confidence level.
    • Speed and costMedian and slow-end response times; cost per 1,000 decisions at list price.
    • Version and dateThe exact Jev version and gateway, and when we ran it.
    • CodeThe script, so anyone can rerun it.
    What we won’t do. Repeat a speed or cost multiple without saying whose test it was, publish a test without its losing cases, or use real customer or policyholder data.