Jev vs OpenAI Decisions API (GPT-6 Luna) for NL-to-SQL

OpenAI opened the Decisions API to everyone. It does the same job Jev does in our NL-to-SQL pipeline: answer yes/no and pick-one questions quickly. We put it in Jev's place and ran our benchmark on ClickHouse. Both got the right numbers. Jev answered faster.

---

Short answer: On 7 October 2026 we ran the same order questions twice. Jev made the judgments in one run. The OpenAI Decisions API (gpt-6-luna) made them in the other. GPT-6 wrote the SQL both times. Both runs returned the right numbers. Intent averaged 257ms on Jev and 369ms on Decisions. Schema linking averaged 263ms vs 294ms. Jev stays the default.

Key takeaways

• Both judges were confident on the same steps, the same number of times. Neither fell back to the LLM.
• Jev was faster on the two steps it runs on every question: intent and schema linking.
• The OpenAI run finished the whole benchmark sooner, 256s vs 296s. That gap came from the GPT-6 writer and the LLM reflection pass, not from the judgments.
• The first Decisions call was slow, 875ms for intent. Drop it and intent is still 313ms vs 258ms.
• Decisions is in public beta and only serves gpt-6-luna. We will repeat this when it reaches GA.

The paired run

Same questions, same orders, same tenant, same GPT-6 writer. The only change was who answered the judgment questions.

Read the table top to bottom. The first two rows are the judgments, and Jev wins both. The next two are GPT-6 writing and checking SQL, which neither judge touches. Reflection came in almost a second quicker on the OpenAI run, and SQL writing 0.2s quicker. That is what makes the OpenAI column look faster overall.

What the Decisions API is

The Decisions API takes text or images and returns typed answers: the probability that a condition is true, one choice from a fixed set, or a score against a rubric. It has its own endpoint, POST /v1/decisions. OpenAI says it is about 10x faster than the Responses API. When we checked on 7 October 2026 it was in public beta, and gpt-6-luna was the only model.

That is the same shape as Jev, TypeSafe's System One model. Jev's yes/no questions became Decisions predicates. Jev's pick-one questions stayed choices. Our thresholds did not change. A refusal, a timeout, or a low-confidence answer falls through to the LLM, the same as with Jev.

Where each judge was faster

The benchmark timings are means over ten questions. We also called both judges directly on the same inputs.

Jev was steadier on the short guardrail question. Decisions was faster on the larger prune, with twelve candidates in one request. These are a handful of samples, not a distribution. On the full benchmark, schema linking still came out ahead on Jev. That step does more than the prune call alone.

The judgments matched

Each judge answered the guardrail 11 times, intent 12 times, schema pruning 12 times, and the reflection check 12 times, with enough confidence that the LLM fallback never ran. The counts were identical on both runs.

The reflection check asks one thing: is this SQL clean enough to skip the full LLM review? Both judges said no on all twelve. So both runs paid for the full review. The Jev and GPT-6 post caught that skip near 300ms. This benchmark did not, on either judge.

The SQL

GPT-6 wrote every query on both runs. Both runs got the same figures, kept the tenant filter, and used half-open date ranges. Five of the ten queries were worded differently: an added ORDER BY, count(*) instead of count(id), a larger row limit. That is the writer's run-to-run variance, not a judge effect. The judges only decide which tables the writer sees and whether a review runs.

Why Jev stays the default

Accuracy was a tie. Jev was faster on the steps it owns. The Decisions API is in beta with one model. We had no reason to move.

Nothing changes for customers. React SDK embeds and calls to qp.ask() or POST /v2/query keep the same requests and responses. The judge is chosen on the server. Tenant isolation is still the prompt, the reflection pass, and the check before execution. A judge can only shorten the path when it is already confident.

The headless boundary stays the same too. The Node SDK runs the SQL in your process. Neither judge receives query results or database credentials.

FAQ

Is the OpenAI Decisions API good enough for NL-to-SQL judgments?

On this benchmark, yes. It answered with confidence as often as Jev, and the final numbers matched. It was slower on the small per-question steps and faster on one larger prune.

Why was the whole benchmark faster with OpenAI if Jev won the judgments?

The LLM review took almost a second less on that run, and SQL writing 0.2s less. GPT-6 did that work on both runs, so the difference is variance between runs, not the judge.

Did either judge see row data?

No. Both got the same evidence: the question, schema chunk summaries, and the draft SQL for the reflection check. Execution stays in the SDK callback against your database.

What happens when a judge is unsure?

The request falls through to the LLM path, the same way it does with Jev. A refusal, a timeout, or an answer below our confidence threshold all count as unsure.

Do embed customers need a new SDK?

No. The React SDK, the Node SDK, and POST /v2/query keep the same shapes. The judge is a server-side setting.

Will we bench Decisions again?

Yes, when OpenAI moves it to GA or adds a second model. We will run the same ClickHouse benchmark with the same tenant.

---

Further reading

• Half the Latency, Better SQL: Jev and GPT-6
• Jev and Gemini 3.8 Flash: Matching SQL, Slower NL-to-SQL
• NL-to-SQL in Production in 2026
• Zero-Trust SDK Architecture
• OpenAI Decisions API guide