GitHub - HalachaBench/halacha-bench-results: HalachaBench early results: AI models on real semicha exam questions (provisional)
1 November 2026, updated 2 November 2026 · Hebrew summary: בעברית, בקצרה
2 November: five models added, all scored on the same questions, rubrics and judge: Gemini 4 Argon, Grok 4.7, Grok 4.3, Claude Opus 4.8 and Claude Sonnet 4.6. See Checks on the Grok results.
HalachaBench tests how well AI models answer real questions from the semicha (rabbinic ordination) exams of the Chief Rabbinate of Israel. These early results cover 22 models on the test set: the 60 questions of five papers of the Cheshvan 5787 sitting (Niddah, Chuppah v'Kiddushin, Mitzvot HaTeluyot BaAretz, Mikvaot, Tefillah u'Brachot), a sitting that began on 12 October 2026. Each model answered every question three times, in Hebrew, closed-book.
Score: how much of the rubric an answer achieves, 0–100 (see Scoring), averaged over 60 questions × 3 attempts. Worst of 3: the lowest of the three attempts at each question, averaged.
† Added on 2 November 2026, after the first results were published, each by a recorded amendment to the pre-registration made before its answers were collected. Claude Opus 4.8 and Claude Sonnet 4.6 are earlier generations, added to see how the scores change across model versions.
95% intervals come from a bootstrap over questions (resampled within each paper, a question's three attempts kept together; 10,000 replicates). They cover only the choice of questions, not judge or rubric error, which is likely larger here. Overlapping intervals do not by themselves mean two models are tied, so models were also compared question by question:
¹ Some runs produced no answer text. The model used up the output limit we set while still thinking (DictaLM 9 of 180 and Qwen3.8 27B 9 of 180; the limits were 57,088 and 65,536 tokens, the latter below what Qwen's model card allows), or stopped after thinking without writing an answer (Qwen3.8 27B 10 of 180). Each such run was tried once more; those still without an answer score 0, as every failed answer does.
² Llama 4 Scout failed one pre-run check: asked whether it had any tools, it claimed a web-search tool it did not have. It ran in a sealed container with no network and no tools, so it was kept, and the failure is disclosed here.
The local models ran at 4-bit quantization on the maintainer's own hardware. gpt-oss runs in MXFP4, the precision its weights were released in; the other local models may lose some points against full-precision versions. All 22 models had every one of their 180 test answers scored.
Grok 4.7 scores well below the leaders, so its result was checked more than the others. In short: its answers are short, and the low score holds up when the grading or the rubrics are changed.
Its answers were complete. All 180 Grok 4.7 answers finished normally, in Hebrew, with no output limit set and none cut off; reasoning effort was high on every one. One request failed with a server error and was run again.
It writes short answers. Grok 4.7 thought at length (a median of about 18,000 reasoning tokens per answer) but wrote a median of 213 words, against 368 for Claude Sonnet 4.6 and 664 for Gemini 4 Argon. The rubrics give points for each source, step and ruling an answer gives, so a short answer that is right on the bottom line still scores low.
A blind second grader agreed with the judge. Claude Opus 5.5, without knowing which model wrote which answer, re-graded 30 answers: 15 by Grok 4.7 and 15 by Claude Sonnet 5.5 to the same questions. It agreed with GPT-6 Astra on 287 of 326 criterion decisions (88%), equally for both models. Grok 4.7's mean was 34.1 from Astra and 32.6 from the second grader, so Astra was not harder on Grok.
