sci-ml/lm-eval
A framework for evaluating language models (lm-evaluation-harness)
Screenshots
View
Download
BrowseChangeLog
commit 4fa38d7824d54ecd1947df00a03d063f2ac81e87
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Thu Sep 10 09:29:20 2026 +0200
sci-ml/lm-eval: condense 0.4.13 dependency rationale
commit c05c76b276d305b421d65367e904d5b153fa0b5f
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Thu Sep 10 09:29:20 2026 +0200
sci-ml/lm-eval: condense 0.4.12 dependency rationale
commit d8e29b741b65d9eb0604304cf257f6ddcabdc75a
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Thu Sep 10 02:00:08 2026 +0200
sci-ml/lm-eval: trim 0.4.12 comment prose
commit 6e0904ed53e48fb81784689d36e3f2d76b275d89
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Thu Sep 10 01:59:30 2026 +0200
sci-ml/lm-eval: trim 0.4.13 comment prose
commit b9ea8e8e7b0d491a5053c3898276fb98e8f5eb0a
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sun Sep 6 10:21:44 2026 +0200
sci-ml/lm-eval: Keyword 0.4.13 for ~arm64
The Python package and its evaluation dependencies support aarch64.
commit 14bd713bb7af95a9d834f5c4e5575f128ecefc52
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sun Sep 6 10:21:43 2026 +0200
sci-ml/lm-eval: Keyword 0.4.12 for ~arm64
The Python package and its evaluation dependencies support aarch64.
commit b9778651f59ac165b051b9fa64dedf1c91f59f34
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Tue Sep 1 15:43:44 2026 +0200
sci-ml/lm-eval: add onnxruntime USE flag
0.4.13 introduced two ONNX extras. onnxruntime-genai stays unwired -- no
ebuild for it exists anywhere -- but the plain onnxruntime one
(onnxruntime>=1.23, transformers, numpy) has every dep in tree, which is
the bar this ebuild sets for wiring an extra. numpy is already an
unconditional dep, so the flag adds exactly two atoms.
Worth having on this overlay specifically: onnxruntime_ort.py is the
backend that reaches the ROCm and MIGraphX execution providers, which
the onnxruntime-genai loop does not build. Upstream scopes it to the
log-likelihood path -- loglikelihood, multiple_choice and
loglikelihood_rolling; generative tasks are not supported yet -- and
metadata.xml says so.
Default-off and purely additive, so no revbump: both atoms sit inside
onnxruntime? ( ) guards, leaving a USE=-onnxruntime resolve
byte-for-byte what it was, and nothing about the installed files changes
either way. Nothing is imported eagerly either -- models/__init__.py
registers the backend by dotted-path string, and both heavy imports are
function-local (import onnxruntime in onnxruntime_ort.py, from
transformers import AutoTokenizer in the shared models/_onnx_base.py).
Dep shapes follow the existing split: sci-libs/onnxruntime is python-r1,
so it goes in the python_gen_cond_dep block with
[python,$]; sci-ml/transformers is DISTUTILS_SINGLE_IMPL,
so it takes $ alongside datasets/evaluate/vllm.
All three packages carry PYTHON_COMPAT=( python3_ ), so both
USEDEP forms are satisfiable, and sci-libs/onnxruntime-1.28.0-r2 and
-1.29.0-r2 both clear the >=1.23 floor with USE=python available.
Build-checked both ways; pkgcheck clean.
commit 06f495383f1d33f59741fca676f86047df5f8a3a
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Tue Sep 1 11:17:59 2026 +0200
sci-ml/lm-eval: drop 0.4.11
Retention: keep the last two (0.4.12, 0.4.13). No in-tree atom names
0.4.11 -- lm-eval is a leaf here, nothing depends on it.
commit ec3603efd8f4edc90dc1c56ab3d47e25c00e2bb7
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Tue Sep 1 11:17:45 2026 +0200
sci-ml/lm-eval: add 0.4.13
Core [project.dependencies] and all six wired extras (api, ifeval, math,
sentencepiece, discrim_eval, vllm) are unchanged from 0.4.12 -- the
whole pyproject delta is two new extras plus pillow in the dev extra.
Both load-bearing upstream assertions the ebuild documents are still in
place: the antlr4-python3-runtime 4.11 startswith() check in
lm_eval/tasks/minerva_math/utils.py, and the nltk>=3.9.1 gate in
lm_eval/tasks/leaderboard/ifeval/instructions_util.py.
Recorded the two new onnxruntime extras in the extras comment, with a
note that the plain one is the sole unwired extra whose deps are all
reachable in this overlay.
commit 3babf19bc10669f93e39b40b42769d2c0cdd5d60
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sun Jul 5 14:30:48 2026 +0200
sci-ml/lm-eval: condense the dependency comments
commit 89137542e87669b8f82c5479b03dad4e6f53614d
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Wed May 13 14:03:03 2026 +0200
sci-ml/lm-eval: add 0.4.12
commit 09a61b5c85e760aae3a0d01a5fe1af9385f8a413
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Mon May 11 17:38:59 2026 +0200
sci-ml/lm-eval: wire ifeval USE flag
Upstream's [project.optional-dependencies].ifeval extra is
{langdetect, immutabledict, nltk>=3.9.1} at v0.4.11. immutabledict
is in ::gentoo and nltk is already in this overlay; langdetect was
just forked from ::guru in the previous commit. With all three
reachable, ifeval joins api/math/sentencepiece/statsmodels/vllm as
a wirable extra (default off — it pulls a language-detection model
and the punkt tokenizer that are only useful for the
leaderboard_ifeval task battery).
The >=nltk-3.9.1 bound is load-bearing, not advisory: lm_eval's
instructions_util.py asserts the version at module import (older
nltk has a remote-code-exec via the `punkt` tokenizer downloader,
see nltk/nltk#3266). Note inline in the ebuild header so a future
bumper does not relax the bound thinking it is cosmetic.
Verified 2026-05-11: USE='api ifeval math sentencepiece statsmodels'
emerge sci-ml/lm-eval solves and installs cleanly on python3_13;
lm_eval.tasks.TaskManager loads the leaderboard_ifeval task end to
end with no ModuleNotFoundError.
commit f2c612aff555b51090727951547f2b36ed9ae54b
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Mon May 11 00:48:02 2026 +0200
sci-ml/lm-eval: wire math USE flag
Adds support for the minerva_math / leaderboard math / hendrycks_math
task families that grade LLM math answers via symbolic equality.
lm_eval/tasks/minerva_math/utils.py asserts
version("antlr4-python3-runtime").startswith("4.11")
at task-load, so the antlr4-4.11.* pin under math? is load-bearing,
not advisory; flipping USE=math triggers an antlr4 downgrade from
4.13.2 to the overlay-local 4.11.0. End-to-end verified on this host:
parse + verify of a boxed LaTeX answer returns the correct verdict.
commit da3078db530336c99abe95bd226ffb26ca2afc9f
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Mon May 11 00:58:48 2026 +0200
sci-ml/lm-eval: default-on api
Most lm-eval users running --model openai-chat-completions / openai-completions
/ anthropic / textsynth / generic API backends hit a NameError on ClientSession
at first request because lm_eval/models/api_models.py wraps the imports in a
try/except ModuleNotFoundError that silently swallows the missing-aiohttp at
import-time and only surfaces later when ClientSession is referenced. The
api-extra deps (aiohttp, requests, tenacity, tiktoken, tqdm) are small and
most lm-eval consumers will want them; default-on matches expected ergonomics.
Users who only run HF/vLLM backends can still set USE=-api.
commit 384e189360f1f03228878937e8c4342f2227578d
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sun May 10 21:07:11 2026 +0200
sci-ml/lm-eval: move vllm dep to $
dev-python/vllm is now SINGLE_IMPL; keeping it inside python_gen_cond_dep
with $ silently auto-satisfies via [X(-)?]. Move the
vllm? optional path out of the multi-impl wrap and pin to
$.
commit 1d01d45249503a960546427cdcb8956fa9a6556a
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sun May 10 14:45:18 2026 +0200
sci-ml/lm-eval: switch to DISTUTILS_SINGLE_IMPL
sci-ml/{datasets,evaluate} are SINGLE_IMPL; depending on them from a
multi-impl ebuild yields python_targets_python3_*(-)? that the child
can't expose. Make lm-eval single-impl, split SINGLE_IMPL deps onto bare
$ and wrap the multi-impl remainder (including
optional-USE deps for api/sentencepiece/statsmodels/vllm) in
python_gen_cond_dep.
commit 5bb48f7173234083230f6146e701cac83bfb8cc5
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sat May 9 22:09:03 2026 +0200
sci-ml/lm-eval: new package, EleutherAI lm-evaluation-harness 0.4.11
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Thu Sep 10 09:29:20 2026 +0200
sci-ml/lm-eval: condense 0.4.13 dependency rationale
commit c05c76b276d305b421d65367e904d5b153fa0b5f
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Thu Sep 10 09:29:20 2026 +0200
sci-ml/lm-eval: condense 0.4.12 dependency rationale
commit d8e29b741b65d9eb0604304cf257f6ddcabdc75a
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Thu Sep 10 02:00:08 2026 +0200
sci-ml/lm-eval: trim 0.4.12 comment prose
commit 6e0904ed53e48fb81784689d36e3f2d76b275d89
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Thu Sep 10 01:59:30 2026 +0200
sci-ml/lm-eval: trim 0.4.13 comment prose
commit b9ea8e8e7b0d491a5053c3898276fb98e8f5eb0a
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sun Sep 6 10:21:44 2026 +0200
sci-ml/lm-eval: Keyword 0.4.13 for ~arm64
The Python package and its evaluation dependencies support aarch64.
commit 14bd713bb7af95a9d834f5c4e5575f128ecefc52
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sun Sep 6 10:21:43 2026 +0200
sci-ml/lm-eval: Keyword 0.4.12 for ~arm64
The Python package and its evaluation dependencies support aarch64.
commit b9778651f59ac165b051b9fa64dedf1c91f59f34
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Tue Sep 1 15:43:44 2026 +0200
sci-ml/lm-eval: add onnxruntime USE flag
0.4.13 introduced two ONNX extras. onnxruntime-genai stays unwired -- no
ebuild for it exists anywhere -- but the plain onnxruntime one
(onnxruntime>=1.23, transformers, numpy) has every dep in tree, which is
the bar this ebuild sets for wiring an extra. numpy is already an
unconditional dep, so the flag adds exactly two atoms.
Worth having on this overlay specifically: onnxruntime_ort.py is the
backend that reaches the ROCm and MIGraphX execution providers, which
the onnxruntime-genai loop does not build. Upstream scopes it to the
log-likelihood path -- loglikelihood, multiple_choice and
loglikelihood_rolling; generative tasks are not supported yet -- and
metadata.xml says so.
Default-off and purely additive, so no revbump: both atoms sit inside
onnxruntime? ( ) guards, leaving a USE=-onnxruntime resolve
byte-for-byte what it was, and nothing about the installed files changes
either way. Nothing is imported eagerly either -- models/__init__.py
registers the backend by dotted-path string, and both heavy imports are
function-local (import onnxruntime in onnxruntime_ort.py, from
transformers import AutoTokenizer in the shared models/_onnx_base.py).
Dep shapes follow the existing split: sci-libs/onnxruntime is python-r1,
so it goes in the python_gen_cond_dep block with
[python,$]; sci-ml/transformers is DISTUTILS_SINGLE_IMPL,
so it takes $ alongside datasets/evaluate/vllm.
All three packages carry PYTHON_COMPAT=( python3_ ), so both
USEDEP forms are satisfiable, and sci-libs/onnxruntime-1.28.0-r2 and
-1.29.0-r2 both clear the >=1.23 floor with USE=python available.
Build-checked both ways; pkgcheck clean.
commit 06f495383f1d33f59741fca676f86047df5f8a3a
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Tue Sep 1 11:17:59 2026 +0200
sci-ml/lm-eval: drop 0.4.11
Retention: keep the last two (0.4.12, 0.4.13). No in-tree atom names
0.4.11 -- lm-eval is a leaf here, nothing depends on it.
commit ec3603efd8f4edc90dc1c56ab3d47e25c00e2bb7
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Tue Sep 1 11:17:45 2026 +0200
sci-ml/lm-eval: add 0.4.13
Core [project.dependencies] and all six wired extras (api, ifeval, math,
sentencepiece, discrim_eval, vllm) are unchanged from 0.4.12 -- the
whole pyproject delta is two new extras plus pillow in the dev extra.
Both load-bearing upstream assertions the ebuild documents are still in
place: the antlr4-python3-runtime 4.11 startswith() check in
lm_eval/tasks/minerva_math/utils.py, and the nltk>=3.9.1 gate in
lm_eval/tasks/leaderboard/ifeval/instructions_util.py.
Recorded the two new onnxruntime extras in the extras comment, with a
note that the plain one is the sole unwired extra whose deps are all
reachable in this overlay.
commit 3babf19bc10669f93e39b40b42769d2c0cdd5d60
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sun Jul 5 14:30:48 2026 +0200
sci-ml/lm-eval: condense the dependency comments
commit 89137542e87669b8f82c5479b03dad4e6f53614d
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Wed May 13 14:03:03 2026 +0200
sci-ml/lm-eval: add 0.4.12
commit 09a61b5c85e760aae3a0d01a5fe1af9385f8a413
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Mon May 11 17:38:59 2026 +0200
sci-ml/lm-eval: wire ifeval USE flag
Upstream's [project.optional-dependencies].ifeval extra is
{langdetect, immutabledict, nltk>=3.9.1} at v0.4.11. immutabledict
is in ::gentoo and nltk is already in this overlay; langdetect was
just forked from ::guru in the previous commit. With all three
reachable, ifeval joins api/math/sentencepiece/statsmodels/vllm as
a wirable extra (default off — it pulls a language-detection model
and the punkt tokenizer that are only useful for the
leaderboard_ifeval task battery).
The >=nltk-3.9.1 bound is load-bearing, not advisory: lm_eval's
instructions_util.py asserts the version at module import (older
nltk has a remote-code-exec via the `punkt` tokenizer downloader,
see nltk/nltk#3266). Note inline in the ebuild header so a future
bumper does not relax the bound thinking it is cosmetic.
Verified 2026-05-11: USE='api ifeval math sentencepiece statsmodels'
emerge sci-ml/lm-eval solves and installs cleanly on python3_13;
lm_eval.tasks.TaskManager loads the leaderboard_ifeval task end to
end with no ModuleNotFoundError.
commit f2c612aff555b51090727951547f2b36ed9ae54b
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Mon May 11 00:48:02 2026 +0200
sci-ml/lm-eval: wire math USE flag
Adds support for the minerva_math / leaderboard math / hendrycks_math
task families that grade LLM math answers via symbolic equality.
lm_eval/tasks/minerva_math/utils.py asserts
version("antlr4-python3-runtime").startswith("4.11")
at task-load, so the antlr4-4.11.* pin under math? is load-bearing,
not advisory; flipping USE=math triggers an antlr4 downgrade from
4.13.2 to the overlay-local 4.11.0. End-to-end verified on this host:
parse + verify of a boxed LaTeX answer returns the correct verdict.
commit da3078db530336c99abe95bd226ffb26ca2afc9f
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Mon May 11 00:58:48 2026 +0200
sci-ml/lm-eval: default-on api
Most lm-eval users running --model openai-chat-completions / openai-completions
/ anthropic / textsynth / generic API backends hit a NameError on ClientSession
at first request because lm_eval/models/api_models.py wraps the imports in a
try/except ModuleNotFoundError that silently swallows the missing-aiohttp at
import-time and only surfaces later when ClientSession is referenced. The
api-extra deps (aiohttp, requests, tenacity, tiktoken, tqdm) are small and
most lm-eval consumers will want them; default-on matches expected ergonomics.
Users who only run HF/vLLM backends can still set USE=-api.
commit 384e189360f1f03228878937e8c4342f2227578d
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sun May 10 21:07:11 2026 +0200
sci-ml/lm-eval: move vllm dep to $
dev-python/vllm is now SINGLE_IMPL; keeping it inside python_gen_cond_dep
with $ silently auto-satisfies via [X(-)?]. Move the
vllm? optional path out of the multi-impl wrap and pin to
$.
commit 1d01d45249503a960546427cdcb8956fa9a6556a
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sun May 10 14:45:18 2026 +0200
sci-ml/lm-eval: switch to DISTUTILS_SINGLE_IMPL
sci-ml/{datasets,evaluate} are SINGLE_IMPL; depending on them from a
multi-impl ebuild yields python_targets_python3_*(-)? that the child
can't expose. Make lm-eval single-impl, split SINGLE_IMPL deps onto bare
$ and wrap the multi-impl remainder (including
optional-USE deps for api/sentencepiece/statsmodels/vllm) in
python_gen_cond_dep.
commit 5bb48f7173234083230f6146e701cac83bfb8cc5
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sat May 9 22:09:03 2026 +0200
sci-ml/lm-eval: new package, EleutherAI lm-evaluation-harness 0.4.11