13,647 research outputs found
Colección: Perfil #3
No full textThis board-book version of LM turns out to be quite creative. Ratoncete comes from school every afternoon and goes through the forest looking for adventures. He apparently blasts a horn into the ear of the sleeping lion. Don Leon wants to spank him as a result, but Ratoncete offers an apology, not an offer of help. Later, he happens upon the lion in his trap of ropes. 8 pages, counting both covers. 6½" x 9".Language note: SpanishNo Autho
EleutherAI/lm-evaluation-harness: v0.4.2
No full text<h1>lm-eval v0.4.2 Release Notes</h1>
<p>We are releasing a new minor version of lm-eval for PyPI users! We've been very happy to see continued usage of the lm-evaluation-harness, including as a standard testbench to propel new architecture design (https://arxiv.org/abs/2402.18668), to ease new benchmark creation (https://arxiv.org/abs/2402.11548, https://arxiv.org/abs/2402.00786, https://arxiv.org/abs/2403.01469), enabling controlled experimentation on LLM evaluation (https://arxiv.org/abs/2402.01781), and more!</p>
<h2>New Additions</h2>
<ul>
<li>Request Caching by @inf3rnus - speedups on startup via caching the construction of documents/requests' contexts</li>
<li>Weights and Biases logging by @ayulockin - evals can now be logged to both WandB and Zeno!</li>
<li>New Tasks<ul>
<li>KMMLU, a localized - not (auto) translated! - dataset for testing Korean knowledge by @h-albert-lee @guijinSON</li>
<li>GPQA by @uanu2002</li>
<li>French Bench by @ManuelFay</li>
<li>EQ-Bench by @pbevan1 and @sqrkl</li>
<li>HAERAE-Bench, readded by @h-albert-lee</li>
<li>Updates to answer parsing on many generative tasks (GSM8k, MGSM, BBH zeroshot) by @thinknbtfly!</li>
<li>Okapi (translated) Open LLM Leaderboard tasks by @uanu2002 and @giux78</li>
<li>Arabic MMLU and aEXAMS by @khalil-hennara</li>
<li>And more!</li>
</ul>
</li>
<li>Re-introduction of <code>TemplateLM</code> base class for lower-code new LM class implementations by @anjor</li>
<li>Run the library with metrics/scoring stage skipped via <code>--predict_only</code> by @baberabb</li>
<li>Many more miscellaneous improvements by a lot of great contributors!</li>
</ul>
<h2>Backwards Incompatibilities</h2>
<p>There were a few breaking changes to lm-eval's general API or logic we'd like to highlight:</p>
<h3><code>TaskManager</code> API</h3>
<p>previously, users had to call <code>lm_eval.tasks.initialize_tasks()</code> to register the library's default tasks, or <code>lm_eval.tasks.include_path()</code> to include a custom directory of task YAML configs.</p>
<p>Old usage:</p>
<pre><code>import lm_eval
lm_eval.tasks.initialize_tasks()
# or:
lm_eval.tasks.include_path("/path/to/my/custom/tasks")
lm_eval.simple_evaluate(model=lm, tasks=["arc_easy"])
</code></pre>
<p>New intended usage:</p>
<pre><code>import lm_eval
# optional--only need to instantiate separately if you want to pass custom path!
task_manager = TaskManager() # pass include_path="/path/to/my/custom/tasks" if desired
lm_eval.simple_evaluate(model=lm, tasks=["arc_easy"], task_manager=task_manager)
</code></pre>
<p><code>get_task_dict()</code> now also optionally takes a TaskManager object, when wanting to load custom tasks.</p>
<p>This should allow for much faster library startup times due to lazily loading requested tasks or groups.</p>
<h3>Updated Stderr Aggregation</h3>
<p>Previous versions of the library incorrectly reported erroneously large <code>stderr</code> scores for groups of tasks such as MMLU.</p>
<p>We've since updated the formula to correctly aggregate Standard Error scores for groups of tasks reporting accuracies aggregated via their mean across the dataset -- see #1390 #1427 for more information.</p>
<p>As always, please feel free to give us feedback or request new features! We're grateful for the community's support.</p>
<h2>What's Changed</h2>
<ul>
<li>Add support for RWKV models with World tokenizer by @PicoCreator in https://github.com/EleutherAI/lm-evaluation-harness/pull/1374</li>
<li>add bypass metric by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1156</li>
<li>Expand docs, update CITATION.bib by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1227</li>
<li>Hf: minor egde cases by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1380</li>
<li>Enable override of printed <code>n-shot</code> in table by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1379</li>
<li>Faster Task and Group Loading, Allow Recursive Groups by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1321</li>
<li>Fix for https://github.com/EleutherAI/lm-evaluation-harness/issues/1383 by @pminervini in https://github.com/EleutherAI/lm-evaluation-harness/pull/1384</li>
<li>fix on --task list by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1387</li>
<li>Support for Inf2 optimum class [WIP] by @michaelfeil in https://github.com/EleutherAI/lm-evaluation-harness/pull/1364</li>
<li>Update README.md by @mycoalchen in https://github.com/EleutherAI/lm-evaluation-harness/pull/1398</li>
<li>Fix confusing <code>write_out.py</code> instructions in README by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1371</li>
<li>Use Pooled rather than Combined Variance for calculating stderr of task groupings by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1390</li>
<li>adding hf_transfer by @michaelfeil in https://github.com/EleutherAI/lm-evaluation-harness/pull/1400</li>
<li><code>batch_size</code> with <code>auto</code> defaults to 1 if <code>No executable batch size found</code> is raised by @pminervini in https://github.com/EleutherAI/lm-evaluation-harness/pull/1405</li>
<li>Fix printing bug in #1390 by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1414</li>
<li>Fixes https://github.com/EleutherAI/lm-evaluation-harness/issues/1416 by @pminervini in https://github.com/EleutherAI/lm-evaluation-harness/pull/1418</li>
<li>Fix watchdog timeout by @JeevanBhoot in https://github.com/EleutherAI/lm-evaluation-harness/pull/1404</li>
<li>Evaluate by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1385</li>
<li>Add multilingual ARC task by @uanu2002 in https://github.com/EleutherAI/lm-evaluation-harness/pull/1419</li>
<li>Add multilingual TruthfulQA task by @uanu2002 in https://github.com/EleutherAI/lm-evaluation-harness/pull/1420</li>
<li>[m_mmul] added multilingual evaluation from alexandrainst/m_mmlu by @giux78 in https://github.com/EleutherAI/lm-evaluation-harness/pull/1358</li>
<li>Added seeds to <code>evaluator.simple_evaluate</code> signature by @Am1n3e in https://github.com/EleutherAI/lm-evaluation-harness/pull/1412</li>
<li>Fix: task weighting by subtask size ; update Pooled Stderr formula slightly by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1427</li>
<li>Refactor utilities into a separate model utils file. by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1429</li>
<li>Nit fix: Updated OpenBookQA Readme by @adavidho in https://github.com/EleutherAI/lm-evaluation-harness/pull/1430</li>
<li>improve hf_transfer activation by @michaelfeil in https://github.com/EleutherAI/lm-evaluation-harness/pull/1438</li>
<li>Correct typo in task name in ARC documentation by @larekrow in https://github.com/EleutherAI/lm-evaluation-harness/pull/1443</li>
<li>update bbh, gsm8k, mmlu parsing logic and prompts (Orca2 bbh_cot_zeroshot 0% -> 42%) by @thnkinbtfly in https://github.com/EleutherAI/lm-evaluation-harness/pull/1356</li>
<li>Add a new task HaeRae-Bench by @h-albert-lee in https://github.com/EleutherAI/lm-evaluation-harness/pull/1445</li>
<li>Group reqs by context by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1425</li>
<li>Add a new task GPQA (the part without CoT) by @uanu2002 in https://github.com/EleutherAI/lm-evaluation-harness/pull/1434</li>
<li>Added KMMLU evaluation method and changed ReadMe by @h-albert-lee in https://github.com/EleutherAI/lm-evaluation-harness/pull/1447</li>
<li>Add TemplateLM boilerplate LM class by @anjor in https://github.com/EleutherAI/lm-evaluation-harness/pull/1279</li>
<li>Log which subtasks were called with which groups by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1456</li>
<li>PR fixing the issue #1391 (wrong contexts in the mgsm task) by @leocnj in https://github.com/EleutherAI/lm-evaluation-harness/pull/1440</li>
<li>feat: Add Weights and Biases support by @ayulockin in https://github.com/EleutherAI/lm-evaluation-harness/pull/1339</li>
<li>Fixed generation args issue affection OpenAI completion model by @Am1n3e in https://github.com/EleutherAI/lm-evaluation-harness/pull/1458</li>
<li>update parsing logic of mgsm following gsm8k (mgsm en 0 -> 50%) by @thnkinbtfly in https://github.com/EleutherAI/lm-evaluation-harness/pull/1462</li>
<li>Adding documentation for Weights and Biases CLI interface by @veekaybee in https://github.com/EleutherAI/lm-evaluation-harness/pull/1466</li>
<li>Add environment and transformers version logging in results dump by @LSinev in https://github.com/EleutherAI/lm-evaluation-harness/pull/1464</li>
<li>Apply code autoformatting with Ruff to tasks/*.py an *<strong>init</strong>.py by @LSinev in https://github.com/EleutherAI/lm-evaluation-harness/pull/1469</li>
<li>Setting trust_remote_code to <code>True</code> for HuggingFace datasets compatibility by @veekaybee in https://github.com/EleutherAI/lm-evaluation-harness/pull/1467</li>
<li>add arabic mmlu by @khalil-Hennara in https://github.com/EleutherAI/lm-evaluation-harness/pull/1402</li>
<li>Add Gemma support (Add flag to control BOS token usage) by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1465</li>
<li>Revert "Setting trust_remote_code to <code>True</code> for HuggingFace datasets compatibility" by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1474</li>
<li>Create a means for caching task registration and request building. Ad… by @inf3rnus in https://github.com/EleutherAI/lm-evaluation-harness/pull/1372</li>
<li>Cont metrics by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1475</li>
<li>Refactor <code>evaluater.evaluate</code> by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1441</li>
<li>add multilingual mmlu eval by @jordane95 in https://github.com/EleutherAI/lm-evaluation-harness/pull/1484</li>
<li>Update TruthfulQA val split name by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1488</li>
<li>Fix AttributeError in huggingface.py When 'model_type' is Missing by @richwardle in https://github.com/EleutherAI/lm-evaluation-harness/pull/1489</li>
<li>Fix duplicated kwargs in some model init by @lchu-ibm in https://github.com/EleutherAI/lm-evaluation-harness/pull/1495</li>
<li>Add multilingual truthfulqa targets by @jordane95 in https://github.com/EleutherAI/lm-evaluation-harness/pull/1499</li>
<li>Always include EOS token as stop sequence by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1480</li>
<li>Improve data-parallel request partitioning for VLLM by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1477</li>
<li>modify <code>WandbLogger</code> to accept arbitrary kwargs by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1491</li>
<li>Vllm update DP+TP by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1508</li>
<li>Setting trust_remote_code to True for HuggingFace datasets compatibility by @veekaybee in https://github.com/EleutherAI/lm-evaluation-harness/pull/1487</li>
<li>Cleaning up unused unit tests by @veekaybee in https://github.com/EleutherAI/lm-evaluation-harness/pull/1516</li>
<li>French Bench by @ManuelFay in https://github.com/EleutherAI/lm-evaluation-harness/pull/1500</li>
<li>Hotfix: fix TypeError in <code>--trust_remote_code</code> by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1517</li>
<li>Fix minor edge cases (#951 #1503) by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1520</li>
<li>Openllm benchmark by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1526</li>
<li>Add a new task GPQA (the part CoT and generative) by @uanu2002 in https://github.com/EleutherAI/lm-evaluation-harness/pull/1482</li>
<li>Add EQ-Bench as per #1459 by @pbevan1 in https://github.com/EleutherAI/lm-evaluation-harness/pull/1511</li>
<li>Add WMDP Multiple-choice by @justinphan3110 in https://github.com/EleutherAI/lm-evaluation-harness/pull/1534</li>
<li>Adding new task : KorMedMCQA by @sean0042 in https://github.com/EleutherAI/lm-evaluation-harness/pull/1530</li>
<li>Update docs on LM.loglikelihood_rolling abstract method by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1532</li>
<li>Minor KMMLU cleanup by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1502</li>
<li>Cleanup and fixes (Task, Instance, and a little bit of *evaluate) by @LSinev in https://github.com/EleutherAI/lm-evaluation-harness/pull/1533</li>
<li>Update installation commands in openai_completions.py and contributing document and, update wandb_args description by @naem1023 in https://github.com/EleutherAI/lm-evaluation-harness/pull/1536</li>
<li>Add compatibility for vLLM's new Logprob object by @Yard1 in https://github.com/EleutherAI/lm-evaluation-harness/pull/1549</li>
<li>Fix incorrect <code>max_gen_toks</code> generation kwarg default in code2_text. by @cosmo3769 in https://github.com/EleutherAI/lm-evaluation-harness/pull/1551</li>
<li>Support jinja templating for task descriptions by @HishamYahya in https://github.com/EleutherAI/lm-evaluation-harness/pull/1553</li>
<li>Fix incorrect <code>max_gen_toks</code> generation kwarg default in generative Bigbench by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1546</li>
<li>Hardcode IFEval to 0-shot by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1506</li>
<li>add Arabic EXAMS benchmark by @khalil-Hennara in https://github.com/EleutherAI/lm-evaluation-harness/pull/1498</li>
<li>AGIEval by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1359</li>
<li>cli_evaluate calls simple_evaluate with the same verbosity. by @Wongboo in https://github.com/EleutherAI/lm-evaluation-harness/pull/1563</li>
<li>add manual tqdm disabling management by @artemorloff in https://github.com/EleutherAI/lm-evaluation-harness/pull/1569</li>
<li>Fix README section on vllm integration by @eitanturok in https://github.com/EleutherAI/lm-evaluation-harness/pull/1579</li>
<li>Fix Jinja template for Advanced AI Risk by @RylanSchaeffer in https://github.com/EleutherAI/lm-evaluation-harness/pull/1587</li>
<li>Proposed approach for testing CLI arg parsing by @veekaybee in https://github.com/EleutherAI/lm-evaluation-harness/pull/1566</li>
<li>Patch for Seq2Seq Model predictions by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1584</li>
<li>Add start date in results.json by @djstrong in https://github.com/EleutherAI/lm-evaluation-harness/pull/1592</li>
<li>Cleanup for v0.4.2 release by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1573</li>
<li>Fix eval_logger import for mmlu/_generate_configs.py by @noufmitla in https://github.com/EleutherAI/lm-evaluation-harness/pull/1593</li>
</ul>
<h2>New Contributors</h2>
<ul>
<li>@PicoCreator made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1374</li>
<li>@michaelfeil made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1364</li>
<li>@mycoalchen made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1398</li>
<li>@JeevanBhoot made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1404</li>
<li>@uanu2002 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1419</li>
<li>@giux78 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1358</li>
<li>@Am1n3e made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1412</li>
<li>@adavidho made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1430</li>
<li>@larekrow made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1443</li>
<li>@leocnj made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1440</li>
<li>@ayulockin made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1339</li>
<li>@khalil-Hennara made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1402</li>
<li>@inf3rnus made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1372</li>
<li>@jordane95 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1484</li>
<li>@richwardle made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1489</li>
<li>@lchu-ibm made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1495</li>
<li>@pbevan1 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1511</li>
<li>@justinphan3110 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1534</li>
<li>@sean0042 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1530</li>
<li>@naem1023 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1536</li>
<li>@Yard1 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1549</li>
<li>@cosmo3769 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1551</li>
<li>@HishamYahya made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1553</li>
<li>@Wongboo made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1563</li>
<li>@artemorloff made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1569</li>
<li>@eitanturok made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1579</li>
<li>@RylanSchaeffer made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1587</li>
<li>@noufmitla made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1593</li>
</ul>
<p><strong>Full Changelog</strong>: https://github.com/EleutherAI/lm-evaluation-harness/compare/v0.4.1...v0.4.2</p>
EleutherAI/lm-evaluation-harness: v0.4.1
No full text<h2>Release Notes</h2>
<p>This PR release contains all changes so far since the release of v0.4.0 , and is partially a test of our release automation, provided by @anjor .</p>
<p>At a high level, some of the changes include:</p>
<ul>
<li>Data-parallel inference using vLLM (contributed by @baberabb )</li>
<li>A major fix to Huggingface model generation--previously, in v0.4.0, due to a bug with stop sequence handling, generations were sometimes cut off too early.</li>
<li>Miscellaneous documentation updates</li>
<li>A number of new tasks, and bugfixes to old tasks!</li>
<li>The support for OpenAI-like API models using <code>local-completions</code> or <code>local-chat-completions</code> ( Thanks to @veekaybee @mgoin @anjor and others on this)!</li>
</ul>
<p>More frequent (minor) version releases may be done in the future, to make it easier for PyPI users!</p>
<p>We're very pleased by the uptick in interest in LM Evaluation Harness recently, and we hope to continue to improve the library as time goes on. We're grateful to everyone who's contributed, and are excited by how many new contributors this version brings! If you have feedback for us, or would like to help out developing the library, please let us know.</p>
<p>In the next version release, we hope to include</p>
<ul>
<li>Chat Templating + System Prompt support, for locally-run models</li>
<li>Improved Answer Extraction for many generative tasks</li>
<li>General speedups and QoL fixes to the non-inference portions of LM-Evaluation-Harness, including startup times and speedups when num_fewshot is large!</li>
<li>A new <code>TaskManager</code> object and the deprecation of <code>lm_eval.tasks.initialize_tasks()</code>, for achieving the easier registration of many tasks and configuration of new groups of tasks</li>
</ul>
<h2>What's Changed</h2>
<ul>
<li>Announce v0.4.0 in README by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1061</li>
<li>remove commented planned samplers in <code>lm_eval/api/samplers.py</code> by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1062</li>
<li>Confirming links in docs work (WIP) by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1065</li>
<li>Set actual version to v0.4.0 by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1064</li>
<li>Updating docs hyperlinks by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1066</li>
<li>Fiddling with READMEs, Reenable CI tests on <code>main</code> by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1063</li>
<li>Update _cot_fewshot_template_yaml by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1074</li>
<li>Patch scrolls by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1077</li>
<li>Update template of qqp dataset by @shiweijiezero in https://github.com/EleutherAI/lm-evaluation-harness/pull/1097</li>
<li>Change the sub-task name from sst to sst2 in glue by @shiweijiezero in https://github.com/EleutherAI/lm-evaluation-harness/pull/1099</li>
<li>Add kmmlu evaluation to tasks by @h-albert-lee in https://github.com/EleutherAI/lm-evaluation-harness/pull/1089</li>
<li>Fix stderr by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1106</li>
<li>Simplified <code>evaluator.py</code> by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1104</li>
<li>[Refactor] vllm data parallel by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1035</li>
<li>Unpack group in <code>write_out</code> by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1113</li>
<li>Revert "Simplified <code>evaluator.py</code>" by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1116</li>
<li><code>qqp</code>, <code>mnli_mismatch</code>: remove unlabled test sets by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1114</li>
<li>fix: bug of BBH_cot_fewshot by @Momo-Tori in https://github.com/EleutherAI/lm-evaluation-harness/pull/1118</li>
<li>Bump BBH version by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1120</li>
<li>Refactor <code>hf</code> modeling code by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1096</li>
<li>Additional process for doc_to_choice by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1093</li>
<li>doc_to_decontamination_query can use function by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1082</li>
<li>Fix vllm <code>batch_size</code> type by @xTayEx in https://github.com/EleutherAI/lm-evaluation-harness/pull/1128</li>
<li>fix: passing max_length to vllm engine args by @NanoCode012 in https://github.com/EleutherAI/lm-evaluation-harness/pull/1124</li>
<li>Fix Loading Local Dataset by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1127</li>
<li>place model onto <code>mps</code> by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1133</li>
<li>Add benchmark FLD by @MorishT in https://github.com/EleutherAI/lm-evaluation-harness/pull/1122</li>
<li>fix typo in README.md by @lennijusten in https://github.com/EleutherAI/lm-evaluation-harness/pull/1136</li>
<li>add correct openai api key to README.md by @lennijusten in https://github.com/EleutherAI/lm-evaluation-harness/pull/1138</li>
<li>Update Linter CI Job by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1130</li>
<li>add utils.clear_torch_cache() to model_comparator by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1142</li>
<li>Enabling OpenAI completions via gooseai by @veekaybee in https://github.com/EleutherAI/lm-evaluation-harness/pull/1141</li>
<li>vllm clean up tqdm by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1144</li>
<li>openai nits by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1139</li>
<li>Add IFEval / Instruction-Following Eval by @wiskojo in https://github.com/EleutherAI/lm-evaluation-harness/pull/1087</li>
<li>set <code>--gen_kwargs</code> arg to None by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1145</li>
<li>Add shorthand flags by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1149</li>
<li>fld bugfix by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1150</li>
<li>Remove GooseAI docs and change no-commit-to-branch precommit hook by @veekaybee in https://github.com/EleutherAI/lm-evaluation-harness/pull/1154</li>
<li>Add docs on adding a multiple choice metric by @polm-stability in https://github.com/EleutherAI/lm-evaluation-harness/pull/1147</li>
<li>Simplify evaluator by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1126</li>
<li>Generalize Qwen tokenizer fix by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1146</li>
<li>self.device in huggingface.py line 210 treated as torch.device but might be a string by @pminervini in https://github.com/EleutherAI/lm-evaluation-harness/pull/1172</li>
<li>Fix Column Naming and Dataset Naming Conventions in K-MMLU Evaluation by @seungduk-yanolja in https://github.com/EleutherAI/lm-evaluation-harness/pull/1171</li>
<li>feat: add option to upload results to Zeno by @Sparkier in https://github.com/EleutherAI/lm-evaluation-harness/pull/990</li>
<li>Switch Linting to <code>ruff</code> by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1166</li>
<li>Error in --num_fewshot option for K-MMLU Evaluation Harness by @guijinSON in https://github.com/EleutherAI/lm-evaluation-harness/pull/1178</li>
<li>Implementing local OpenAI API-style chat completions on any given inference server by @veekaybee in https://github.com/EleutherAI/lm-evaluation-harness/pull/1174</li>
<li>Update README.md by @anjor in https://github.com/EleutherAI/lm-evaluation-harness/pull/1184</li>
<li>Update README.md by @anjor in https://github.com/EleutherAI/lm-evaluation-harness/pull/1183</li>
<li>Add tokenizer backend by @anjor in https://github.com/EleutherAI/lm-evaluation-harness/pull/1186</li>
<li>Correctly Print Task Versioning by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1173</li>
<li>update Zeno example and reference in README by @Sparkier in https://github.com/EleutherAI/lm-evaluation-harness/pull/1190</li>
<li>Remove tokenizer for openai chat completions by @anjor in https://github.com/EleutherAI/lm-evaluation-harness/pull/1191</li>
<li>Update README.md by @anjor in https://github.com/EleutherAI/lm-evaluation-harness/pull/1181</li>
<li>disable <code>mypy</code> by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1193</li>
<li>Generic decorator for handling rate limit errors by @zachschillaci27 in https://github.com/EleutherAI/lm-evaluation-harness/pull/1109</li>
<li>Refer in README to main branch by @BramVanroy in https://github.com/EleutherAI/lm-evaluation-harness/pull/1200</li>
<li>Hardcode 0-shot for fewshot Minerva Math tasks by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1189</li>
<li>Upstream Mamba Support (<code>mamba_ssm</code>) by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1110</li>
<li>Update cuda handling by @anjor in https://github.com/EleutherAI/lm-evaluation-harness/pull/1180</li>
<li>Fix documentation in API table by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1203</li>
<li>Consolidate batching by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1197</li>
<li>Add remove_whitespace to FLD benchmark by @MorishT in https://github.com/EleutherAI/lm-evaluation-harness/pull/1206</li>
<li>Fix the argument order in <code>utils.divide</code> doc by @xTayEx in https://github.com/EleutherAI/lm-evaluation-harness/pull/1208</li>
<li>[Fix #1211 ] pin vllm at < 0.2.6 by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1212</li>
<li>fix unbounded local variable by @onnoo in https://github.com/EleutherAI/lm-evaluation-harness/pull/1218</li>
<li>nits + fix siqa by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1216</li>
<li>add length of strings and answer options to Zeno metadata by @Sparkier in https://github.com/EleutherAI/lm-evaluation-harness/pull/1222</li>
<li>Don't silence errors when loading tasks by @polm-stability in https://github.com/EleutherAI/lm-evaluation-harness/pull/1148</li>
<li>Update README.md by @anjor in https://github.com/EleutherAI/lm-evaluation-harness/pull/1195</li>
<li>Update race's README.md by @pminervini in https://github.com/EleutherAI/lm-evaluation-harness/pull/1230</li>
<li>batch_schedular bug in Collator by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1229</li>
<li>Update openai_completions.py by @StellaAthena in https://github.com/EleutherAI/lm-evaluation-harness/pull/1238</li>
<li>vllm: handle max_length better and substitute Collator by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1241</li>
<li>Remove self.dataset_path post_init process by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1243</li>
<li>Add multilingual HellaSwag task by @JorgeDeCorte in https://github.com/EleutherAI/lm-evaluation-harness/pull/1228</li>
<li>Do not escape ascii in logging outputs by @passaglia in https://github.com/EleutherAI/lm-evaluation-harness/pull/1246</li>
<li>fixed fewshot loading for multiple input tasks by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1255</li>
<li>Revert citation by @StellaAthena in https://github.com/EleutherAI/lm-evaluation-harness/pull/1257</li>
<li>Specify utf-8 encoding to properly save non-ascii samples to file by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1265</li>
<li>Fix evaluation for the belebele dataset by @jmichaelov in https://github.com/EleutherAI/lm-evaluation-harness/pull/1267</li>
<li>Call "exact_match" once for each multiple-target sample by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1266</li>
<li>MultiMedQA by @tmabraham in https://github.com/EleutherAI/lm-evaluation-harness/pull/1198</li>
<li>Fix bug in multi-token Stop Sequences by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1268</li>
<li>Update Table Printing by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1271</li>
<li>add Kobest by @jp1924 in https://github.com/EleutherAI/lm-evaluation-harness/pull/1263</li>
<li>Apply <code>process_docs()</code> to fewshot_split by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1276</li>
<li>Fix whitespace issues in GSM8k-CoT by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1275</li>
<li>Make <code>parallelize=True</code> vs. <code>accelerate launch</code> distinction clearer in docs by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1261</li>
<li>Allow parameter edits for registered tasks when listed in a benchmark by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1273</li>
<li>Fix data-parallel evaluation with quantized models by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1270</li>
<li>Rework documentation for explaining local dataset by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1284</li>
<li>Update CITATION.bib by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1285</li>
<li>Update <code>nq_open</code> / NaturalQs whitespacing by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1289</li>
<li>Update README.md with custom integration doc by @msaroufim in https://github.com/EleutherAI/lm-evaluation-harness/pull/1298</li>
<li>Update nq_open.yaml by @Hannibal046 in https://github.com/EleutherAI/lm-evaluation-harness/pull/1305</li>
<li>Update task_guide.md by @daniellepintz in https://github.com/EleutherAI/lm-evaluation-harness/pull/1306</li>
<li>Pin <code>datasets</code> dependency at 2.15 by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1312</li>
<li>Fix polemo2_in.yaml subset name by @lhoestq in https://github.com/EleutherAI/lm-evaluation-harness/pull/1313</li>
<li>Fix <code>datasets</code> dependency to >=2.14 by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1314</li>
<li>Fix group register by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1315</li>
<li>Update task_guide.md by @djstrong in https://github.com/EleutherAI/lm-evaluation-harness/pull/1316</li>
<li>Update polemo2_in.yaml by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1318</li>
<li>Fix: Mamba receives extra kwargs by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1328</li>
<li>Fix Issue regarding stderr by @lintangsutawika in https://github.com/EleutherAI/lm-evaluation-harness/pull/1327</li>
<li>Add <code>local-completions</code> support using OpenAI interface by @mgoin in https://github.com/EleutherAI/lm-evaluation-harness/pull/1277</li>
<li>fallback to classname when LM doesnt have config by @nairbv in https://github.com/EleutherAI/lm-evaluation-harness/pull/1334</li>
<li>fix a trailing whitespace that breaks a lint job by @nairbv in https://github.com/EleutherAI/lm-evaluation-harness/pull/1335</li>
<li>skip "benchmarks" in changed_tasks by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1336</li>
<li>Update migrated HF dataset paths by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1332</li>
<li>Don't use <code>get_task_dict()</code> in task registration / initialization by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1331</li>
<li>manage default (greedy) gen_kwargs in vllm by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1341</li>
<li>vllm: change default gen_kwargs behaviour; prompt_logprobs=1 by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1345</li>
<li>Update links to advanced_task_guide.md by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1348</li>
<li><code>Filter</code> docs not offset by <code>doc_id</code> by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1349</li>
<li>Add FAQ on <code>lm_eval.tasks.initialize_tasks()</code> to README by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1330</li>
<li>Refix issue regarding stderr by @thnkinbtfly in https://github.com/EleutherAI/lm-evaluation-harness/pull/1357</li>
<li>Add causalLM OpenVino models by @NoushNabi in https://github.com/EleutherAI/lm-evaluation-harness/pull/1290</li>
<li>Apply some best practices and guideline recommendations to code by @LSinev in https://github.com/EleutherAI/lm-evaluation-harness/pull/1363</li>
<li>serialize callable functions in config by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1367</li>
<li>delay filter init; remove <code>*args</code> by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/1369</li>
<li>Fix unintuitive <code>--gen_kwargs</code> behavior by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1329</li>
<li>Publish to pypi by @anjor in https://github.com/EleutherAI/lm-evaluation-harness/pull/1194</li>
<li>Make dependencies compatible with PyPI by @haileyschoelkopf in https://github.com/EleutherAI/lm-evaluation-harness/pull/1378</li>
</ul>
<h2>New Contributors</h2>
<ul>
<li>@shiweijiezero made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1097</li>
<li>@h-albert-lee made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1089</li>
<li>@Momo-Tori made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1118</li>
<li>@xTayEx made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1128</li>
<li>@NanoCode012 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1124</li>
<li>@MorishT made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1122</li>
<li>@lennijusten made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1136</li>
<li>@veekaybee made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1141</li>
<li>@wiskojo made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1087</li>
<li>@polm-stability made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1147</li>
<li>@seungduk-yanolja made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1171</li>
<li>@Sparkier made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/990</li>
<li>@anjor made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/1184</li>
<li>@zachschillaci27 made their first contribution in htt
Screening process of prohibited section of hazardous materials based on GA-LM-NN.
No full textScreening process of prohibited section of hazardous materials based on GA-LM-NN.</p
Angular dependence of tunneling magneto resistance in La0.7Sr0.3MnO3 step-edge junctions
No full textFlux pinning in superconducting YBCO thin films with an artificial magnetic pinning lattice
No full text
Reduction of acute respiratory illness (ARI) due to a voluntary workplace influenza vaccination program: Who are more likely to get the benefit?
No full textNippobodes peniculatus Chen & Wang, 2007, sp. nov.
No full textNippobodes peniculatus sp. nov. (Figs. 23–26, 38– 39) Material examined: Holotype: adult (in alcohol, W- 90 - 65), CHINA: Fujian Province: Jiangle County, Longxi Mt. (26.5°N, 117.2°E), Shejiaping, 11 Sept., 1990, Hui-Fu Wang, from litter under forest. Paratypes: 1 adult (in alcohol, W- 90 - 64), CHINA: Fujian Province: Jiangle County, Longxi Mt. (26.5°N, 117.2°E), Shejiaping, 11 Sept., 1990, Hui-Fu Wang, from moss; 1 adult (in alcohol, W- 90 - 87), CHINA: Fujian Province: Jiangle County, Longxi Mt. (26.5°N, 117.2°E), Shaxizi, 19 Sept., 1990, Hui-Fu Wang, from litter under bamboo; 1 adults (in alcohol, W- 91 - 6), CHINA: Fujian Province: Jiangle County, Longxi Mt. (26.5°N, 117.2°E), 26 June, 1991, Xiao-Mei Zhang; 1 adult (in alcohol, W- 91 - 11), CHINA: Fujian Province: Jiangle County, Longxi Mt. (26.5°N, 117.2°E), 18–20 May, 1991, Hong Liu; 1 adult (in alcohol, W- 89 - 68), CHINA: Fujian Province: Liancheng County, Quxi (25.6°N, 116.9°E), Meihua Mt., 24 May, 1989, Yun-Qi Cui, from litter; 1 adult (in alcohol, Li- 95 - 1), CHINA: Anhui Province: Xiuning County, Qiyun Mt. (29.8°N, 118.0°E), 23 Oct., 1995, Qing-Tian Li. Etymology. The specific epithet “ peniculatus ” is from the Latin for “tufted tip of a hair” and refers to the shape of most dorsal setae of this species. Diagnosis. Adult. Total length 595–730 Μm. Setae ro thin and long, smooth; setae le and in smooth proximally, strongly barbed distally. Tutorium pointed distally. Humeral condyles covering posterior part of corresponding prodorsal condyles. Ten pairs of notogastral setae present, thick; setae c flagellate, smooth and curved, other notogastral setae relatively short, smooth proximally, strongly barbed distally; la and lm inserted almost at same level. Notogaster ornamented with small tubercles. Adult. Measurements: total length 647 Μm (range 595–730), notogastral length 414 Μm (range 355–460), notogastral width 427 Μm (range 380–495). Prodorsum. Rostrum round and blunt, protruding dorsally. Setae ro curved slightly, thin, long, smooth and inserted on small tubercles on prodorsum; curved transverse ridge connecting tubercles; le and in inserted on tubercles at anterior and posterior of lamella, respectively, setae straight, smooth proximally, strongly barbed distally (Fig. 23); ro length 180 Μm, distance between ro 75 Μm, le length 145 Μm, distance between le 110 Μm, in length 167 Μm, distance between in 257 Μm. Bothridial seta curved strongly, sickle-shaped, attenuate, length 172 Μm. Tutorium well developed, tip sharp. Pair of condyles arise posterolaterally on prodorsum and extend posteriorly. Notogaster. Pair of triangular humeral condyles covering posterior of corresponding prodorsal condyles, with triangular apophysis extending anteromedially from medial margin bearing seta c distally (Fig. 23). Notogaster ornamented with small tubercles. Ten pairs of notogastral setae present, thick; setae c flagellate, smooth, basal half directing anteromedially, distal half curved posteriorly; other notogastral setae relatively short, smooth proximally, strongly barbed distally (Figs. 23, 39); setae la and lm inserted almost at same level (Fig. 23). Setae length: c 175 Μm, la 107 Μm, lm 88 Μm, lp 97 Μm, h 1 90 Μm, h 2 90 Μm, h 3 77 Μm, p 1 52 Μm, p 2 87 Μm, p 3 94 Μm. Ve nt e r. Setae ad 3 inserted almost at same level as anterior margin of anal opening, ad 2 inserted anterior of level of posterior margin of anal opening, ad 1 inserted posterior of posterior margin of anal opening (Fig. 24). Legs. As for family. Immatures. Unknown. Distribution. CHINA: Fujian, Anhui. Remarks. This species is easily distinguished from other known species of this genus by: prodorsal and notogastral setae other than ro, bo and c smooth proximally, strongly barbed distally; setae c flagellate, smooth and curved; notogaster ornamented with small tubercles.Published as part of Chen, Jun & Wang, Hui-Fu, 2007, Taxonomic study on the family Nippobodidae (Acari: Oribatida) from China, pp. 45-63 in Zootaxa 1464 on pages 55-57, DOI: 10.5281/zenodo.17650
Markets Equilibrium: The Is-Lm Model
Full text link. The purpose of this study is to analyze how the concept of markets equilibrium: the IS-LM Model. This research uses library research method by using reference sources from books and journals according to the theme. The author uses a qualitative method which is explained graphically, namely the market balance of the IS-LM model where the focus is on money and goods markets associated with macroeconomics where researchers take the side of investors. The results of this study are that the balance in the economy is the point where the IS and LM curves intersect. This point provides an interest rate (r) and income level (Y) that satisfies the equilibrium conditions that occur in the goods market and money market. In other words, planned spending equals actual spending, and the demand for real money balances equals the supply. So that the IS-LM balance, it is stated that IS=LM
- …
