9 research outputs found
The Quadratic Optimization Bias Of Large Covariance Matrices
No full textWe describe a puzzle involving the interactions between an optimization of a multivariate quadratic function and a plug-in estimator of a spiked covariance matrix. When the largest eigenvalues (i.e., the spikes) diverge with the dimension, the gap between the true and the out-of-sample optima typically also diverges. We show how to fine-tune the plug-in estimator in a precise way to avoid this outcome. Central to our description is a quadratic optimization bias function, the roots of which determine this fine-tuning property. We derive an estimator of this root from a finite number of observations of a high dimensional vector. This leads to a new covariance estimator designed specifically for applications involving quadratic optimization. Our theoretical results have further implications for improving low dimensional representations of data, and principal component analysis in particular.40 pages, 6 figures, 5 table
Two-loop Sudakov form factor in ABJM
No full textThis article is distributed under the terms of the Creative Commons Attribution License (CC-BY 4.0), which permits any use, distribution and reproduction in any medium, provided the original author(s) and source are credited
The last of the simple remainders
No full textThis article is distributed under the terms of the Creative Commons
Attribution License (CC-BY 4.0), which permits any use, distribution and reproduction in
any medium, provided the original author(s) and source are credited
On super form factors of half-BPS operators in N=4 super Yang-Mills
No full textOpen Access, (c) The Authors. Article funded by SCOAP3. This article is distributed under the terms of the Creative Commons Attribution License (CC-BY 4.0), which permits any use, distribution and reproduction in any medium, provided the original author(s) and source are credited
Humanity's Last Exam
No full textBenchmarks are important tools for tracking the rapid advancements in large
language model (LLM) capabilities. However, benchmarks are not keeping pace in
difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like
MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In
response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at
the frontier of human knowledge, designed to be the final closed-ended academic
benchmark of its kind with broad subject coverage. HLE consists of 2,700
questions across dozens of subjects, including mathematics, humanities, and the
natural sciences. HLE is developed globally by subject-matter experts and
consists of multiple-choice and short-answer questions suitable for automated
grading. Each question has a known solution that is unambiguous and easily
verifiable, but cannot be quickly answered via internet retrieval.
State-of-the-art LLMs demonstrate low accuracy and calibration on HLE,
highlighting a significant gap between current LLM capabilities and the expert
human frontier on closed-ended academic questions. To inform research and
policymaking upon a clear understanding of model capabilities, we publicly
release HLE at https://lastexam.ai
Recommended from our members
Humanity's Last Exam
No full textBenchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai
Humanity's Last Exam
No full textBenchmarks are important tools for tracking the rapid advancements in large
language model (LLM) capabilities. However, benchmarks are not keeping pace in
difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like
MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In
response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at
the frontier of human knowledge, designed to be the final closed-ended academic
benchmark of its kind with broad subject coverage. HLE consists of 2,500
questions across dozens of subjects, including mathematics, humanities, and the
natural sciences. HLE is developed globally by subject-matter experts and
consists of multiple-choice and short-answer questions suitable for automated
grading. Each question has a known solution that is unambiguous and easily
verifiable, but cannot be quickly answered via internet retrieval.
State-of-the-art LLMs demonstrate low accuracy and calibration on HLE,
highlighting a significant gap between current LLM capabilities and the expert
human frontier on closed-ended academic questions. To inform research and
policymaking upon a clear understanding of model capabilities, we publicly
release HLE at https://lastexam.ai
A benchmark of expert-level academic questions to assess AI capabilities
No full textInternational audienceBenchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve more than 90% accuracy on popular benchmarks such as Measuring Massive Multitask Language Understanding1, limiting informed measurement of state-of-the-art LLM capabilities. Here, in response, we introduce Humanity’s Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be an expert-level closed-ended academic benchmark with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable but cannot be quickly answered by internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a marked gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai
