9607 research outputs found
Sort by
RiddleBench:A New Generative Reasoning Benchmark for LLMs
No full textLarge Language Models have demonstrated strong performance on many established reasoning benchmarks. However, these benchmarks primarily evaluate structured skills like quantitative problem-solving, leaving a gap in assessing flexible, multifaceted reasoning abilities that are central to human intelligence. These abilities require integrating logical deduction with spatial awareness and constraint satisfaction, which current evaluations do not measure well. To address this, we introduce RiddleBench, a benchmark of 1,737 challenging puzzles in English designed to probe these core reasoning capabilities. Evaluation of state-of-the-art models on RiddleBench shows fundamental weaknesses. Even top proprietary models like Gemini 2.5 Pro, o3, and Claude 4 Sonnet achieve accuracy just above 60% (60.30%, 63.37%, and 63.16%). Analysis further reveals deep failures, including hallucination cascades (accepting flawed reasoning from other models) and poor self-correction due to a strong self-confirmation bias. Their reasoning is also fragile, with performance degrading significantly when constraints are reordered or irrelevant information is introduced. RiddleBench functions as a diagnostic tool for these issues and as a resource for guiding the development of more robust and reliable language models
On the Notion that Language Models Reason
No full textLanguage models (LMs) are said to be exhibiting reasoning, but what does this entail? We assess definitions of reasoning and how key papers in the field of natural language processing (NLP) use the notion and argue that the definitions provided are not consistent with how LMs are trained, process information, and generate new tokens. To illustrate this incommensurability we assume the view that transformer-based LMs implement an implicit finite-order Markov kernel mapping contexts to conditional token distributions. In this view, reasoning-like outputs correspond to statistical regularities and approximate statistical invariances in the learned kernel rather than the implementation of explicit logical mechanisms. This view is illustrative of the claim that LMs are "statistical pattern matchers"" and not genuine reasoners and provides a perspective that clarifies why reasoning-like outputs arise in LMs without any guarantees of logical consistency. This distinction is fundamental to how epistemic uncertainty is evaluated in LMs. We invite a discussion on the importance of how the computational processes of the systems we build and analyze in NLP research are described
Superpolynomial Lower Bounds Against Low-Depth Algebraic Circuits
No full textAn Algebraic Circuit for a polynomial is a computational model for constructing the polynomial P using only additions and multiplications. It is a syntactic model of computation, as opposed to the Boolean Circuit model, and hence lower bounds for this model are widely expected to be easier to prove than lower bounds for Boolean circuits. Despite this, we do not have superpolynomial lower bounds against general algebraic circuits of depth 3 (except over constant-sized finite fields) and depth 4 (over fields other than ), while constant-depth Boolean circuit lower bounds have been known since the early 1980s. In this article, we prove the first superpolynomial lower bounds against general algebraic circuits of all constant depths over all fields of characteristic 0 (or large). We also prove the first lower bounds against homogeneous algebraic circuits of constant depth over any field. Our approach is surprisingly simple. We first prove superpolynomial lower bounds for constant-depth Set-Multilinear circuits. While strong lower bounds were already known against such circuits, most previous lower bounds were of the form f(d) ⋅ poly( N ), where d denotes the degree of the polynomial. In analogy with Parameterized complexity, we call this an FPT lower bound. We extend a well-known technique of Nisan and Wigderson (FOCS 1995) to prove non-FPT lower bounds against constant-depth set-multilinear circuits computing the Iterated Matrix Multiplication polynomial IMM n,d (which computes a fixed entry of the product of d n × n matrices). More precisely, we prove that any set-multilinear circuit of depth Δ computing IMM n,d must have size at least . This result holds over any field, as long as d = o (log n ). We then show how to convert any constant-depth algebraic circuit of size s to a constant-depth set-multilinear circuit with a blow-up in size that is exponential in d but only polynomial in s over fields of characteristic 0. (For depths greater than 3, previous results of this form increased the depth of the resulting circuit to Ω (log s ).) This implies our constant-depth circuit lower bounds. We can also use these lower bounds to prove a Depth Hierarchy theorem for constant-depth circuits. We show that for every depth Γ , there is an explicit polynomial which can be computed by a depth Γ circuit of size s , but requires circuits of size s ω (1) if the depth is Γ -1. Finally, we observe that our superpolynomial lower bound for constant-depth circuits implies the first deterministic sub-exponential time algorithm for solving the Polynomial Identity Testing (PIT) problem for all small depth circuits using the known connection between algebraic hardness and randomness
History-grounded Design Speculation as a Method for AI Impact Anticipation
No full textAs Artificial Intelligence continues to permeate everyday life, concerns over its societal consequences are becoming increasingly pressing. Anticipatory practices have emerged as central to responsible AI development, offering ways to envision and mitigate potential harms. While policymakers engage with anticipation through forecasting and risk assessment, speculative design offers an alternative, more experiential approach to also fosters public engagement and critical reflection. However, most speculative explorations focus on future possibilities, often neglecting the continuum between these and past phenomena. In this pictorial, we argue for integrating historical perspectives into speculative design to enrich anticipatory work on AI. Through a week-long international summer school, we engaged with the legacy of phrenology and the work of Cesare Lombroso. Using this as a springboard for speculation, we illustrate that incorporating historical trajectories into speculative design can deepen understanding of current dilemmas around AI, but dedicated methodological resources are still needed to achieve this value
The beginning of AI-driven welfare? An inquiry into how public sector AI experiments shape the Danish welfare state
No full textThis study investigates the nascent stages of an AI-driven welfare state by focusing on 40 signature projects initiated by the Danish Government between 2020 and 2022. These projects represent a paradigmatic case of AI experimentation within the welfare state. We adopt an empirical approach based on mixed methods to explore the objectives, organizational actors, goals, and outcomes of the projects, shedding light on the evolving landscape of AI-driven welfare from 2020 to 2023. The chapter highlights the variance between welfare domains in the objectives set for AI models and the differences between municipal and regional experimentation with welfare, focusing on how they differ with respect to implementation and research goals. While only some of the projects to date have been operationally implemented, several live on as prototypes or partial systems with the potential to transform the welfare state beyond their formal termination
From Queries to Candidates: Exploring Search and Source Interaction Behavior of Recruiters
No full textRecruitment is a professional search domain that has been largely overlooked in IR research, even though better support of recruiters could have a big impact on job seekers, companies and society as a whole. In this paper, we analyze the search formulation and source selection behavior of the recruiters at one of Scandinavia’s largest job portals and recruitment agencies using search logs for close to 18,000 recruitment search tasks. We provide an analysis of the search sessions of recruiters in terms search tactics, query operators, query length, term re-use and filter usage, and break down their behavior both by task type and task complexity. We also relate their short-term tactics to different learning stages in the search process and investigate their influence on search success. We find that identifying and assessing relevant candidates for a job posting is a complex task: recruiters usually submit multiple queries during sessions that can last for hours and that increase in complexity. Recruiters all spend more time per query as their session progresses. We also observed query reformulation strategies that indicate distinct patterns of knowledge gaining during sessions. Relating these tactics to positive responses from candidates we aim at predicting successful strategies.<br/
Blog - Danish Game Industry Timeline
No full textThis website accompanies the research into the Danish Game Industry Timeline. The timeline collects landmarks and events to present an overview of Danish digital game development. The website enables ongoing communication and revision of timeline in dialog with community. On this website, and on the parallel spiltidslinje.dk (TBA), we update further versions of the timeline. We also credit everyone who contributes and make changes visible
Is it getting harder to make a hit? Evidence from 65 years of US music chart history
No full textAbstract Since the creation of the Billboard Hot 100 music chart in 1958, the chart has been a window into the music consumption of Americans. Since its introduction, the chart has documented music consumption through eras of globalization, economic growth, and the emergence of new technologies for music listening. In recent years, artists have voiced their worry that the music world is changing: Many claim that it is getting harder to make a hit. Until now, however, the claims have not been backed using chart data. Here we show that the dynamics of the Billboard Hot 100 chart have changed significantly since the chart’s founding in 1958, and, in particular, in the past 15 years. Whereas most songs spend less time on the chart now than songs did in the past, we show that top-1 songs have tripled their chart lifetime since the 1960s, and the highest-ranked songs maintain their positions for far longer than previously. At the same time, churn has increased drastically, and the lowest-ranked songs are replaced more frequently than ever. Together, these observations support two competing and seemingly contradictory theories of digital markets: The Winner-takes-all theory and the Long Tail theory. Who occupies the chart has also changed over the years: In recent years, fewer new artists make it into the chart and more positions are occupied by established hit makers. Finally, investigating how song chart trajectories have changed over time, we show that historical song trajectories cluster into clear trajectory archetypes characteristic of the time period they were part of. Our results are interesting in the context of collective attention: Whereas recent studies have documented that other cultural products such as books, news, and movies fade in popularity quicker in recent years, music hits seem to last longer now that in the past.Since the creation of the Billboard Hot 100 music chart in 1958, the chart has been a window into the music consumption of Americans. Since its introduction, the chart has documented music consumption through eras of globalization, economic growth, and the emergence of new technologies for music listening. In recent years, artists have voiced their worry that the music world is changing: Many claim that it is getting harder to make a hit. Until now, however, the claims have not been backed using chart data. Here we show that the dynamics of the Billboard Hot 100 chart have changed significantly since the chart’s founding in 1958, and, in particular, in the past 15 years. Whereas most songs spend less time on the chart now than songs did in the past, we show that top-1 songs have tripled their chart lifetime since the 1960s, and the highest-ranked songs maintain their positions for far longer than previously. At the same time, churn has increased drastically, and the lowest-ranked songs are replaced more frequently than ever. Together, these observations support two competing and seemingly contradictory theories of digital markets: The Winner-takes-all theory and the Long Tail theory. Who occupies the chart has also changed over the years: In recent years, fewer new artists make it into the chart and more positions are occupied by established hit makers. Finally, investigating how song chart trajectories have changed over time, we show that historical song trajectories cluster into clear trajectory archetypes characteristic of the time period they were part of. Our results are interesting in the context of collective attention: Whereas recent studies have documented that other cultural products such as books, news, and movies fade in popularity quicker in recent years, music hits seem to last longer now that in the past
Defining Matters of Compassion: Designing Care Technology within a Care Crisis
No full textAccess to mental health care is increasingly strained, with rising demand and long waiting times leaving many to manage their mental wellbeing alone. This work-in-progress paper responds to this care crisis by exploring how technology can support mental and emotional wellbeing. First, building on the concepts of matters of concern and matters of care, we propose a shift toward matters of compassion as a specific way to enact care that moves away from reductive problem-solving by embracing complexity, vulnerability, and uncertainty. Then, we further untangle the concept of matters of compassion by illustrating design tactics to invite compassionate engagement with experiences where resolution is not possible or desirable, where the experience inevitably brings uncertainty and vulnerability that need to be dealt with. To do so, we examine three design cases—focused on premenstrual disorders (PMDs), pregnancy, and abortion
Mapping the Climate Change Landscape on TikTok
Get PDFSocial media platforms shape climate action discourse. Mapping these online conversations is essential for effective communication strategies. TikTok’s climate discussions are particularly relevant given its young, climate-concerned audience. In this work, we collect the first TikTok dataset on climate topics. We collected 590K videos from 14K creators along with their follower networks. By applying topic modeling to the video descriptions, we map the topics discussed on the platform on a climate taxonomy that we construct by consolidating existing categorizations. Results show TikTok creators primarily approach climate through the angle of lifestyle and dietary choices. By examining semantic connections between topics, we identified non-climate” gateway” topics that could draw new audiences into climate discussions