33 research outputs found
Evaluating the persuasive influence of political microtargeting with large language models
No full textThe recent development of large language models (LLMs) has raised the prospect of scalable, automated, and fine-grained microtargeting on a scale previously unseen; however, the persuasive influence of microtargeting with LLMs remains unclear. In this study, we investigate the extent to which micro-targeted messages generated by GPT-4, the current most powerful publicly accessible LLM, can influence political attitudes. In a pre-registered experiment (n = 8,600), we employ a novel experimental design to integrate self-reported demographic and political data into GPT-4 prompts in real-time, facilitating the live creation of thousands of unique messages tailored to persuade individual participants on four political issues
Evaluating the persuasive influence of political microtargeting with large language models
No full textThe recent development of large language models (LLMs) has raised the prospect of scalable, automated, and fine-grained microtargeting on a scale previously unseen; however, the persuasive influence of microtargeting with LLMs remains unclear. In this study, we investigate the extent to which micro-targeted messages generated by GPT-4, the current most powerful publicly accessible LLM, can influence political attitudes. In a pre-registered experiment (n = 8,600), we employ a novel experimental design to integrate self-reported demographic and political data into GPT-4 prompts in real-time, facilitating the live creation of thousands of unique messages tailored to persuade individual participants on four political issues
Quantifying the persuasive influence of political microtargeting with large language models
No full textThe recent development of large language models (LLMs) has raised the prospect of scalable, automated, and fine-grained microtargeting on a scale previously unseen; however, the persuasive influence of microtargeting with LLMs remains unclear. In this study, we investigate the extent to which micro-targeted messages generated by GPT-4, the current most powerful publicly accessible LLM, can influence political attitudes. In a pre-registered experiment (n = 8,600), we employ a novel experimental design to integrate self-reported demographic and political data into GPT-4 prompts in real-time, facilitating the live creation of thousands of unique messages tailored to persuade individual participants on four political issues
Can coached humans match AI at conversational persuasion?
No full textStudy 1 of this research program found that frontier LLMs are substantially more persuasive than elite
human debaters in text-based conversations about political issues. This raises an important question: Is
AI’s persuasive advantage driven by superior rhetorical abilities, or simply by humans’ lack of awareness
about effective persuasion strategies?
Study 2 directly tests this hypothesis by coaching elite debaters on effective persuasion strategies (particularly
information-based persuasion), providing them with results from Study 1, and giving them opportunities to
learn from AI-generated conversations.
Additionally, Study 2 investigates whether AI’s advantage stems from “technical” capabilities—specifically,
the ability to generate longer messages faster than humans. By constraining AI to produce human-length
messages with human-like response delays, we can isolate whether the persuasive advantage persists even
when these technical advantages are removed
Comparing the persuasiveness of role-playing large language models and human experts on polarized U.S. political issues
No full textLarge language models (LLMs) can generate persuasive political messages, raising concerns about their potential to influence political attitudes. Here, we investigate how a prompting technique known as “role- playing” — in which the model is instructed to generate text that adopts a particular perspective or persona — can enhance the persuasive power of the current most powerful publicly accessible LLM, GPT-4. Through an experiment conducted on a balanced sample of Americans (N = 5,000), we assess the persuasive potential of content generated by a role-playing LLM across a range of political issues and compare it to that of human experts. Our findings will reveal the extent to which partisan role-playing affects the persuasive power of LLMs and how their persuasiveness compares to a human baseline of political messaging experts. This work has important implications for regulating LLMs with regard to their potential use for influence operations and online deception in political contexts
Comparing the persuasiveness of role-playing large language models and human experts on polarized U.S. political issues
No full textLarge language models (LLMs) can generate persuasive political messages, raising concerns about their potential to influence political attitudes. Here, we investigate how a prompting technique known as “role- playing” — in which the model is instructed to generate text that adopts a particular perspective or persona — can enhance the persuasive power of the current most powerful publicly accessible LLM, GPT-4. Through an experiment conducted on a balanced sample of Americans (N = 5,000), we assess the persuasive potential of content generated by a role-playing LLM across a range of political issues and compare it to that of human experts. Our findings will reveal the extent to which partisan role-playing affects the persuasive power of LLMs and how their persuasiveness compares to a human baseline of political messaging experts. This work has important implications for regulating LLMs with regard to their potential use for influence operations and online deception in political contexts
Partisan role-play and the persuasive power of large language models on polarized political issues
No full textLarge language models (LLMs) can generate persuasive political messages, raising concerns about their potential to influence political attitudes. Here, we investigate how a prompting technique known as “role- playing” — in which the model is instructed to generate text that adopts a particular perspective or persona — can enhance the persuasive power of the current most powerful publicly accessible LLM, GPT-4. Through an experiment conducted on a balanced sample of Americans (N = 5,000), we assess the persuasive potential of content generated by a role-playing LLM across a range of political issues and compare it to that of human experts. Our findings will reveal the extent to which partisan role-playing affects the persuasive power of LLMs and how their persuasiveness compares to a human baseline of political messaging experts. This work has important implications for regulating LLMs with regard to their potential use for influence operations and online deception in political contexts
Evaluating the persuasive influence of political microtargeting with large language models
No full textRecent advancements in large language models (LLMs) have raised the prospect of scalable, automated, and fine-grained political microtargeting on a scale previously unseen; however, the persuasive influence of microtargeting with LLMs remains unclear. Here, we build a custom web application capable of integrating self-reported demographic and political data into GPT-4 prompts in real-time, facilitating the live creation of unique messages tailored to persuade individual users on four political issues. We then deploy this application in a pre-registered randomized control experiment (n = 8,587) to investigate the extent to which access to individual-level data increases the persuasive influence of GPT-4. Our approach yields two key findings. First, messages generated by GPT-4 were broadly persuasive, in some cases increasing levels of support for an issue stance by nearly 50%. Second, in aggregate, the persuasive impact of microtargeted messages was not statistically different from that of non-microtargeted messages (5.68% vs 7.32%, respec- tively, P = 0.082). These trends hold even when manipulating the type and number of attributes used to tailor the message. Taken together, these findings suggest — contrary to widespread speculation — that the influence of current LLMs may reside not in their ability to tailor messages to individuals, but rather in the persuasiveness of their generic, non-targeted messages. This work secondarily contributes by offering a robust and replicable approach – through a custom web-based pipeline – to integrating LLMs into experimental designs, and a novel dataset, GPTarget2023, containing metadata for thousands of tailored AI-generated messages
Scaling laws for political persuasion with large language models
No full textRapid advances in the persuasive capabilities of large language models (LLMs) have raised concerns over their (mis)use for widespread political attitude change; however, it remains unclear how the persuasive capabilities of LLMs scale with expansions in model architecture and training scope. In a pre-registered experiment (N ≈ 25, 000) we test the persuasiveness of a range of instruction-tuned models spanning several orders of magnitude in model parameters
Mapping moral language on US presidential primary campaigns reveals rhetorical networks of political division and unity
No full textDuring political campaigns, candidates use rhetoric to advance competing visions and assessments of their country. Research reveals that the moral language used in this rhetoric can significantly influence citizens’ political attitudes and behaviors; however, the moral language actually used in the rhetoric of elites during political campaigns remains understudied. Using a data set of every tweet (N=139,412) published by 39 US presidential candidates during the 2016 and 2020 primary elections, we extracted moral language and constructed network models illustrating how candidates’ rhetoric is semantically connected. These network models yielded two key discoveries. First, we find that party affiliation clusters can be reconstructed solely based on the moral words used in candidates’ rhetoric. Within each party, popular moral values are expressed in highly similar ways, with Democrats emphasizing careful and just treatment of individuals and Republicans emphasizing in-group loyalty and respect for social hierarchies. Second, we illustrate the ways in which outsider candidates like Donald Trump can separate themselves during primaries by using moral rhetoric that differs from their parties’ common language. Our findings demonstrate the functional use of strategic moral rhetoric in a campaign context and show that unique methods of text network analysis are broadly applicable to the study of campaigns and social movements
