Seminar: STEER: Assessing the Economic Rationality of Large Language Models

Speaker:  Dr. Kevin Leyton-Brown | Professor Computer Science | University of British Columbia

Date: April, 11
Time: 2 pm
Room: 2-104 (Dr. Alvin Woods Building) & Hybrid

Summary:  There is increasing interest in using LLMs as decision-making "agents." Doing so
includes many degrees of freedom: which model should be used; how should it be prompted;
should it be asked to introspect, conduct chain-of-thought reasoning, etc? Settling these
questions -- and more broadly, determining whether an LLM agent is reliable enough to be
trusted -- requires a methodology for assessing such an agent's economic rationality. This talk
describes one. We survey the economic literature on both strategic and non-strategic decision
making, taxonomizing 124 fine-grained "elements" that an agent should exhibit, each of which
can be tested in up to 3 distinct ways, grounded in up to 10 distinct domains, and phrased
according to 5 perspectives (first-person, second-person, etc). The generation of benchmark
data across this combinatorial space is powered by a novel LLM-assisted data generation
protocol that we dub auto-STEER, which generates questions by adapting handcrafted
templates to new domains and perspectives. Because it offers an automated way of generating
fresh questions, auto-STEER mitigates the risk that LLMs will be trained to overfit evaluation
benchmarks; we thus hope that it will serve as a useful tool both for evaluating and fine-tuning
models for years to come. Finally, we describe the results of a large-scale empirical experiment
with 28 different LLMs, ranging from small open-source models to the current state of the art.
We examined each model's ability to solve problems across our whole taxonomy and present
the results across a range of prompting strategies and scoring metrics

Biography: Kevin Leyton-Brown, Distinguished University Scholar and Professor Computer Science and at the University of British Columbia, holds a Canada CIFAR AI Chair at the Alberta Machine Intelligence Institute, is an associate member of the Vancouver School of Economics. He is a Fellow of the Royal Society of Canada (RSC; awarded in 2023), the Association for Computing Machinery (ACM; awarded in 2020), and the Association for the Advancement of Artificial Intelligence (AAAI; awarded in 2018). He was a member of a team that won the 2018 INFORMS Franz Edelman Award for Achievement in Advanced Analytics, Operations Research and Management Science, described as "the leading O.R. and analytics award in the industry." He holds a PhD and M.Sc. from Stanford University (2003; 2001) and a B.Sc. from McMaster University (1998). He studies artificial intelligence and machine learning with a focus on connections both to microeconomic theory and to the design of algorithms for hard combinatorial problems. He is the Director of UBC's Center for AI Decision-making and Action and has been a visiting professor at Harvard, Berkley, Stanford, and Microsoft Research New York, and in several countries including Israel. He has received multiple research awards including from Amazon, Facebook and Google.