Summary: A team of researchers predicts artificial intelligence (AI), particularly large language models (LLMs), could redefine social science research.
They believe LLMs, trained on vast amounts of text data, can mimic human responses to aid in extensive and rapid human behavior studies. Traditional data collection methods in social sciences could see a significant shift due to these advancements.
However, researchers warn of potential pitfalls, like the inability of AI to replicate socio-cultural biases, and the need for open-source, transparent AI models for ensuring research equity and quality.
Key Facts:
- LLMs could potentially replace human participants for data collection, as they have already demonstrated the ability to generate realistic survey responses in fields like consumer behavior.
- The use of AI in social sciences opens up novel ways to generate hypotheses that can be subsequently confirmed in human populations.
- While LLMs offer vast potential, they often exclude the socio-cultural biases that exist in real human populations, presenting a significant challenge for researchers studying these biases.
Source: University of Waterloo
In an article published yesterday in the prestigious journal Science, leading researchers from the University of Waterloo, University of Toronto, Yale University and the University of Pennsylvania look at how AI (large language models or LLMs in particular) could change the nature of their work.
“What we wanted to explore in this article is how social science research practices can be adapted, even reinvented, to harness the power of AI,” said Igor Grossmann, professor of psychology at Waterloo.
Grossmann and colleagues note that large language models trained on vast amounts of text data are increasingly capable of simulating human-like responses and behaviours. This offers novel opportunities for testing theories and hypotheses about human behaviour at great scale and speed.
Traditionally, social sciences rely on a range of methods, including questionnaires, behavioral tests, observational studies, and experiments.
A common goal in social science research is to obtain a generalized representation of characteristics of individuals, groups, cultures, and their dynamics. With the advent of advanced AI systems, the landscape of data collection in social sciences may shift.
“AI models can represent a vast array of human experiences and perspectives, possibly giving them a higher degree of freedom to generate diverse responses than conventional human participant methods, which can help to reduce generalizability concerns in research,” said Grossmann.
“LLMs might supplant human participants for data collection,” said UPenn psychology professor Philip Tetlock.
“In fact, LLMs have already demonstrated their ability to generate realistic survey responses concerning consumer behaviour. Large language models will revolutionize human-based forecasting in the next 3 years.
“It won’t make sense for humans unassisted by AIs to venture probabilistic judgments in serious policy debates. I put an 90% chance on that. Of course, how humans react to all of that is another matter.”
While opinions on the feasibility of this application of advanced AI systems vary, studies using simulated participants could be used to generate novel hypotheses that could then be confirmed in human populations.
But the researchers warn of some of the possible pitfalls in this approach – including the fact that LLMs are often trained to exclude socio-cultural biases that exist for real-life humans. This means that sociologists using AI in this way couldn’t study those biases.
Professor Dawn Parker, a co-author on the article from the University of Waterloo, notes that researchers will need to establish guidelines for the governance of LLMs in research.
“Pragmatic concerns with data quality, fairness, and equity of access to the powerful AI systems will be substantial,” Parker said.
“So, we must ensure that social science LLMs, like all scientific models, are open-source, meaning that their algorithms and ideally data are available to all to scrutinize, test, and modify.
“Only by maintaining transparency and replicability can we ensure that AI-assisted social science research truly contributes to our understanding of human experience.”
About this AI research news
Author: Ryon Jones
Source: University of Waterloo
Contact: Ryon Jones – University of Waterloo
Image: The image is credited to Neuroscience News
Original Research: Open access.
“AI and the transformation of social science research” by Igor Grossmann et al. Science
Abstract
AI and the transformation of social science research
Advances in artificial intelligence (AI), particularly large language models (LLMs), are substantially affecting social science research.
These transformer-based machine-learning models pretrained on vast amounts of text data are increasingly capable of simulating human-like responses and behaviors, offering opportunities to test theories and hypotheses about human behavior at great scale and speed.
This presents urgent challenges: How can social science research practices be adapted, even reinvented, to harness the power of foundational AI? And how can this be done while ensuring transparent and replicable research?
