Towards Measuring the Representation of Subjective Global Opinions in Language Models
Whose opinions are the responses of Large Language Models (LLMs) most similar with when considering the perspectives of participants across the world?
We develop a quantitative framework to understand how similar the responses of models and different participants across the world are in order to help guide the development of inclusive AI that serves all people worldwide. We first build a dataset, GlobalOpinionQA, composed of questions and answers from cross-national surveys designed to capture diverse opinions on global issues. Next, we derive a metric that captures the similarity between LLM responses and people’s responses, conditioned on country. With our framework, we run three experiments on an LLM trained to be helpful, honest, and harmless with Constitutional AI.
Default Prompting
Cross National Prompting
Linguistic Prompting
Default Prompting
Experimental Setup. We ask the model World Values Survey (WVS) and Pew Research Center’s Global Attitudes (GAS) multiple-choice survey questions as they were originally written. The goal of the default prompt is to measure the intrinsic opinions reflected by the model, relative to people's aggregate responses from a country. We hypothesize that responses to the default prompt may reveal biases and challenges models may have at representing diverse views.
Model responses are most similar to the opinion distributions of the USA and Canada, some of the European and South American countries.
100