A recent study conducted by the International Labour Organization (ILO) has delved into the use of artificial intelligence (AI) in evaluating the prestige and social value of various job occupations, shedding light on both its potential benefits and challenges.
The research, titled “A Technological Construction of Society: Comparing GPT-4 and Human Respondents for Occupational Evaluation in the UK,” compared AI assessments performed by GPT-4, a Large Language Model (LLM), with evaluations from human respondents in the United Kingdom.
Comparing AI and human assessments
In this groundbreaking research, human respondents in the UK were initially asked to rank the prestige and social value of different occupations. Subsequently, GPT-4 was tasked with providing a similar ranking, simulating the perspectives of 100 random respondents, each representing an “average UK profile.”
The study aimed to gauge how closely the AI system’s evaluations aligned with those of human participants and whether the AI exhibited any demographic biases.
Notably, the study found a significant correlation between the results obtained through these two distinct approaches. GPT-4 demonstrated a strong proficiency in predicting the average UK views regarding the prestige and social value of individual occupations, effectively creating relative occupational rankings.
This ability to understand general human opinions algorithmically holds the potential for AI to contribute to occupational research, offering advantages such as increased efficiency, cost-effectiveness, speed, and enhanced accuracy in capturing societal perceptions.
AI’s complex relationship with occupational prestige
While the study highlighted the promise of AI in occupational research, it also uncovered several challenges. GPT-4 tended to overestimate the prestige and value of occupations associated with the digital economy or those involving robust marketing and sales components.
Conversely, it underestimated the prestige and social value attributed to certain illicit or traditionally stigmatized occupations compared to human evaluators. Moreover, when the AI’s algorithmic instructions were manipulated, it struggled to grasp the hierarchies of prestige and social value of occupations as perceived by demographic minorities in the UK context.
The significance of demographic biases
One crucial aspect emphasized in the research is the presence of demographic biases in AI models. Current Large Language Models, including GPT-4, primarily reflect the opinions of Western, educated, industrialized, rich, and demographic (WEIRD) populations.
These groups, while being a global minority, have significantly contributed to the data used for training AI models. As a result, AI systems like GPT-4 might inadvertently exclude the perspectives of demographic minorities or vulnerable groups.
Implications for AI in the world of work
The findings of this study have far-reaching implications for the application of AI systems in the world of work. While AI can serve as a valuable complementary tool, particularly in processing vast amounts of unstructured text, voice, and image data, its limitations must be carefully considered.
This includes its potential to omit the views and perspectives of demographic minorities. In contexts such as providing career advice or conducting algorithmic performance evaluations, the study underscores the importance of addressing these limitations and ensuring fairness and inclusivity.
