A research agenda for assessing the economic impacts of code generation models

OpenAI is growing a analysis program to evaluate the financial impacts of code technology fashions and is inviting collaboration with exterior researchers. Speedy advances within the capabilities of enormous language fashions (LLMs) skilled on code have made it more and more necessary to check their financial impacts on people, corporations, and society. Codex – an LLM developed by OpenAI by fine-tuning GPT-3 on billions of strains of publicly out there code from GitHub – has been proven to generate functionally appropriate code 28.8% of the time on a pattern of analysis issues (Chen et al. 2021). This will have necessary implications for the way forward for coding and the economics of the industries that rely on it. On this doc, we lay out a analysis agenda to evaluate the results of Codex on financial elements of curiosity to policymakers, corporations, and the general public. We make a case for this analysis agenda by highlighting the possibly broad applicability of code technology fashions to software program growth, the potential for different LLMs to create vital social and financial influence as mannequin capabilities advance, and the worth of utilizing Codex to generate proof and set up methodologies which may be relevant to analysis on the financial impacts of future fashions. We suggest that tutorial and coverage analysis concentrate on learning code technology fashions and different LLMs in order that proof on their financial impacts can be utilized to tell decision-making in three key areas: Deployment coverage, AI system design, and public coverage. To assist information this analysis, we define six precedence consequence areas throughout the realm of financial impacts that we intend to make use of Codex to check: Productiveness, Employment, Ability Improvement, Inter-firm Competitors, Client Costs, and Financial Inequality. For every space, we briefly talk about earlier literature on the impacts of synthetic intelligence on every of those outcomes, describe questions that we consider to be key inputs to the three decision-making areas talked about above, and supply examples of analysis that may very well be performed with Codex. To catalyze work that builds off of this preliminary analysis agenda, we’re saying a Call for Expressions of Interest from exterior researchers to collaborate with OpenAI researchers and prospects to raised measure the financial impacts of code technology fashions and different LLMs.