Textbook of AI

The GPT-3 launch on 11 June 2020 was OpenAI's release of the 175-billion-parameter GPT-3 language model through a closed-beta API. Trained on a curated mixture dominated by Common Crawl, WebText2, books and Wikipedia (~570 GB after filtering, ~300 billion training tokens), GPT-3 was at the time roughly an order of magnitude larger than its predecessor GPT-2 (1.5B parameters, February 2019) and set in motion most of the trajectory of large language models since.

What was new

The launch was unusual along several dimensions:

• Frontier-scale commercial deployment. It was the first commercial deployment of a frontier-scale language model, with the "wait-list" API access becoming a coveted developer credential through 2020 and 2021.
• API-only access. OpenAI declined to release the weights, an explicit shift from GPT-2 (whose staged-release controversy presaged the policy), and offered access only through a metered API. This marked the beginning of OpenAI's transition from a non-profit research organisation (its 2015 founding posture) to a capped-profit commercial frontier-AI company, formalised through the OpenAI LP structure created in 2019.
• Few-shot in-context learning. The accompanying paper Language Models are Few-Shot Learners (Brown et al., 2020) demonstrated that GPT-3 could perform new tasks given only a handful of examples in its prompt, with no gradient updates. This emergent capability, attributed to the combination of scale and the diverse training corpus, transformed the field's understanding of what was possible at sufficient parameter counts.

Knock-on effects

The release initiated several lasting trends. The modern LLM API economy, now also represented by Anthropic (Claude), Cohere, Google (Gemini), AI21 (Jurassic), Mistral, Meta (Llama, originally weights-out), and others, descends directly from the GPT-3 commercial pattern. The practice of accessing frontier models via API rather than running them locally became the default for most application developers. The prompting-rather-than-fine-tuning paradigm that has dominated since, write a careful prompt, possibly with a handful of in-context demonstrations, rather than fine-tuning task-specific weights, was crystallised by GPT-3's few-shot results.

The OpenAI–Microsoft exclusive licence, signed in September 2020, gave Microsoft preferential commercial access to GPT-3 and presaged the much deeper Microsoft–OpenAI relationship that crystallised from January 2023 onwards (Microsoft's $10B+ investment, Azure OpenAI Service, integration into Bing, Office 365 Copilot and the Windows shell).

Lineage

GPT-3 was soon joined by InstructGPT (Ouyang et al., early 2022), a fine-tuned version aligned to user intent via Reinforcement Learning from Human Feedback (RLHF), which became the basis of the much-larger-impact ChatGPT release on 30 November 2022. The conceptual line GPT → GPT-2 → GPT-3 → InstructGPT → ChatGPT → GPT-4 → o1 traces the dominant arc of public-facing AI from 2018 to the mid-2020s, but the GPT-3 launch is generally taken as the moment the frontier-LLM era became commercially real.