Textbook of AI

Also known as: NLP

Natural Language Processing (NLP) is the field concerned with enabling computers to process, understand, and generate human language. NLP spans a wide range of tasks: classification (sentiment analysis, spam detection), sequence labelling (named entity recognition, part-of-speech tagging), sequence-to-sequence (translation, summarisation), question answering, dialogue, and text generation. Its applications pervade modern computing—search engines, virtual assistants, translation services, content moderation, chatbots, and countless enterprise systems.

NLP has undergone several paradigm shifts. The earliest era relied on hand-crafted rules and symbolic representations. The statistical era (1990s–2010s) used feature-engineered models like HMMs, CRFs, and SVMs. The deep learning era began with word embeddings (Word2Vec, GloVe) and recurrent neural networks, then accelerated with the transformer architecture (2017). The current paradigm is dominated by large pretrained language models fine-tuned or prompted for specific tasks: tasks that once required separate, specialised pipelines are now addressed by a single pretrained transformer.

Modern NLP applications include machine translation (Google Translate, DeepL), search engines augmented with generative summaries, code generation (GitHub Copilot, Cursor), content moderation at platform scale, legal and medical document analysis, and conversational AI assistants. The release of capable LLM-based chatbots has brought NLP technology directly to hundreds of millions of end users. Persistent challenges include low-resource languages, truthfulness and hallucination, cultural and demographic fairness, reasoning and common sense, and handling of long contexts.