Textbook of AI

Feature Engineering is the process of designing and constructing informative input features from raw data, guided by domain knowledge. Historically, it has been the aspect of applied machine learning that most rewards domain expertise and creative thinking. A well-chosen feature set can render a simple linear model highly effective, while a poor feature set can doom even the most powerful neural network.

Key techniques include: encoding categorical variables (one-hot, ordinal, target encoding); scaling numerical features (standardisation, min-max, robust scaling) so gradient-based optimisers converge quickly and distance-based algorithms compare features fairly; transformations (log, square root, Box–Cox) to reduce skew; creating interaction terms and polynomial features; binning continuous variables; extracting date and time features (day of week, hour, seasonality); and deriving domain-specific features (technical indicators in finance, TF-IDF in text).

The advent of deep learning has dramatically reduced the need for manual feature engineering in domains with large datasets. Convolutional networks learn hierarchical features from pixels; transformers learn contextual representations from raw text; graph neural networks learn node representations from adjacency structure. This paradigm, called representation learning, shifts the burden from engineer to model. However, feature engineering remains valuable for tabular data, small datasets, and interpretability requirements, and understanding its principles is essential even in the deep learning era.