Encoder-Decoder Model
An Encoder–Decoder architecture is a neural network design in which an encoder maps an input into a latent representation and a separate decoder generates the output sequence from that representation, making it well-suited for tasks like translation and summarization.
An encoder-decoder model is a neural network architecture in which an encoder maps an input into a latent representation and a separate decoder generates the output sequence from that representation, making it well-suited for tasks like translation and summarization.
The transformer (Vaswani et al., 2017) recast the encoder–decoder structure using self-attention and cross-attention. The encoder stack processes the full input sequence in parallel, producing contextualized representations; the decoder stack generates output tokens autoregressively, using causal self-attention over its own past outputs and cross-attention over the encoder's output. This design scales efficiently and underpins models including T5 (Google, 2019), BART (Meta, 2019), and mT5 for multilingual settings.
The architecture is particularly well-suited to tasks where input and output are structurally different: machine translation (source and target languages), abstractive summarization (article to summary), speech recognition (OpenAI's Whisper uses a CNN encoder for audio and a transformer decoder for text), and image captioning (visual encoder, text decoder). The encoder handles understanding; the decoder handles generation, and this separation of concerns is beneficial when the two processes require different attention patterns.
As of 2026, encoder–decoder models remain the standard for structured prediction tasks. T5-style models and their multilingual successors are deployed in Google Search, document translation, and summarization pipelines. However, for open-ended language generation at scale, decoder-only models have largely taken over because they are architecturally simpler to train and scale. Parameter-efficient fine-tuning techniques frequently couple a frozen encoder with a small decoder head for low-resource adaptation.