A Comprehensive Guide to the Decoder Architecture and Calculation Workflow of the Transformer Model

09 Mar, 2026
- AI

This article provides a detailed summary of the overall network architecture of the Decoder part in the Transformer model, along with the data operations and flow processes within it.

Previously, in the article A Comprehensive Guide to the Encoder Architecture and Calculation Workflow of the Transformer Model, we thoroughly organized the Encoder architecture of the Transformer. This article continues by examining and analyzing the Decoder structure and the complete calculation workflow of the Transformer. Since the model structure and calculation workflow of the Decoder are very similar to those of the Encoder, it is best to first review A Comprehensive Guide to the Encoder Architecture and Calculation Workflow of the Transformer Model and How to Understand Multi-Head Attention in the Transformer Architecture? to get a clear grasp of the self-attention probabilities and calculation workflow of the Transformer.

The following diagram illustrates the complete network architecture of the Transformer Encoder-Decoder:

The left side shows the Transformer Encoder, composed of $N$ consecutive Encoder Blocks, which is used to convert input natural language text into vector-represented semantic information.
The right side shows the Transformer Decoder, composed of $N$ consecutive Decoder Blocks, which is used to generate natural language outputs (such as machine translation or text generation) based on the semantic vector representations output by the Encoder.

Decoder Inputs and Outputs

The input and output structures of the Transformer Decoder are shown in the diagram below.

The Decoder input consists of two parts of data:

Lower Input (Outputs Shifted Right): Used to perform Masked Self-Attention calculation. Its content is the sequence of tokens already generated by the model, allowing the model to know “what I have already said.”
Middle Input (Semantic vectors extracted by the Encoder): Used to perform Cross Attention calculation. Its input consists of the $K$ and $V$ matrices derived from the Encoder’s semantic vectors, enabling the model to understand “what the source sentence means.” The calculation logic is that the Decoder uses its own $Q$ (what semantic information I am looking for) to match against the Encoder’s $K$ and $V$.

The output of the Decoder is the “Output Probabilities” located at the top of the above diagram. The dimension of the output content is a vector equal to the vocabulary size of the model. After being processed by the final Linear layer and Softmax layer, this vector represents the probability distribution of each word in the model’s vocabulary appearing as the “next output word.” Typically, the index with the highest probability is selected, and the corresponding word is looked up in the dictionary to serve as the next output word.

Outputs Shifted Right Input and Autoregressive Prediction

As mentioned above, the content of the lower Decoder input (Outputs Shifted Right) is the sequence of tokens already generated by the model. How should this be understood during the inference process?

In the initial state, the model has no output. At this point, the input content of the Outputs Shifted Right part is merely a start-of-sentence signal <SOS> (Start of Sentence). Based on the Encoder’s semantics and the start signal, the model performs a forward inference pass on the Decoder side to predict and generate the first word (e.g., Hello). Next, this newly generated word Hello is appended to the Decoder’s input end. At this moment, the Decoder’s input sequence becomes: [<SOS>, Hello]. The model continues to perform forward inference based on this new input sequence to predict the second word, World. The second word is similarly appended to the input end, making the input sequence [<SOS>, Hello, World], and so on, until the final output generated by the model is

<EOS> (End of Sentence), at which point inference stops.

This is the complete workflow of how the Decoder performs forward inference to predict and generate words one by one. Because every time a word is generated, it is re-appended to the end of the input prediction sequence to infer the next word, this process is known as the autoregressive prediction process.

Structure of the Decoder Block

The Decoder part in the Transformer consists of multiple identical Decoder Blocks. Its primary function is to generate the final output sequence based on the semantic representation provided by the Encoder (such as the output of the Encoder part in the diagram above) and the previously generated output tokens (such as the Outputs Shifted Right part in the diagram above).

Each Decoder Block consists of three main sub-layers:

Masked Multi-Head Attention: Similar to the encoder’s self-attention mechanism, but its primary purpose is to prevent attending to future tokens to preserve the autoregressive property (ensuring no “cheating” occurs during generation).
Multi-Head Attention: This sub-layer allows the decoder to focus on relevant parts of the encoder’s output representation. This enables the decoder to concentrate on pertinent segments of the input, which is crucial for tasks like translation.
Feed Forward: A feed-forward neural network that processes the combined output of the masked self-attention and encoder-decoder attention mechanisms.

In terms of network structure and inference calculation workflow, the Multi-Head Attention module and the Feed Forward module are identical to their counterparts in the Encoder. The main differences lie in the Masked Multi-Head Attention module and the cross-attention mechanism utilizing the Encoder’s output as the Decoder’s input.

Masked Multi-Head Attention

From an engineering implementation perspective, Masked Self-Attention and standard Self-Attention are 90% identical in code structure and calculation workflow. The only difference lies in the field of view (or visibility range) of the two attention mechanisms:

Standard Self-Attention (e.g., used in the Encoder): When this module extracts semantics from a text sequence, it can simultaneously see all words from the beginning to the end. Therefore, it can understand each word based on the context of the entire sentence and extract its semantics. Thus, the field of view of the standard Self-Attention module is global.
Masked Self-Attention (used in the Decoder): When using the masked self-attention module, the output at the current word position can only reference words that have already been generated previously; it cannot reference words that appear after the current word position. In other words, the field of view of the masked self-attention module is historical—it can only see preceding words.

The historical viewpoint maintained in the calculation of Masked Self-Attention is crucial for the training process.

During the training phase, training samples contain complete, correct answer sequences for a given problem. To improve efficiency, the training process does not run the model $N$ times word by word like inference decoding; instead, the entire sequence is fed in all at once, and a single matrix operation is executed to obtain the answer sequence through one forward pass. If a standard self-attention module were used in this scenario, a problem would arise: during text generation by the decoder, preceding words would be able to see subsequent words. Consequently, words further down the sequence would influence those positioned earlier, which is equivalent to the model seeing the answers directly during training—a situation we naturally want to avoid.

To prevent this, during the matrix operations of attention, a triangular mask matrix can be applied (as shown in the diagram below) to mathematically force the $t$-th word to only see words from position $1$ to $t$, blocking it from seeing subsequent words. This achieves parallelized training while perfectly simulating the causal law of not being able to see into the future.

Accordingly, the calculation formula for the Masked Self-Attention module becomes:

$$ \text{Attention}(Q, K, V) = \text{softmax}(\frac{QK^T}{\sqrt{d_k}} + M)V $$

Where $M$ is a lower triangular matrix. Its logic is as follows:

For all positions where $j > i$ (i.e., words after the current position), set $M_{ij}$ to $-\infty$.
For all positions where $j \le i$ (i.e., current and preceding words), set $M_{ij}$ to $0$.

Note: During normal forward inference, the model actually executes one forward pass to generate a single token. In this case, during each forward pass, the text sequence passed to the Decoder module via outputs shifted right naturally contains no future tokens (because future tokens have not yet been generated at that moment). Thus, the mask mechanism is actually unnecessary. However, because the parameters in the Transformer are generated under masked conditions during training, removing this masked self-attention module during forward inference could cause a subtle shift in the numerical distribution of parameters, thereby affecting prediction accuracy. Therefore, even though the masking mechanism is practically no longer needed during the forward inference stage, this masked self-attention module must still be retained to ensure that the model’s output during inference is largely consistent in accuracy with its output during training.

Encoder Output and Cross-Attention

In the implementation of the Encoder’s Self-Attention mechanism, $Q$, $K$, and $V$ all originate from the same input. For the Decoder’s multi-head attention module, however, the inputs come from two different sources—this is known as the cross-attention mechanism:

Query ($Q$) comes from the output of the previous layer of the Decoder, which is actually the output of the masked attention module. Its semantic connotation represents: “I have generated these words; what kind of context information do I need now to translate the next word?”
Key ($K$) and Value ($V$) come from the final output of the Encoder. Their semantic connotation signifies: “What are the features and meanings of each word in the source text (source language)?”

Therefore, the cross-attention mechanism can be conceptualized as the Decoder holding queries ($Q$) and searching for answers in the Encoder’s knowledge base ($K, V$).

As shown in the diagram below, based on the output of the previous Decoder module, Queries are obtained via matrix multiplication with $W_q$. Based on the Encoder’s output, Keys and Values are obtained via matrix multiplication with $W_k$ and $W_v$ respectively. Then, using Queries, Keys, and Values, the standard attention calculation workflow is invoked (refer to How to Understand Multi-Head Attention in the Transformer Architecture?) to yield the computation results of the cross-attention module.

Summary of the Decoder Workflow

Finally, let us summarize the workflow of the entire Decoder section:

The first step is inputting the token sequence generated by model inference from the Outputs Shifted Right end. This token sequence undergoes the same preprocessing steps as the Encoder: token vectorization (Embedding) and the addition of positional encoding information.

The second step is the processing flow of the Masked Self-Attention module. Its input is the token vector sequence augmented with positional encoding information. This calculation workflow has been explained in great detail in the Masked Multi-Head Attention section above. The output of the masked self-attention module undergoes residual connection and layer normalization via standard Add & Norm.

The third step is the processing flow of the Cross-Attention module. This calculation workflow has been detailed in the Encoder Output and Cross-Attention section above. Similarly, its output undergoes residual connection and layer normalization via standard Add & Norm.

The fourth step is the Feed Forward network, composed of two fully connected neural network layers. Its role is to perform non-linear transformations and further feature extraction on the complex features captured after passing through the two attention mechanisms (Masked Self-Attention and Cross-Attention). Finally, residual connection and layer normalization are applied via Add & Norm.

Steps two through four above constitute a single Decoder Block. The input and output data dimensions of each Decoder Block are identical. The entire Decoder section contains $N$ Decoder Blocks, where the output of the previous Decoder Block serves as the input to the next, until all $N$ Decoder Blocks have finished processing.

The final part is the output structure’s conversion layer, which consists of two components: a Linear layer and a Softmax layer.

Linear layer: A fully connected mapping layer used to project the abstract vectors extracted by the Decoder (e.g., 512-dimensional, 768-dimensional, etc.) into a massive space—specifically, the space of the full vocabulary supported by the model’s output. If there are 30,000 words in the output dictionary, the output dimension of this layer is 30,000.
Softmax layer: Converts the raw numerical values (Logits) output by the linear layer into a probability distribution via Softmax, ultimately outputting a predicted probability vector. Each value in this vector represents the probability that the corresponding word in the dictionary becomes the next word. The system then selects the word with the highest probability based on this predicted probability vector as the output result, feeds it back to the bottom input end, and begins predicting the next word.