Transformers Explained: The Architecture Behind GPT

Transformers are a neural network architecture introduced by Google researchers in the groundbreaking 2017 paper Attention Is All You Need. They replaced recurrent neural networks (RNNs) and convolutional neural networks (CNNs) as the dominant architecture for natural language processing tasks.

The key innovation is self-attention, which allows the model to weigh the importance of different words in a sentence when processing each word. Unlike RNNs that process sequences word-by-word, transformers can process all words simultaneously, dramatically improving training speed and enabling longer context windows.

Every major language model today uses transformer architecture: GPT-4, Claude, Gemini, and LLaMA are all transformer-based. This architecture powers everything from chatbots to code generation and has enabled the current AI revolution.

Attention mechanisms solve a fundamental problem: how can a model focus on relevant parts of the input when processing each element? The transformer uses scaled dot-product attention with three key components:

• Queries (Q): What information are we looking for?
• Keys (K): What information is available in each position?
• Values (V): The actual information content at each position

The attention mechanism computes attention scores by taking the dot product of queries and keys, scaling by the square root of the key dimension, and applying softmax to get attention weights. These weights determine how much each value contributes to the output.

The core attention formula that powers all transformers:

import torch
import torch.nn.functional as F

def scaled_dot_product_attention(Q, K, V):
    # Q, K, V are matrices of shape (batch_size, seq_len, d_model)
    d_k = Q.size(-1)  # dimension of keys
    
    # Compute attention scores
    scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
    
    # Apply softmax to get attention weights
    attention_weights = F.softmax(scores, dim=-1)
    
    # Apply weights to values
    output = torch.matmul(attention_weights, V)
    
    return output, attention_weights

The scaling factor (√d_k) prevents the dot products from becoming too large, which would push the softmax function into regions with extremely small gradients, making training difficult.

Single attention heads can only capture one type of relationship at a time. Multi-head attention runs multiple attention mechanisms in parallel, each learning different aspects of the input relationships.

For example, in the sentence 'The cat sat on the mat', different attention heads might focus on:

• Syntactic relationships: Subject-verb connections (cat → sat)
• Semantic relationships: Object associations (cat → mat)
• Positional relationships: Spatial prepositions (sat → on)
• Contextual relationships: Overall sentence meaning

GPT-3 uses 96 attention heads across 96 layers, creating an incredibly rich representation space. This parallel processing of different relationship types enables the nuanced understanding that makes modern AI agents so effective.

Transformers process all positions simultaneously, which is great for speed but creates a problem: how does the model know word order? 'Cat chased dog' and 'Dog chased cat' have the same words but opposite meanings.

Positional encoding solves this by adding position information to each word's embedding. The original transformer uses sinusoidal functions that create unique patterns for each position:

import numpy as np

def positional_encoding(seq_len, d_model):
    pos = np.arange(seq_len)[:, np.newaxis]
    div_term = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model))
    
    pe = np.zeros((seq_len, d_model))
    pe[:, 0::2] = np.sin(pos * div_term)  # Even dimensions
    pe[:, 1::2] = np.cos(pos * div_term)  # Odd dimensions
    
    return pe

Modern models like GPT use learned positional embeddings instead of fixed sinusoidal patterns, but the principle remains: adding position information allows parallel processing while preserving order.

The original transformer has two main components working together:

• Encoder: Processes the input sequence and creates rich representations
• Decoder: Generates output sequences using encoder representations and previous outputs

Each encoder layer contains multi-head attention followed by a feed-forward network, with residual connections and layer normalization. The decoder adds a third component: cross-attention that attends to encoder outputs.

This full architecture was designed for sequence-to-sequence tasks like translation, but most modern applications use simplified versions optimized for specific tasks.

Different AI applications use different parts of the transformer architecture:

Understanding transformer implementation helps you debug models, optimize performance, and build custom architectures. Here's a simplified transformer block:

import torch
import torch.nn as nn

class TransformerBlock(nn.Module):
    def __init__(self, d_model, num_heads, d_ff, dropout=0.1):
        super().__init__()
        self.attention = nn.MultiheadAttention(d_model, num_heads, dropout)
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        
        # Feed-forward network
        self.ff = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.ReLU(),
            nn.Linear(d_ff, d_model),
            nn.Dropout(dropout)
        )
        
    def forward(self, x):
        # Multi-head attention with residual connection
        attn_out, _ = self.attention(x, x, x)
        x = self.norm1(x + attn_out)
        
        # Feed-forward with residual connection
        ff_out = self.ff(x)
        x = self.norm2(x + ff_out)
        
        return x

This simplified implementation shows the key components: multi-head attention, feed-forward layers, residual connections, and layer normalization. Production models like GPT add optimizations like attention mechanisms and specialized initialization schemes.