Don't Just Read the Paper, Implement It

Paperman converts bleeding-edge papers into hands-on coding challenges you can solve in your browser.

Start Coding Browse Papers

arXiv:1706.03762

Attention Is All You Need

Vaswani et al., 2017

Abstract

Attention(Q,K,V) = softmax(QK^T/√d_k)V

PaperMan

Attention Is All You Need

Implement the scaled dot-product attention:

def attention(Q, K, V):

# Your code here

scores = Q @ K.T / √d_k

3/5 tests passing

Used by engineers from

Google

HOW PAPERMAN WORKS

1. Select a Paper

Choose from a curated list of top AI research papers.

📄

2. Understand the Math

Grasp the underlying concepts with interactive explanations.

∂L/∂w = -2Σ(yᵢ - ŷᵢ)xᵢ

Attention(Q,K,V) = softmax(QKᵀ/√dₖ)V

σ(z) = 1 / (1 + e⁻ᶻ)

Interactive

Step-by-step

3. Implement the Code

Build the model function by function in our live coding environment.

def

attention(Q, K, V):

# Your implementation

scores = Q @ K.T

weights = softmax(scores)

return weights @ V

✓ 3/3 tests passed

FEATURED PAPERS

20150 submissions

Deep Residual Learning (ResNet)

Introduces residual learning with skip connections to train very deep networks. The key insight is that learning residual mappings $\mathcal{F}(x) := \mathcal{H}(x) - x$ is easier than learning unreferenced mappings directly. The degradation problem motivated the design: adding more layers to a suitably deep model leads to higher training error. This is not caused by overfitting, but by the difficulty of optimizing deeper networks. Residual connections provide an identity shortcut that allows gradients to flow directly through the network, enabling training of networks with 100+ layers.

4 modules