Basic

I. One Node
II. Four Nodes
III. One Hidden Layer
IV. Three Inputs
V. Seven Layers
Advanced
1. Mixture of Experts (MOEs)
2. Recurrent Neural Network (RNN)
3. Mamba
4. Matrix Multiplication
5. LLM Sampling
6. MLP in PyTorch
7. Backpropagation
8. Transformer
9. Batch Normalization
10. Generative Adversarial Network (GAN)
11. Self Attention
12. Dropout
13. Autoencoder
14. Vector Database
15. CLIP
16. Residual Network (ResNet)
17. Graph Convolution Network (GCN)
18. SORA’s Diffusion Transformer (DiT)
19. Gemini 1.5's Switch Transformer
20. Reinforcement Learning with Human Feedback (RLHF)

© 2024 Tom Yeh

Link to my original LinkedIn post
with animation and explanation

Date originally posted

 I. One Node 406

12.5.23 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

 II. Four Nodes 148

12.6.23 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

 III. Hidden Layer 82

12.7.23 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

 IV. Three Inputs 105

12.13.23 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

 V. Seven Layers 197

12.13.23 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

 1. Mixture of Experts 683

X1 X2
2 3
1 1
3 2
1 1
Gate 1 1 0 0 Max
Network 0 1 1 0 ≈

1 0 1 0 0 1 1 0
1 1 0 0 1 1 0 0 ReLU

0 0 1 0 1 -1 0 0 ≈
-1 0 1 0 1 0 1 0
Y1 Y2
Expert 1 Expert 2

12.15.23 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

 2. Recurrent Neural Network (RNN)
406
Input Sequence X 3 4 5 6

1 -1 1
Parameters A B C -1 1
1 1 2

Activation Function ɸ: ReLU

Hidden States H0
0
0

Output Sequence Y

1 1 -1 ɸ ɸ ɸ ɸ
2 1 1 ≈ ≈ ≈ ≈
-1 1

12.18.23 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

 3. Mamba’s S6 Model 263

Input Sequence 3 4 5 6 Parameters

Output Sequence

Selective Structured State-Space

Scan

1 0
1 -1 0 0
0 -1
0 -1 0 1
1 0 -1 0
1 0 0 -1
1 0 -1 0 1 0

0 1 0 -1 0 -1

1 -1 0 0
0 0 -1 1
-1 0 0 0 -1 0
1 0 0 0 0 1
0 0 -1 0
0 1 0 0
1 -1 0 0
-1 0
0 0 -1 1
0 1
1 0 0 0
0 -1 1 0

12.19.23 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

 4. Matrix Multiplication 127

1 1 1 5 2
X =?
-1 1 2 4 2

1.5.24 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

5. How does an LLM sample a sentence?
Input Embeddings 1123

LLM
Probability Distributions Random
Numbers
I .01 .01 .03
Vocab you .01 .01 .50
they .01 .01 .40
are .01 .40 .01 .34
am .01 .40 .01 .52
how .50 .05 .01 .92
why .10 .05 .01 .65
where .10 .05 .01
who .15 .01 .01
what .10 .01 .01

______ ______ ______

1.6.24 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

 6. Multi Layer Perceptron in pytorch
2
337
1 mlp_model = nn.Sequential(
1
3
2 nn._______( ___, ___, bias = ___ ),
1

1 -1 1 -5 -1 0
3
nn._______(),
1 1 0 0 3 ReLU 3
0 1 1 1 5 ≈ 5
4
nn._______( ___, ___, bias = ___ ),
1 0 1 -2 3 3
5
nn._______(), 1 -1 1 0 2 ReLU 2
0 1 -1 1 1
≈ 1
nn._______( ___, ___, bias = ___ ),
6 1

1 -1 2 3 .95
7
nn._______() -1 1 1 0 .50
σ
8
)
1 -2 -2 -2
≈ .12
2 1 0 5 .99
-3 0 1 -5 .01

Hints:

Linear Layer: { Identity | Linear | Bilinear }

Activation Function: { ReLU | Tanh | Sigmoid }
in_features: { int }
out_features: { int }
bias: { T | F }

1.8.24 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

 7. Backpropagation 1260

X
2
1
3
1

Layer 1
1 -1 1 -5 -1 0
1 1 0 0 3 ReLU 3
0 1 1 1 5 ≈ 5
1 0 1 -2 3 3
1

Layer 2
1 -1 1 0 0 2 ReLU 2
0 1 -1 1 3 4 ≈ 4
1

Layer 3 3
Soft
0
2 0 -1 max .5
0 2 -5 3 ≈ .5 1
1 1 1 -1 0 0
YPred YTarget
L: Cross-Entropy Loss

1.9.24 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

 8. Transformer 1281

Features from the

Previous Block
↓ ↓ ↓ ↓ ↓

1 0 0 0 1
Attention
1 1 0 0 0
Q Attention
0 1 1 0 0 Weight
K
0 0 1 1 0 Matrix (A)
0 0 0 1 1

X1 X2 X3 X4 X5 Z1 Z2 Z3 Z4 Z5
5 6 0 7 0 Attention
0 2 4 0 3 Weighted
1 0 1 1 0 Features

1 1 1 1 1

1 -1 0 1
1 1 0 0 ReLU
0 1 1 1 ≈
-1 1 1 0
1 1 1 1 1

Position-wise 1 0 0 -1 0
Feed-Forward 0 1 1 0 0
Network (FFN) 0 0 1 -1 1

↓ ↓ ↓ ↓ ↓

Next Block

1.11.24 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

 9. Batch Normalization 504

Mini-batch: X1 X2 X3 X4
1 0 3 0
0 3 1 1 Batch Statistics
2 1 0 2
Linear Layer 1 1 1 1 Σ µ σ2 σ
1 0 1 0
ReLU
1 1 0 -1
0 2 -1 0 ≈
Normalize µ
Sum (Σ)
- Mean (µ)
Variance (σ2)
Std Dev (σ)

σ
÷

1 1 1 1

Scale & Shift 2 0 0 0 Trainable

0 3 0 0 Parameters
0 0 -1 1

Next Layer

1.14.24 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

 10. Generative Adversarial Network (GAN)
Noise: N1 N2 N3 N4
1004
1 1 0 1
1 0 1 -1
1 1 1 1

Generator 1 1 0
0 1 2
-1 1 0
[≈ ReLU] 1 1 1 1 Real:
Fake: F1 F2 F3 F4 X1 X2 X3 X4
-1 1 0 0 2 3 3 4
1 0 1 0 1 1 1 1
0 1 1 0 2 3 4 3
0 0 1 1 1 1 1 1
[≈ ReLU] 1 1 1 1 1 1 1 1

Discriminator 1 0 0 -1 0
0 1 1 0 0
0 0 1 -1 1
[≈ ReLU] 1 1 1 1 1 1 1 1

1 1 -1 -1 Z
[≈ σ] [≈ σ]
Predictions: Y

Training the
Discriminator
Targets: - YD 0 0 0 0 1 1 1 1
𝜕𝐿𝐷
Loss Gradients:
𝜕𝑍

Training the
Generator
Targets: - YG 1 1 1 1
𝜕𝐿𝐺
Loss Gradients:
𝜕𝑍

1.15.24 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

 11. Self Attention 1376
q1 q2 q3 q4
x1 x2 x3 x4
2 0 0 2 MatMul
Features
0 1 0 0 (KTQ)
0 2 1 0
0 0 1 1 k1 T
2 0 0 0 k2 T
1 0 1 1 k3 T
WQ q1 q2 q3 q4
k4 T

1 1 0 0 0 0
Scale
0 1 0 1 0 0
0 0 1 0 1 1
□
!!
WK k1 k2 k3 k4
0 0 1 0 0 0
Softmax
0 1 0 0 0 0
1 0 0 0 0 -1
e□

÷ Σ

Attention
Weight
Matrix (A)
MatMul

WV v1 v2 v3 v4 z1 z2 z3 z4
10 0 0 0 0 0 Attention
Weighted
0 0 0 10 0 0 Features
0 10 0 0 0 0

FFN
1.16.24 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh
 12. Dropout 557

Random
Sequence
X1 X2 Inference
3 5 3 3
Training Data: Unseen Data:
.61 4 1 2 1
.39 1 1 1 1

.75 Linear 1 0 0 1 0 0
.40 1 1 0 1 1 1
.65 0 1 1 -1 1 1
.42 1 -1 0 1 -1 0
.23 [≈ ReLU] 1 1 [≈ ReLU] 1 1

.19 Dropout 0 0 0 0 0 0
.93 (p=0.5) 0 0 0 0 0 0
.42 0 0 0 0 0 0
.87 0 0 0 0 0 0
.53
Linear 1 0 0 1 0 1 0 1 1 0
.27
.69 0 1 1 0 0 1 1 1 0 0
.50 1 0 -1 -1 1 1 0 -1 0 1
[≈ ReLU] 1 1 [≈ ReLU] 1 1
.11
Dropout 0 0 0 0
.42 (p=0.33) 0 0 0 0
0 0 0 0
1 1
Linear 1 -1 0 0 Outputs 1 1 0 0
0 1 -1 -2 Y 0 1 -1 -1

-4 7 Targets
Training
- 10 5 Y’ MSE Loss
Gradients
𝜕𝐿
X2
𝜕𝑌

1.19.24 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

13. Autoencoder X1 X2 X3 X4
1 1 2 1
849
2 1 2 0
3 2 4 1
1 1 2 1
1 1 1 1

Encoder 1 0 0 1 0
0 1 1 0 0
-1 0 1 0 -1
[≈ ReLU] 1 1 1 1

1 0 1 0
Bottleneck
-1 1 0 0
[≈ ReLU] 1 1 1 1

1 0 0
Decoder
0 1 1
1 -1 0
[≈ ReLU] 1 1 1 1

1 0 -1 0 Outputs

1 -1 0 0 Y
0 0 1 1
0 1 1 -3

Targets
Reconstruction
Loss Y’

- MSE Loss 𝜕𝐿
Gradients 𝜕𝑌

X2

1.22.24 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

 14. Vector Database 2224
Query
Data how are you who are you who am I am I you

Word Embeddings a an the how why who what are is am be was you we I they she he she me him her

0 -1 0 1 0 1 0 0 -1 1 0 0 0 3 1 0 -1 0 0 0 -1 0
2 0 2 0 0 0 -1 1 0 0 0 2 1 0 2 0 2 0 0 2 0 0
-1 0 -1 1 2 0 0 1 0 1 -1 0 0 -1 0 3 0 0 -1 0 2 -1
0 1 0 0 1 0 1 0 1 0 1 -2 0 0 0 1 0 1 0 1 0 1

Text Embeddings 1 1 1 1 1 1 1 1 1 1 1 1

Encoder 1 1 0 0 0
0 1 0 1 0
1 0 1 0 -1
Linear &
ReLU 1 -1 0 0 0

Mean Pooling

Indexing Projection
1 1 0 0 Vector
Storage
0 0 1 1

Retrieval Dot Products

Nearest Neighbor (argmax)

2.1.24 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

mini batch of text-image pairs
15. CLIP 400 millions more …
885

big table mini chair top hat table top big chair

word2vec mini big top hat chair table Flatten

0 1 1 0 1 0 Patches
1 0 0 0 1 1
0 1 0 1 0 1
Word
Image Encoder
Embeddings 1 1 1 1 1 1

1 1 0 0 0
Text Encoder 1 1 1 1 1 1 0 1 0 1 1
1 0 1 0 0 0 1 -1 1
0 3 0 -2 0 1 1 1 -1
1 1 0 1

[Mean Pooling] [Mean Pooling]

(round) (round)

[Projection] 1 1 1 [Projection] 1 1 1

1 1 0 -2 -1 1 1 0 -1
0 1 1 -2 0 0 -1 1 0

T1 T2 T3 Shared Embedding Space

Cross Entropy
[Softmax] Loss Gradients
÷ Similarity -
e□ Σ ImageàText Target ImageàText
I1 1 0 0
I2 0 1 0
I3 0 0 1
÷ Σ TextàImage
1 0 0
Similarity
TextàImage - 0 1 0
0 0 1
2.10.24 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh
 16. Residual Network 625
2 1 0
0 3 4
X
1 0 2
1 1 1
weight
layer 1 1 0 0
ReLU 0 1 -1 0
1 0 1 -1
weight 1 1 1
layer
1 0 0 0 0 1 0
+ 0 1 0 -1 0 0 0
ReLU 0 0 1 0 -1 0 0
1 1 1

Transformer’s Encoder Block

1 0 0
0 1 0
Input
Embedding 0 0 1

1 0 1 1 0 0
Q
K 1 1 0 0 1 0
Attention 0 1 1 0 0 1
Add & Norm
2 0 1
1 3 -2
Feed 1 1 1

Forward 1 1 1
Add & Norm
1 -1 2
1 1 1

1 0 0 -1 3
Next Block
0 1 -1 0 0
↓ ↓ ↓
Next Block
2.15.24 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh
 17. Graph Convolutional Network
573
Graph Data A B C D E

A B C D E
Graph A B C D E A
Convolutional 2 0 1 0 1 Adjacency
Network B
Matrix
1 1 0 0 0
C
0 0 -1 1 1 A B C D E
D
0 3 0 1 0
E
1 1 1 1 1

1 1 0 0 0
0 1 0 -1 0
1 0 0 1 -1
[ReLU] Messages 1 1 1 1 1

0 1 1 0
1 -1 0 -2
1 0 0 0
[ReLU] Messages 1 1 1 1 1

Fully 1 0 0 -2
Connected 0 1 0 -2
Network
0 0 1 -5
1 -1 0 0
1 0 -1 0
[ReLU] 1 1 1 1 1

1 1 1 1 1 -9

2.17.24 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

 18. SORA’s Diffusion Transformer 2118
Training Video Diffusion Prompt
Step t = 3
1 0 2 0 0 1 0 1 “sora is sky”
0 1 1 0 3 0 4 0
Text
Encoder
Spacetime
Patches 0
1
(Pixels) 1
1 0 0 1 0 0 -1
1 1 1 1 0 1 0 -1 1 0
Visual Encoder Latent
1 1 0 0 1 0 Self-Attention
1 0 -1 0 0
0 2 0 1 0 2 1 0 0 0
0 1 0 1 1
[ReLU] Adaptive 1 1 0 0
Sampled 0 2 1 -1 Q 0 1 1 0
Layer Norm
Noise K
+ -1 0 -2 1 0 0 1 1
X +
Noised
Latent
1 1 1 1

Predicted Pointwise -1 1 -2
Noise FFN
- 0 1 -5

Noise-free Train Sampled 0 2 1 -1

Noise
Latent
- -1 0 -2 1
Visual Decoder 1 1 1 1

MSE Loss
1 0 1
Gradients
0 1 0
1 1 0 Generated Video
-1 1 0
[ReLU]

2.19.24 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh

 19. Switch Transformer 576
(Gemini 1.5’s Sparse Mixture of Experts)

1 0 0 0 1 Attention
Previous Block 1 1 0 0 0
↓ ↓ ↓ ↓ ↓ Q Attention
0 1 1 0 0 Weight
K
0 0 1 1 0 Matrix (A)
0 0 0 1 1

X1 X2 X3 X4 X5 Z1 Z2 Z3 Z4 Z5
5 6 0 7 0 Attention
0 2 4 0 3 Weighted
1 0 1 1 0
Features

Switch A 1 -1 0 Gate
B 1 0 -2 Values
C -1 1 1

argmax Expert IDs

Expert A 1 1
Expert B 1 1
Expert C 1 1

1 0 -1 0 0 0 -1 0 1 0 0 0
0 1 0 0 1 0 0 0 0 1 1 1
0 0 -1 1 1 1 -1 1 0 1 0 0

Position-wise 1 2 3 4 5
Feed-Forward x3
Network (FFN)

↓ ↓ ↓ ↓ ↓

AI by Hand ✍ Vol. 1 © 2024 Tom Yeh Next Block

2.24.24
 20. Reinforcement
LLM 5 1 0
prompt
0 4 1 Learning with Human
[S] CEO is 0 0 4 Feedback (RLHF)
550
Preferences
{Winner | Loser} {Winner | Loser}
1 0 -1 2 prompt next prompt next
[ReLU] -1 1 0 0 doc is him doc is them

him 1 1 0
her 0 1 0 Word Embeddings
them 1 0 0 him her them is doc CEO [S]
is/are -1 1 1 0 1 1 1 1 0 -1
doc 2 0 -2 1 0 0 1 1 1 -1
0 1 0 1 0 1 -1
CEO 2 0 -1

Sample (max) Mean Mean

Pool Pool
1/3 1/3 1/3
Reward
1/3 1/3 1/3
Model (RM)
1/3 1/3 1/3

1 0 1 0
0 1 0 0
1 0 -1 0
1 1 0 0
[ReLU]
3 3 3 -3 1 Reward

Align LLM Loss Train RM Winner - Loser

Loss Gradient Predicted σ

- Target -1

3.4.24 AI by Hand ✍ Vol. 1 © 2024 Tom Yeh Loss Gradient