Concepts

Definitions and explanations of machine learning terms, organized as an interconnected graph.

• LLM
• Softmax Attention
• Scaled Dot-Product Attention
• Multi-Head Attention
• Residual Connections
• Gradient Highway
• Mixture of Experts
• Expert Routing
• Linear Attention
• RNN

• Layer Normalization
• PreNorm
• PostNorm
• Hidden-State Growth
• Vanishing Gradients

• Model Pruning
• Layer Pruning
• Block AttnRes
• Pipeline Communication

• Scaling Laws
• Chinchilla Scaling
• Neural Scaling
• Kimi Linear