Summary

Introduction

Table of Contents

• Positional Embeddings
  • Fixed Positional Embeddings
  • Learned Positional Embeddings
  • Rotary Positional Embeddings (RoPE)
  • No Positional Embeddings (NoPE)
• Attention
  • Multi-Headed Attention (MHA)
  • Multi-Query Attention (MQA)
  • Grouped-Query Attention (GQA)
  • Sliding-Window Attention (SWA)
  • Attention Sink
  • KV Cache
• Sampling
  • Top-K
  • Top-P
  • Temperature
  • Speculative Sampling
• Finetuning
  • Low Rank Adaptation (LoRA) of LLMs
  • Efficient Finetuning of Quantized LLMs (QLoRA)
  • Reinforcement Learning from Human Feedback (RLHF)
  • Reinforcement Learning from AI Feedback (RLAIF)
• Prompting
  • Chain of Thought (CoT)
  • Tree of Thought (ToT)
• Batching
  • Continuous Batching