Exploring Graphics Processing Units (GPUs)

3 minute read

Overall Computational Power of GPUs

⚡ Incredible Calculation Speed: Modern GPUs can perform tens of trillions of calculations per second (e.g., 36 trillion for Cyberpunk 2077).
🌍 Human Comparison: Achieving this manually would require the equivalent of over 4,400 Earths full of people, each doing one calculation every second.

🚢 Cargo Ship vs. Airplane Analogy: GPUs are like cargo ships (massive capacity, slower), and CPUs are like jets (fast, versatile, fewer tasks at once).
⚖️ Different Strengths: CPUs handle operating systems, flexible tasks, and fewer but more complex instructions. GPUs excel at huge amounts of simple, repetitive calculations.
🔀 Parallel vs. General Purpose: GPUs are less flexible but highly parallel, CPUs are more general-purpose and can run a wide variety of programs and instructions.

💽 Central GPU Die (GA102): A large chip with 28.3 billion transistors organized into Graphics Processing Clusters (GPCs), Streaming Multiprocessors (SMs), and cores.
🏗️ Hierarchical Structure: GA102 has 7 GPCs → 12 SMs per GPC → 4 Warps per SM → 32 CUDA Per Wrap and 4 Tensor Per Warmp and 1 Ray Tracing Per GPC.
🔢 Types of Cores:
- ⚙️ CUDA Cores: Handle basic arithmetic (addition, multiplication) most commonly used in gaming.
- 🧩 Tensor Cores: Perform massive matrix calculations for AI and neural networks.
- 💎 Ray Tracing Cores: Specialized for lighting and reflection calculations in real-time graphics.

🔧 Shared Chip Design: Different GPU models (e.g., 3080, 3090, 3090 Ti) share the same GA102 design.
🕳️ Defects & Binning: Manufacturing imperfections result in some cores being disabled. This leads to different “tiers” of the same GPU architecture.

➕ Simple Calculator Design: Each CUDA core is basically a tiny calculator that does fused multiply-add (FMA) and a few other operations.
💻 Common Operations: Primarily handles 32-bit floating-point and integer arithmetic. More complex math (division, trignometry) is done by fewer, special function units.

💾 Graphics Memory: GDDR6X chips (by Micron) feed terabytes of data per second into the GPU’s thousands of cores.
🚀 High Bandwidth: GPU memory operates at huge bandwidths (over 1 terabyte/s) compared to typical CPU memory (~64 GB/s).
🔢 Beyond Binary: GDDR6X and GDDR7 use multiple voltage levels (PAM-4 and PAM-3) to encode more data per signal, increasing transfer rates.
🏗️ Future Memory Tech: Micron also develops HBM (High Bandwidth Memory) for AI accelerators, stacking memory chips in 3D, greatly boosting capacity and speed while reducing power.

♻️ Embarrassingly Parallel: Tasks like graphics rendering, Bitcoin mining, or AI training are easily split into millions of independent calculations.
📜 Single Instruction Multiple Data (SIMD): Apply the same instruction to many data points at once—perfect for transforming millions of vertices in a 3D scene.
🔓 From SIMD to SIMT: Newer GPUs use Single Instruction Multiple Threads (SIMT), allowing threads to progress independently and handle complex branching more efficiently.

📦 Thread Hierarchy: Threads → Warps (groups of 32 threads) → Thread Blocks → Grids.
🎛️ Gigathread Engine: Manages the allocation of thread blocks to streaming multiprocessors, optimizing parallel processing.

🎮 Video Games: GPUs transform coordinates, apply textures, shading, and handle complex rendering pipelines. Millions of identical operations on different vertices and pixels are done in parallel.
₿ Bitcoin Mining: GPUs can run the SHA-256 hashing algorithm in parallel many millions of times per second. Though now replaced by ASIC miners, GPUs were initially very efficient at this.
🤖 AI & Neural Networks: Tensor cores accelerate matrix multiplications critical for training neural nets and powering generative AI.
💡 Ray Tracing: Specialized cores handle ray tracing calculations for realistic lighting and reflections in real-time graphics.

🏭 Micron Memory Chips: GDDR6X and future GDDR7 designed by Micron power high-speed data transfers on GPUs.
🔮 Innovations in Memory: High Bandwidth Memory (HBM) for AI chips stacks DRAM vertically, creating high-capacity, high-throughput solutions at lower energy costs.
📚 Technological Marvel: Modern graphics cards are a blend of advanced materials, clever architectures, and innovative manufacturing. They enable astonishing levels of visual realism, parallel computation, and AI capabilities.