To Begin With: The Core Triangle of AI Infra
New concepts keep popping up in this AI era, but the filter for noise is surprisingly straightforward.
After decades of evolution, the fundamental constraints of computing systems have never changed: computation, data, and communication - like a cycle.
Every innovation with genuine, long-term technical moats essentially pushes the limits on one of these three fronts: cutting redundant overhead in computation scheduling, reducing bandwidth waste from data movement, or lowering communication latency across processing units and nodes.
New AI-era technologies like CUDA Graph, KV Cache, and FlashAttention do trace their core ideas back to classic system theories — but they are far from rebranded old tricks. They take timeless principles and push them to the limit under entirely new hardware constraints: GPU tensor computing, HBM memory, and ten-thousand-GPU clusters, delivering measurable performance leaps. That is where real hard-core value lives.
On the flip side, “new technologies” that only do high-level process orchestration, UI wrapping (not saying UI design here..), or concept rebranding, without touching the underlying physical cost of computation, data, or communication, are essentially just engineering integration. They have practical value, but no irreplaceable technical barrier, and they fade first when industry hype cools down.
The one question to ask before diving deep into any AI Infra technology: does it actually move the needle on the physical overhead of this core triangle?
Tensor Library
The underlying mathematical engine of neural networks, responsible for defining the data structure of tensors, memory management, and matrices & activation functions-related numerical operations.
GGML is a lightweight C/C++ tensor library designed specifically for efficient inference of neural networks, especially LLMs.
Specifically, it contains:
- Tensor Management Layer (
ggml_tensor) - Operator Library
- Multi-Hardware Backends and Task Scheduling Layer
- Computational Graph Scheduling Layer (lazy evaluation)
GGML can simultaneously schedule CPU / AMD-Vulkan / Apple-Metal / Nvidia-CUDA.
When GGML needs to compute, e.g. Transformer forward passes on an Nvidia card, the GGML-CUDA backend calls the CUDA API to launch custom GPU kernel functions that perform tensor operations.
CUDA: a hardware programming API, the bridge between programmers and GPUs.
GPU kernal function (Kernel): A C-like function to be executed in GPU, assigned by CPU.
For example:
__global__ void add_kernel(float *a,float *b,float *out) {
int idx = threadIdx.x;
out[idx] = a[idx] + b[idx];
}Summary - the entire procedure:
- llama-cpp initiates a request: it needs to compute matrix multiplication for the attention layer;
- GGML creates the weight tensors and copies the quantized weights into GPU memory;
- GGML’s CUDA backend calls the CUDA runtime API;
- It loads the custom Kernel (a 4-bit matrix multiplication kernel) that was hand-coded in advance;
- It launches tens of thousands of GPU threads to perform the tensor computations in parallel;
- Once the computations are complete, the result tensor is returned.
Reference
GGML Joined Hugging Face! But… What Is GGML?: https://huggingface.co/blog/salihfurkaan/ggml-introduction?f_link_type=f_linkinlinenote&flow_extra=eyJkb2NfaWQiOiJmZjYwNGY2ZDg2YTU5YjYzLWEzNTBjMzM1ZTc5MjI4OWIiLCJpbmxpbmVfZGlzcGxheV9wb3NpdGlvbiI6MCwiZG9jX3Bvc2l0aW9uIjowfQ%3D%3D