Skip to main content

ML Systems, Hardware, and Code

LLM Systems, Hardware, and Code​

In the transformer and llm section, we cover the theory behind LLM's, Transformers and Attention, and specific models like BERT and GPT. In this section, we cover the systems, hardware, and code that make it possible to train and run LLM's at scale, and talk about some of the systems behind them like training, inference, optimizations, and more

There's a great example of utilizing an event loop in Python / Go and C++ infrence process on GPU's in a monolithic way to have 8 models running on a single VM to handle a high concurrency and low latency inference system in a monolith. This ties into showcasing Triton, which is an open source NVIDIA software package that handles all of this for us on NVIDIA GPU's

GPT, BERT, and Others

BERT vs GPT

Training​

LLM Training is done over gigantic corpa of data, and requires a lot of compute. We talk about the systems behind training, including data pipelines, distributed training, and more

Most of these systems have Human In The Loop components where human feedback is used to improve the model, and that will get into how a sentence with multiple potential outputs is framed as a reward function for the model to optimize for, and how that is used in practice to train LLM's

Various training datasets are used, mostly around sourced web content from social media, online encyclopedias, and structured curated text LLM Training Dataset Sources