GPUs and CUDA language

• SIMD vs. MIMD: Intel SIMD instructions; NVIDIA specializes SIMD in a different direction.
• GPU is daughterboard: Chips on board: Streaming Multiprocessors (e.g., 8 or 16 SMs)
• A Survey on Parallel Computing and its Applications in Data-Parallel Problems Using GPU Architectures by Cristóbal A Navarro, Nancy Hitschfeld, Luis Mateu
  • On-chip (the SM, or streaming multiprocessor)
  • CPU versus SM (The cache, register, control unit and ALU exist in both architectures.)
  • In CUDA, the space of computation is composed of a grid, blocks and threads. 2-D or 3-D grid to index into a specific block
    • A block has access to local memory ("shared memory"), including cache, registers assigned to each SIMD core, etc.
    • A thread executes on a SIMD core inside an SM.
    • A block (of threads) may be divided into multiple warps. Each warp has the same number of threads as the number of physical cores on that chip. A block is the number of threads operating on that chip. Can be four times as many threads as cores, for example. This allows overlap of computation and communication: bring in registers for next warp, while doing computation with the current warp.
• WEAK POINTS OF GPUs:
  1. many SM chips with local memory, "talking" to one very large RAM (e.g. 100 GB) simultaenously. Many optimimizations to lower this penalty: paraellel read of read-only memory, switch can do small operations on multiple writes from different SMs, etc.
  2. If your code needs a synchronization lock across multiple SMs, this is a huge penalty. Barrier between global tasks is okay, but giving exclusive write-access to a single SM is expensive.
  3. Very large LLMs have more memory than fits in the GPU. There are techniques like UVM, back-end network among GPUs, etc., but these are not yet commonly used.

Large Language Models (LLMs)

• Large language model (Wikipedia) (and related term: Generative AI):
  General Properties:
  • emergent abilities (with scale)
  • transformer architecture: replaces recurrent neural nets (RNN)
    • - LLMs are trained in an unsupervised manner on unannotated text.
    • - A 2020 study estimated the cost of training a 1.5 billion parameter model (2 orders of magnitude smaller than the state of the art at the time) at $1.6 million.
    • - But advances in software and hardware have brought the cost substantially down, with a 2023 paper reporting a cost in the hundreds of thousands of dollars to train a 12 billion parameter model.
    • - Power law based on:
      * model size (number of parameters)
      * number of training tokens
      * compute time
• Try an LLM: (I currently use Google Bard because it has nice support for programming, and because OpenAI GPT currently requires me to give them my phone number if I want a free account. Microsoft BART also exists, and more are being developed all the time.)
• Articles on LLM for a general, scientific audience:
  Large Language Models Explained
  • "There are two key innovations that make transformers particularly adept for large language models: positional encodings and self-attention."
  • "Positional encoding embeds the order of which the input occurs within a given sequence." -- For example, choose noun first, followed by its adjectives.
  • "Self-attention assigns a weight to each part of the input data while processing it."
  • "... what parts of the input the neural network needs to pay attention to is learnt over time as the model sifts and analyzes mountains of data."
  • A Survey of Large Language Models (June, 2023)
    Wayne Xin Zhao, Kun Zhou*, Junyi Li*, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, Ji-Rong Wen
    (22 authors, Gaoling School of Artificial Intelligence and School of Information, Renmin University of China)
    • STEPS for training in "Transformer" architecture by Google:
      • Tokenize (clean-up input, and do obvious groupings of input char's) Also label out-of-vocabulary tokens (to be ignored)
      • Encoder (see below); The encoder produces a vector from the input
      • Decoder (see below); The decoder takes the vector and produces the output
      • Fine tuning (Take pre-trained LLM, and train further on a particular domain; The pre-training (e.g., decoder) trains on knowledge; and the fine tuning extracts that knowledge; Knowledge can be domain-specifice, such as "programming", etc.)
      • (NOTE: Encoder-only and Decoder-only are also possible.)
      • A figure of merit as the traing progresses is: perplexity
    • HISTORY (pp. 7 and 8):
      Google: GPT (Generative Pre-Training): targeting machine translation from English to French or German
      unsupervised pretraining, supervised fine-tuningp [ But fine-tuning of existing LLMs also exists: Add task-specific layers to LLM and train on input from specific task. ]
      OpenAI then developed GPT-2 based on the Google Transformer model
    • Side note: [ China: 11 LLM models authorized by government: presumably because they passed the fine-tuning tests;
      fine-tuning: expose to specific tasks, suppress wrong answers.
      Outsourcing for fine-tuning to people (usually in third world).
      Depends on large input text. So, English is favored. Rumors are that the Chinese LLM models train largely on an English corpus, and then do English-to-Chinese translation. (Some students here at Northeastern can report examples from Chinese LLM where they can see artifcats from a poor Chinese-to-English translation. ]
  • Attention is all you need (or alt)
    (original Google GPT Transfomer model; paper from 2017)
    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin,
    in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 (NIPS'17), 2017, pp. 5998–6008.
    (8 authors, NIPS-2016)
    • GPT-1. In 2017, the Transformer model [22] was introduced by Google, and the OpenAI team quickly adapted their language modeling work to this new neural network architecture
    • Uses: Recurrent neural networks (RNN)
    • Based on recurrent language models and encoder-decoder architectures
    • From "Introduction": "This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples. Recent work has achieved significant improvements in computational efficiency through factorization tricks [21] and conditional computation [32], while also improving model performance in case of the latter. The fundamental constraint of sequential computation, however, remains."
    • RNN: Standard neural network, but with feedback paths from output to earlier inputs. This enables a "long-term memory" of the training, as well as "short-term memory".
  • OpenAI blog (2019): "Language Models are Unsupervised Multitask Learners", Radford et al., OpenAI
    This is where OpenAI presented their GPT-2 model.
    They went to 1542M parameters, 48 layers, d_{model} (model depth) = 1600
    (These numbers refer to the RNN being used.)
  • "Understanding Encoder And Decoder LLMs" by Sebastian Raschka
    (This is an excellent popular-science style article on encoder-decoder architecture.)
    And he's written a book:
    "Machine Learning Q and AI", which includes: Multi-GPU training paradigms.
  • Other possibly useful blogs:
    • A popular-science type blog 2023): Training Large Language Models (LLMs): Techniques and Best Practices (2023)
    • Good popular article on fine-tuning: "A Complete Guide to Fine Tuning Large Language Models" (July, 2023)
    • Training Compute-Optimal Large Language Models; Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre (22 authors)

  The Software

• SOFTWARE: THE CODE
  • GPT-2 source code; (407 lines of Python code); GPT-2 dataset
  • Bert (from Microsoft)
  • "Democratizing access to large-scale language models with OPT-175B" (pre-trained transformer, ready for "fine-tuning" for a particular task)
  • TensorFlow Core (tutorial for using TensorFlow for machine learning)
• SOFTWARE: DESIGN FOR SCALING UP
  • Efficient Training on Multiple GPUs (Nice overview of software optimization techniques)
    • Methods and tools for efficient training on a single GPU (for training Neural Nets, in general)
  • OTHER DESIGNS FOR SCALING UP (NVIDIA): SCALING UP WITH MULTIPLE GPUs: (background for understanding the systems constraints; We will not go head-to-head with Nvidia searching for novelty; We'll find novelty elswhere; We simply need to understand the constraints that Nvidia is describing for us.)

    Fundamental Problem:

    Training datasets are large. We can parallelize across multiple GPUs. But if we do it naively, the lower bandwidth of the inter-node network slows us down, and we no longer get linear scaling. This is called weak scaling, while near-linear scaling is called strong scaling. (See this blog on scaling. We will usually see strong scaling on one node, but weak scaling when we parallelize across many nodes, even when we use the very low-latency RDMA-type networks of HPC (e.g., InfiniBand).

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2020) -- NVIDIA, part 1 , Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro (arXiv technical report, 2020)
    See Section 2.3 for the "meat" of this paper:

    MORAL:

    When the problem no longer fits in the memory of the GPU, we need to split it up into chunks. There are two primary techniques:

    Data Parallelism:

    Split the training data. Since it's high-dimensional (it's an n-dimensional tensor, not a 2-dimensional matrix), either take a slice of the data (fix the indices across some of the dimensions), or take some high-dimensional neighborhoods of some n-dimensional index, and use that smaller training data on one GPU. Split the data so that all data is sent to at least one of the GPUs. Later, combine it all.

    Model Parallelism:

    Split the weights of the RNN neural net (or split the neurons). Each GPU trains on the entire dataset, but across fewer parameters (weights or neurons). Later, combine it all.

    Reducing Activation Recomputation in Large Transformer Models (2022) -- NVIDIA, part 2, Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, Bryan Catanzaro (7 authors; published at MLSYS'23; arXiv technical report in 2022); Reduce memory pressure of storing activations 5x reduction in memory; recover 90% of compute overhead of full activation recomputation (model parallelism: sequence parallelism + tensor parallelism)

    MORAL:

    Trade extra computation time for reduced use of network bandsiwth.

    From abstract:

    "Activation recomputation is commonly used to work around memory capacity constraints. ... In this work, we show most of this redundant compute is unnecessary because we can reduce memory consumption sufficiently without it. We present two novel yet very simple techniques: sequence parallelism [NOTE: on top of the tensor parallelism from above] and selective activation recomputation [NOTE: on top of the sequence parallelism]."

    "Efficiently Scale LLM Training Across a Large GPU Cluster with Alpa and Ray" -- NVIDIA, part 3 , Jiao Dong, Hao Zheng, Lianmin Zheng and Phi Nguyen (on-line Technical Blog)

• RESEARCH QUESTIONS (especially automated programming, automated debugging):
  • NOTE: Depending on the student, one possible personal goal for this course is to write and submit a workshop paper. Here are some research ideas that can direct our literature search. If you get lucky, you will ahve a paper, and the last half of this course will include assistance on polished technical writing.
  • "AI-Assistend Programming": Can we provide systems tools (more powerful than GDB) that automatically introsepect and determine correctness of code, correctness of a spec, etc. (Consider what an LLM does for citations, when asked to write a survey paper. It can do something similar for programming.)
  • Combination with Algorithmic Programmaing? (see course home page)
  • Google Bard promises "Export Python code to Replit". Replit claims to automatically assist in programming in some limimted cases. What is this lacking? Can we fill in some holes? [ Google Bard can also export to Google Code (Jupyter notebooks), but that doesn't impact on automated programming, automated debugging. )
  • Can we automatically generate unit tests with ChatGPT, etc.? People hate writing unit tests (just as they hate Q&A). But it's necessary for software reliability. Can we describe a unit to ChatGPT, and then use systems techniques to test and "fix" it? If we can ask for the spec of a unit test in English, that makes life much more pleasant.
  • Can transparent checkpointing complement AI-assisted programming?
    (With transparent checkpointing, we can stop (checkpoint) a program at a point that we believe is correct, and then speculate on different implementations of the next few lines of code. We restart repeatedly with each new speculation, and look at the difference in situ.

    For shorter programs, it may be sufficient to rerun the entire program deterministically, and stop at the n-th instruction, to try new speculations. But if the program is multi-threaded, even this will not suffice, since it is not deterministic.)

  • OTHER???