A Hackers Guide to Large Language Models
Tubopedia Mission
In this conversation, we delved into the intricacies of fine-tuning language models, exploring various options and tools such as Llama.cpp and Axolotl for model optimization. The user demonstrated the process of running language models on Macs. Additionally, retrieval augmented generation, a technique incorporating document search to enhance model responses, was explored. The user emphasized the collaborative nature of the field, encouraging participation in platforms like the Fast AI Discord channel. Overall, the discussion provided insights into the evolving landscape of language model usage, offering practical tips and considerations for enthusiasts and practitioners alike. [A Hackers Guide to Large Language Models](https://www.youtube.com/watch?v=jkrNMKz9pWU) ## Intro & Basic Ideas of Large Language Models All right, let me break it down for you. Jeremy Howard from fast.ai is giving a code-first approach to understanding language models. He starts by explaining what a language model is—it predicts the next word in a sentence. He uses OpenAI's language model, text DaVinci 003, as an example. The core idea behind [language models, like GPT-4](/posts/What-is-an-LLM), comes from a 2017 algorithm called ULM fit, developed by Howard. The process involves three steps: language model training (pre-training), language model fine-tuning, and classifier fine-tuning. The language model is initially trained on a large dataset (like Wikipedia) to predict the next word in a sentence. Fine-tuning is then done on a more specific dataset related to the final task. Classifier fine-tuning refines the model further, often using [reinforcement learning](/posts/An-Introduction-to-Reinforcement-Learning) from human feedback. Howard emphasizes that [language models are a form of compression](/posts/Introduction-to-Large-Language-Models)—they learn a lot about the world to predict words effectively. While some argue about their limitations, he recommends using GPT-4 as the current best language model. The transcript touches on how GPT-4 can be used for various tasks and the importance of being an effective user of language models. ## Limitations and Capabilities of GPT-4 1. **Examples of GPT-4's Abilities:** - Contrary to claims, GPT-4 can handle certain reasoning tasks effectively. Examples include understanding riddles and logic puzzles. - The transcript presents cases where GPT-4 successfully answers questions that were initially believed to be beyond its capabilities. 2. **Training and Awareness:** - Users are encouraged to understand how GPT-4 was trained. It initially predicts the next word in a sentence, and the training data may not guarantee correctness. - GPT-4 lacks awareness about itself, including its training details, context length, and Transformer architecture. Users cannot ask it about information beyond its knowledge cutoff in September 2021. 3. **Challenges and Limitations:** - GPT-4 struggles with certain types of reasoning problems, especially those involving complex logic puzzles or scenarios where it has been primed to answer in a specific way. - The language model may provide confident but incorrect answers, and users need to be cautious in interpreting its responses. 4. **Custom Instructions and Editing:** - Users can influence GPT-4's responses by providing custom instructions. These instructions can guide the model to produce more accurate and relevant information. - When GPT-4 makes a mistake, the user can edit the input to correct it. Continuous correction is necessary to guide the model in the right direction. 5. **Advanced Data Analysis:** - Advanced Data Analysis is mentioned as a helpful tool to ask more complex questions and extract valuable information from GPT-4. In summary, while GPT-4 demonstrates impressive capabilities, users need to be aware of its limitations and actively guide it to produce accurate and meaningful responses. The transcript emphasizes the importance of understanding the training process and using custom instructions to improve the model's performance. ## AI Applications in Code Writing, Data Analysis & OCR 1. **AI Applications in Code Writing, Data Analysis & OCR:** - The transcript discusses the use of AI, specifically GPT-4, in various applications, including code writing, data analysis, and Optical Character Recognition (OCR). 2. **Example of Code Writing:** - The user shares an example where they wanted to split a document into markdown headings and sought help from GPT-4. The model provided code, and the user iteratively tested and refined it, showcasing the collaborative coding capabilities. 3. **Challenges in Code Interpretation:** - The limitations of AI in handling complex logic are acknowledged. The transcript emphasizes that AI, even in the form of GPT-4, is not a substitute for skilled programmers and may struggle with certain logic and reasoning tasks. 4. **OCR Example:** - The transcript illustrates an OCR example where GPT-4 is used to extract text from an image. It efficiently generates an OCR script based on the uploaded image, demonstrating the model's ability to handle familiar patterns. 5. **Data Analysis and Table Creation:** - The user describes a scenario where they wanted to create a table summarizing pricing information from the OpenAI webpage. GPT-4 is used to generate a markdown table from the selected information, showcasing its data analysis capabilities. 6. **Cost and API Comparison:** - The discussion shifts to the cost of using GPT-4 and the OpenAI API. GPT-4 via ChatGPT has a fixed monthly cost, while the API has a per-token pricing structure. A comparison is made between GPT-4 and GPT 3.5 in terms of cost per token, highlighting the affordability of GPT 3.5. 7. **API Usage for Programmability:** - The advantage of using the OpenAI API is emphasized, as it allows programmable access to GPT models, enabling users to integrate AI capabilities into their applications. ## Practical Tips on Using OpenAI API 1. **Introduction to OpenAI API:** - The transcript discusses the practical applications of the OpenAI API, such as analyzing datasets and automating repetitive tasks. It emphasizes that working with the API is a different way of programming. 2. **Simple Example of API Usage:** - The user demonstrates a basic example of using the OpenAI API by creating a chat completion using GPT-3.5 Turbo. The code involves importing the necessary libraries, creating a system message, and generating a response based on a user query. 3. **Choice between GPT-4 and GPT 3.5 Turbo:** - The user recommends using GPT-4 for more challenging tasks but suggests trying GPT 3.5 Turbo first due to its cost-effectiveness. The cost per token for each model is compared, and the importance of monitoring usage is highlighted. 4. **Follow-Up Conversations:** - The transcript explains how follow-up conversations work with the OpenAI API. The entire conversation is passed back for follow-up queries, allowing users to reference and modify previous interactions seamlessly. 5. **Creating Functions for API Interaction:** - The user demonstrates creating a function to simplify API interactions, making it easier to ask questions and receive responses. The usage of a system prompt and user messages is illustrated in generating responses. 6. **Analogies and Creativity:** - The API's ability to generate creative analogies is showcased. The user provides examples of asking questions related to the meaning of life and money, and the model responds with imaginative analogies. 7. **Awareness of Usage and Rate Limits:** - Users are advised to keep an eye on their API usage to avoid unexpected costs. The transcript also provides code to handle rate limits, ensuring that users stay within the API's usage constraints. 8. **Exploring Additional Capabilities:** - The transcript concludes by mentioning the potential to create a code interpreter within Jupyter using the OpenAI API. It encourages users to spend time experimenting and becoming familiar with the capabilities of the language models. ## Creating a Code Interpreter with Function Calling Certainly, here are the main ideas and key points from this transcript: 1. **Introduction to Function Calling:** - The user introduces the concept of function calling in the OpenAI API, a feature that allows passing custom functions to the model. This is done by providing a function schema, which describes the function's name, parameters, defaults, and requirements. 2. **Creating a Simple Function - Sums:** - A simple Python function called `sums` is created, which adds two numbers. The user emphasizes the importance of providing a clear docstring for the function, as this is crucial for OpenAI models to understand its purpose. 3. **Function Calling in OpenAI API:** - The user demonstrates how to use function calling in the OpenAI API by passing the schema of the `sums` function to `chatcompletion.create`. The model is now aware of this function and can utilize it in responses. 4. **Requesting Calculation Using the Custom Function:** - An example is given where the user asks GPT what six plus three is, instructing it to use the `sums` function. The model responds not with the direct answer but with a request to call the function, providing arguments. 5. **Creating a More Powerful Function - Python Execution:** - A more advanced function called `python` is introduced, which allows the execution of Python code. The user explains that caution is taken to avoid running arbitrary code without verification. 6. **Asking for Factorial Calculation:** - The user demonstrates how to ask GPT to calculate 12 factorial using the `python` function. The response includes a request to call the `python` function with specific arguments. 7. **Executing Python Code and Getting the Result:** - The user calls the `python` function, which imports the math module, calculates the factorial, and returns the result. The process involves additional verification steps to ensure safe execution. 8. **Formatting the Result in Chat Format:** - The user mentions an optional step to format the result in a chat-friendly manner. This involves repeating the inputs to GPT to create a more coherent and chat-like response. In summary, the transcript explores the functionality of function calling in the OpenAI API, starting with a simple function (`sums`) and progressing to a more powerful function (`python`) capable of executing Python code. The importance of safety checks and clear documentation for functions is highlighted throughout the discussion. ## Using Local Language Models & GPU Options 1. **Function Role Response:** - Instead of adding an assistant role response, the user introduces the concept of a function role response. This involves providing a function role response by putting the result obtained from the function. This leads to obtaining prose responses for specific queries, such as the calculation of factorial. 2. **Custom Functions Usage:** - The user emphasizes that even with custom functions like Python, the model can still be queried about non-Python topics. The model will selectively use custom functions if needed, allowing for a combination of model-generated responses and user-provided tools. 3. **Building a Code Interpreter from Scratch:** - The user expresses amazement at having built a code interpreter from scratch using OpenAI. This demonstrates the versatility of OpenAI models and their ability to handle custom functions and execute code. 4. **Benefits of In-House Processing:** - The discussion shifts to the benefits of using local language models on one's computer. Reasons for going in-house include the ability to query proprietary documents, access information beyond the knowledge cutoff date, and create models tailored to specific needs through fine-tuning. 5. **GPU Options for Local Processing:** - To use language models on a local machine, a GPU is required. Various options for obtaining GPUs are discussed, including Kaggle, Colab, GPU server providers, and purchasing GPUs like the GTX 3090. The importance of memory speed and size for language models is highlighted. 6. **Renting GPU Services:** - The user introduces options for renting GPU services from providers like Run Pod, Lambda Labs, and Fast AI. The availability, pricing, and considerations for sensitive tasks are discussed. 7. **Choosing GPUs for Language Models:** - Recommendations for GPU choices are provided, suggesting the GTX 3090 as a cost-effective option for language models due to its memory speed. The trade-offs between GPU models and their memory size/cost implications are discussed. 8. **Using Transformers Library:** - The user mentions the use of the Transformers library from Hugging Face, which offers pre-trained models uploaded to the Hugging Face Hub. There's a leaderboard indicating the best models, making it a valuable resource for the community. ## Fine Tuning Models & Decoding Tokens 1. **Model Selection:** - The user discusses the process of selecting a pre-trained model for fine-tuning, emphasizing the importance of considering model size, evaluation metrics, and reliable leaderboards such as the fast eval leaderboard. 2. **Fine-Tuning Process:** - The user introduces the concept of fine-tuning models to make them more useful for specific tasks. The example provided involves fine-tuning the llama model, which is based on llama too, using the ULMFiT (Universal Language Model Fine-Tuning) approach. 3. **16-Bit vs. 8-Bit Representation:** - The user explains the considerations regarding the representation of weights in models, particularly the choice between 16-bit and 8-bit floating-point numbers. Despite the reduced precision, 8-bit representation can be effective due to discretization. 4. **Tokenization and Decoding:** - The tokenization process is highlighted, showcasing how to use the appropriate tokenizer for the model and decode tokens back to human-readable text. The decoding involves generating text autoregressively using the model. 5. **Performance Optimization:** - Performance optimization techniques are discussed, such as using B float 16 for 16-bit floating-point format, resulting in reduced processing time. Another method mentioned is gptq discretization, which optimizes the model for performance. 6. **Results and Efficiency:** - The user demonstrates the efficiency gains achieved through different representations, showcasing the time reduction achieved by using 16-bit and 8-bit representations. ## Testing and Optimizing Models 1. **Optimizing Model Precision:** - The user discusses the optimization of model precision using methods like B float 16 and gptq, explaining that gptq can be more efficient even with lower internal precision. 2. **Optimized Model Versions:** - The user introduces a version of the model called gptq, which is optimized for performance. It's highlighted that models optimized by experts, like the "bloke," are available on Hugging Face after undergoing the optimization process. 3. **Performance Comparisons:** - The user compares the performance of models using different precision formats, demonstrating that gptq can be faster than 16-bit representations due to reduced memory movement. 4. **Combining Techniques:** - The user combines various optimization techniques, such as tokenization, batch decoding, and gptq precision, to demonstrate the effectiveness of these approaches in creating efficient language models. 5. **Fine-Tuning for Specific Tasks:** - The user emphasizes the importance of fine-tuning models for specific tasks, particularly when using instruction-tuned or RLHF (Reinforcement Learning from Human Feedback) models. The example involves creating a stable Beluga model. 6. **Prompt Formatting:** - The user highlights the significance of [prompt formatting](/posts/What-is-Prompt-Tuning) during instruction tuning and recommends checking the web page for the specific model to obtain the correct prompt format. A function called `make_prompt` is created to generate the correct format for questions. 7. **Scaling Up with Larger Models:** - The user explores the scalability of models, progressing from smaller models to larger ones like the 13B model. The open Orca platypus model is introduced, which is a larger version fine-tuned on multiple datasets. ## Retrieval Augmented Generation 1. **Retrieval Augmented Generation:** - The user introduces the concept of retrieval augmented generation, a technique where documents, such as Wikipedia pages, are retrieved to assist language models in generating more accurate and up-to-date responses. 2. **Web Scraping Wikipedia:** - The user demonstrates the process of scraping information from Wikipedia, in this case, the Wikipedia page of Jeremy Howard, to use as context for generating responses. 3. **Knowledge Cutoff and Model Limitations:** - The user acknowledges the knowledge cutoff in models like GPT-4 and introduces the idea that while models like LLM2 might have more up-to-date information, they still have their own limitations and may provide hallucinated responses. 4. **Sentence Transformer for Similarity:** - The user employs the Sentence Transformer model to calculate similarity between a given question and different documents, helping to determine which document is most relevant for generating a response. 5. **Vector Database and Pre-built Systems:** - The user mentions the use of a vector database, such as H2O GPT, for handling similarity calculations with a large number of documents, making the process more efficient. 6. **Retrieval Augmented Generation in Action:** - A practical example is given where the user uploads documents to H2O GPT and tests the system's ability to generate responses based on context retrieved from those documents. 7. **Challenges and Considerations:** - The user highlights the challenges of retrieval augmented generation, such as the need for careful handling of context, potential limitations in understanding specific follow-up questions, and the importance of choosing relevant documents for context. ## Fine Tuning Models 1. **Introduction to Fine-Tuning:** - The user introduces the concept of fine-tuning as an interesting option where the model can be adapted based on the available documents, enabling it to behave differently. 2. **Use of NoSQL Dataset for Fine-Tuning:** - The user demonstrates an example of fine-tuning using a NoSQL dataset, which includes examples of database schema, questions, and corresponding SQL answers. The goal is to create a tool for generating SQL queries based on English questions. 3. **Hugging Face Datasets Library:** - The user mentions the use of the Hugging Face Datasets library to easily access and load datasets for fine-tuning. This library is compared to the Transformers library, which is used for grabbing models. 4. **Axolotl for Fine-Tuning:** - The user introduces Axolotl, an open-source piece of software that facilitates fine-tuning. Axolotl comes with pre-built examples, including one for LLM2, and the user adapts it for fine-tuning using a SQL dataset. 5. **Accelerated Fine-Tuning Process:** - The user shares that the fine-tuning process took about an hour on their GPU using Axolotl. The resulting model is stored in a "qLaura" directory, with "q" standing for quantized. 6. **Creation of Custom SQL Query:** - The user creates a custom SQL query using the fine-tuned model. The question involves counting competition hosts by theme, and the generated SQL query is presented as an accurate response. 7. **Contextual Information in Fine-Tuning:** - The user emphasizes the use of contextual information in fine-tuning, including providing context such as creating tables, listing competition hosts, and sorting them in ascending order. 8. **Remarkable Results:** - The user expresses satisfaction with the results, considering the generated SQL query as remarkable and demonstrating the potential of fine-tuning for specific use cases, such as creating tools for business users. ## Running Models on Macs 1. **Fine-Tuning SQL Queries:** - The user highlights the exciting idea of fine-tuning models for generating SQL queries based on a schema. The process involves using a NoSQL dataset, and the user expresses satisfaction with the results obtained after an hour of training. 2. **Running Models on Macs:** - The user briefly mentions options for running models on Macs, particularly highlighting mlc and lima.cpp. mlc, in particular, is described as an underappreciated project that allows running language models on various devices, including iPhone, Android, and web browsers. 3. **Demonstration on Mac:** - The user provides a demonstration of running a language model on a Mac using a Python program called "chat." The program imports a discretized 7B model and poses a question about the meaning of life. 4. **Accessibility of Models on Various Platforms:** - The user emphasizes the versatility of mlc, stating that it enables running language models on a wide range of devices, making it a convenient and accessible option. 5. **Implementation on Mac:** - The user showcases a practical implementation on their Mac, demonstrating the ease of using mlc to interact with language models for tasks such as answering questions. ## Llama.cpp & Its Cross Platform Abilities 1. **Running Models on Macs:** - The user demonstrates running a language model on a Mac using a Python program called "chat." The program imports a discretized 7B model, and a question about the meaning of life is posed, showcasing the ability to run models on Macs. 2. **Llama.cpp and Cross-Platform Abilities:** - Llama.cpp is introduced as another option for running models on various platforms, including Macs. It is highlighted that llama.cpp runs on different platforms, and there is a Python wrapper available. The user emphasizes its cross-platform capabilities and ease of use. 3. **gguf Format in Llama.cpp:** - Llama.cpp uses the gguf format, and the user explains how to download the gguf file from Hugging Face. A demonstration of using Llama.cpp to answer a question about the planets of the solar system is provided, indicating its functionality. 4. **Options for Model Usage:** - The user suggests that Python programmers with Nvidia graphics cards might prefer using PyTorch and the Hugging Face ecosystem. However, the user acknowledges that options might change over time, and llama.cpp is evolving rapidly. 5. **Fast AI Discord Channel:** - The user encourages others to join the Fast AI Discord channel, specifically the "generative" channel, to ask questions, seek help, and share experiences. The user emphasizes the collaborative nature of the field and the support available in Discord communities. 6. **Exciting but Early Days:** - The user acknowledges that working with language models is both exciting and early in its development. While it may be a bit frustrating due to early-stage challenges and edge cases, the user finds it to be an exciting time to explore this field. 7. **Conclusion:** - The user expresses enjoyment in working with language models and hopes that the information provided serves as a useful starting point for others embarking on their journey in this field. The transcript concludes with a thank you and farewell.