Fine-Tuning LLaMA 2

A hands-on example of fine-tuning a Llama model

Author

Sinan Ozdemir
November 27, 2023

Fine-tuning Llama 2 - a hands-on example

Hello everyone! Today, I am diving deep with a new (relatively) simple notebook to help people fine-tune Llama-2, Meta’s latest open source large language model on Hugging Face. Here are some fascinating insights from this journey (the notebook link is at the bottom of the post 🙂).

Dataset and Model: Our dataset for this experiment was the guanaco-llama2-1k from HuggingFace, comprising instructional texts. The model of choice was NousResearch/Llama-2-7b-hf, the 7 billion parameter model. Unlike my previous venture with App Reviews, this dataset explores a different facet of language understanding, focusing on instructional text comprehension and generation.

Feel free to change up the dataset/model but of course that might require some code changes. It will be easier to change out the dataset/conversation format because we’re using the blank slate non-aligned version of Llama 7b that has no expectations for conversation format.

Key Takeaways:

  1. Efficient Fine-Tuning with LoRA: The LoRA (Low-Rank Adaptation) technique allowed for efficient fine-tuning of the LLaMA 2 model without the need for extensive computational resources. This approach is a nod to the evolving practices in AI, where efficiency is becoming as crucial as effectiveness.

  2. Quantization and Performance: Implementing BitsAndBytes for model quantization significantly reduced the memory footprint. This optimization meant we could run a larger model on the same hardware, a critical factor in practical AI applications.

  3. Improved Responses Post Fine-Tuning: The difference in the model's responses pre and post fine-tuning was stark. This improvement underscores the impact of fine-tuning on model performance, especially for specific use cases. Check out this before and after of asking the model "Who is Leonardo Da Vinci?"

Before fine-tuning: the model has no idea how to answer questions

After fine-tuning: the model now knows how to answer questions, even when asked about items not found in the instructional dataset

  1. Cost vs. Performance: Similar to my previous analysis with fine-tuning OpenAI models, the cost-effectiveness of fine-tuning LLaMA 2 was noteworthy. While not as resource-intensive as GPT-3.5, the performance gains were substantial, offering a compelling middle ground between efficiency and effectiveness.

  2. Masking Loss for Targeted Learning: We employed a custom data collator, DataCollatorForCompletionOnlyLM to selectively mask the loss calculation to focus only on the model's responses given a conversation, effectively ignoring irrelevant parts of the input. By doing so, we ensured that the model's learning was concentrated on generating accurate and relevant responses, improving its efficiency and effectiveness in understanding and replying to instructional texts.

Conclusion: The world of AI and language models continues to be a thrilling landscape of endless possibilities and learning. Fine-tuning LLaMA 2 has been an enriching experience, revealing the importance of model efficiency, the power of specific optimizations, and the constant need to balance cost with performance. As always, for those keen to dive deeper, the updated notebook on GitHub awaits. Until our next AI adventure, happy coding!

Next Steps: RLHF/RLAIF and Custom Pre-Training

  1. Implementing RLHF/RLAIF: We could use Reinforcement Learning from Human Feedback (RLHF) to enhance LLaMA 2's performance by fine-tuning it based on interactive feedback, aiming for responses that better align with human expectations.

  2. Custom Pre-Training Corpus: We can also fine-tune LLaMA 2 on a custom corpus tailored to specific domains, significantly enhancing its expertise and accuracy in niche areas without losing its versatile applicability.

Notebook link please!

Here is the notebook and have fun!