ubiai deep learning
Contact us
ubiai deep learning
Screenshot 2024-05-28 at 12.33.00 PM

How to Fine-Tune LLaMA3 Using AutoTrain: A Step-by-Step Guide

May 29th, 2024

LLaMA3 (Large Language Model) is a state-of-the-art language model that has shown remarkable performance in various natural language processing (NLP) tasks. Its ability to generate human-like text, understand context, and perform tasks such as translation, summarization, and question-answering makes it a valuable tool for many applications. By leveraging the power of large-scale training on diverse datasets, LLaMA3 can provide high-quality outputs that are useful in both academic and industrial settings.

Importance of Fine-Tuning

Fine-tuning is the process of taking a pre-trained model and adapting it to a specific task or dataset. While LLaMA3 comes pre-trained on a large corpus of text, fine-tuning allows it to specialize in particular domains or tasks, thereby improving its performance and relevance. Fine-tuning is crucial because it helps the model to learn the nuances and specifics of a given task, which the general pre-training may not cover comprehensively. This customization is essential for applications that require high precision and relevance. 

What’s Hugginface AutoTrain

AutoTrain is a powerful framework that simplifies the process of training and fine-tuning large language models. It automates many of the complex and tedious steps involved in model training, such as data preprocessing, hyperparameter tuning, and model evaluation. By using AutoTrain, developers and researchers can focus more on the application and less on the intricacies of model training. AutoTrain is designed to be user-friendly and efficient, making it accessible to both beginners and experts in the field of machine learning.

Llama 3: Embracing the Open-Source

Meta demonstrates its dedication to ethical AI by unveiling Llama 3 as an open-source model. With Llama 3, developers and researchers gain access to its complete architecture, training techniques, and model specifications under permissive licenses. This transparent approach not only encourages collaboration but also invites external review, bolstering accountability in the realm of AI advancement.

Dataset Preparation

For fine-tuning, you need a well-prepared dataset. The dataset should be relevant to the task you want to fine-tune the model for. Here’s how you can prepare your dataset: 

 

  1. Format and Structure: Ensure your dataset is in a structured format (e.g., CSV, JSON). Each entry should have the input text and the corresponding output or label. 
  2. Example Datasets: You can use publicly available datasets from sources like Kaggle, Hugging Face Datasets, or UCI Machine Learning Repository for practice. For instance, if you’re fine-tuning for sentiment analysis, you might use the IMDb movie reviews dataset. 

Setting Up the Environment

To fine-tune the LLM with Python API, we need to install the Python package, which you can run using the following code.



				
					!pip install pandas autotrain-advanced -q

These libraries are essential for loading the model, preparing the dataset, and running the fine-tuning process.

Loading and Preparing the Dataset

Loading the Dataset 

Load your dataset using the `datasets` library. Here’s an example of how to load a dataset from a CSV file: 

				
					from datasets import load_dataset
dataset = load_dataset('csv', data_files='path/to/your/dataset.csv')

Data Preprocessing 

Your Dataset must have just one feature called “text”

Configuring the Fine-Tuning Process

Model Configuration 

Configuring the model is the next step. This involves setting up the model parameters and specifying the architecture.

 

Here’s an example configuration for the LLaMA3 model:

				
					import os
project_name = 'UBIAI-finetuned-llama3'
model_name = 'meta-llama/Meta-Llama-3-8B'


#Push to Hub?
#Use these only if you want to push your trained model to a private repo in your Hugging Face Account
#If you don't use these, the model will be saved in Google Colab and you are required to download it manually.
#Please enter your Hugging Face write token. The trained model will be saved to your Hugging Face account.
#You can find your token here: https://huggingface.co/settings/tokens
push_to_hub = False
hf_token = "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"
#repo_id = "username/repo_name"


#Hyperparameters
learning_rate = 2e-4
num_epochs = 3
batch_size = 4
block_size = 1024
trainer = "sft"
warmup_ratio = 0.1
weight_decay = 0.01
gradient_accumulation = 4
mixed_precision = "fp16"
peft = True
quantization = "int4"
lora_r = 16
lora_alpha = 32
lora_dropout = 0.05


os.environ["PROJECT_NAME"] = project_name
os.environ["MODEL_NAME"] = model_name
os.environ["PUSH_TO_HUB"] = str(push_to_hub)
os.environ["HF_TOKEN"] = hf_token
#os.environ["REPO_ID"] = repo_id
os.environ["LEARNING_RATE"] = str(learning_rate)
os.environ["NUM_EPOCHS"] = str(num_epochs)
os.environ["BATCH_SIZE"] = str(batch_size)
os.environ["BLOCK_SIZE"] = str(block_size)
os.environ["WARMUP_RATIO"] = str(warmup_ratio)
os.environ["WEIGHT_DECAY"] = str(weight_decay)
os.environ["GRADIENT_ACCUMULATION"] = str(gradient_accumulation)
os.environ["MIXED_PRECISION"] = str(mixed_precision)
os.environ["PEFT"] = str(peft)
os.environ["QUANTIZATION"] = str(quantization)
os.environ["LORA_R"] = str(lora_r)
os.environ["LORA_ALPHA"] = str(lora_alpha)
os.environ["LORA_DROPOUT"] = str(lora_dropout)

Running the Fine-Tuning Process 

				
					!autotrain llm \
--train \
--model ${MODEL_NAME} \
--project-name ${PROJECT_NAME} \
--data-path /content/data \
--lr ${LEARNING_RATE} \
--batch-size ${BATCH_SIZE} \
--epochs ${NUM_EPOCHS} \
--block-size ${BLOCK_SIZE} \
--warmup-ratio ${WARMUP_RATIO} \
--lora-r ${LORA_R} \
--lora-alpha ${LORA_ALPHA} \
--lora-dropout ${LORA_DROPOUT} \
--weight-decay ${WEIGHT_DECAY} \
--gradient-accumulation ${GRADIENT_ACCUMULATION} \
--quantization ${QUANTIZATION} \
--target-modules q_proj,v_proj \
--mixed-precision ${MIXED_PRECISION} \
$( [[ "$PEFT" == "True" ]] && echo "--peft" ) \
$( [[ "$PUSH_TO_HUB" == "True" ]] && echo "--push-to-hub --token ${HF_TOKEN} --repo-id ${REPO_ID}" )

INFO    || __main__:train:574 – Finished training, saving model…

This will print out the evaluation metrics such as accuracy, precision, recall, and F1 score, allowing you to assess the model’s performance. 

If the fine-tuning process succeeds, we will have a new directory of our fine-tuned model. We would use this directory to test our newly fine-tuned model.

				
					from transformers import AutoModelForCausalLM, AutoTokenizer


model_path = "/content/UBIAI-finetuned-llama3"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path)

Fine-Tuned LLM inference:

				
					input_text = """
YOUR PROMPT HERE !!!
"""
input_ids = tokenizer.encode(input_text, return_tensors="pt")
output = model.generate(input_ids, max_new_tokens = 400)
predicted_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(predicted_text)

Conclusion

Fine-tuning LLaMA3 using AutoTrain simplifies the process of customizing a powerful language model for specific tasks. By leveraging AutoTrain’s automated workflow, you can efficiently prepare your dataset and configure the fine-tuning process, reducing the complexity and expertise required.

This guide covered the importance of fine-tuning, AutoTrain’s capabilities, and the steps for preparing your dataset and configuring the model. By following these steps, you can optimize LLaMA3 for applications like sentiment analysis, translation, and summarization.

Fine-tuning enhances model performance and relevance, ensuring precise and accurate outputs for your specific needs. With your fine-tuned LLaMA3, you can now integrate it into your projects and further evaluate and refine its performance.

What are you waiting for?

Automate your process!

Get in touch
ubiai deep learning
The Services provided are really great, we received a genuine advice and at very reasonable cost. all the work went hassle-free and no complication.

Features

Case Studies

Company

Legal

© 2023 UBIAI Web Services — All rights reserved.
close menu icon

Unlocking the Power of SLM Distillation for Higher Accuracy and Lower Cost​

How to make smaller models as intelligent as larger ones

Recording Date : March 7th, 2025

Unlock the True Potential of LLMs !

close menu icon

Harnessing AI Agents for Advanced Fraud Detection

How AI Agents Are Revolutionizing Fraud Detection

Recording Date : February 13th, 2025

Unlock the True Potential of LLMs !

close menu icon

Thank you for registering!

Check your email for the live demo details

see you on February 19th

While you’re here, discover how you can use UbiAI to fine-tune highly accurate and reliable AI models!

Explore UbiAI
close menu icon

Thank you for registering!

Check your email for webinar details

see you on March 5th

While you’re here, discover how you can use UbiAI to fine-tune highly accurate and reliable AI models!

Explore UbiAI
close menu icon

Fine Tuning LLMs on Your Own Dataset ​

Fine-Tuning Strategies and Practical Applications

Recording Date : January 15th, 2025

Unlock the True Potential of LLMs !