Develop Solutions on Intel® Gaudi® AI Accelerators

Run and Fine-Tune

Fine-tuning a Simple GPT Model 

Start with a simple example of fine-tuning from the Hugging Face* language modeling page. This is using the wikitext dataset to fine-tune the gpt2 model. The fine-tuning of this model takes only a few minutes and the fine-tuned model output is placed in the test_clm folder. 

python run_clm.py \
--model_name_or_path gpt2 \
--dataset_name wikitext \
--dataset_config_name wikitext-2-raw-v1 \
--per_device_train_batch_size 4 \
--per_device_eval_batch_size 4 \
--do_train \
--do_eval \
--overwrite_output_dir \
--report_to none \
--output_dir ./test-clm \
--gaudi_config_name Habana/gpt2 \
--use_habana \
--use_lazy_mode \
--use_hpu_graphs \
--throughput_warmup_steps 3

Fine-tuning the Llama 3 70B Model  

Once simple fine-tuning is complete, start running the full Llama 3 70 model for fine-tuning. Since the Llama 3 70B is a large model, employ the DeepSpeed* library to more efficiently manage the memory usage of the local HBM memory on each Intel Gaudi card. This example also deploys some additional techniques for fine-tuning:  

• Parameter Efficient fine-tuning (PEFT) is a strategy for adapting large pre-trained language models to specific tasks.  Instead of fine-tuning the entire pre-trained model, PEFT adds a task-specific layer or a few task-specific layers on top of the pre-trained model. These additional layers are relatively smaller and have fewer parameters compared to the base model.

• DeepSpeed significantly optimizes training efficiency, reducing both computational and memory requirements. It enables the handling of extremely large models by providing advanced parallelism techniques and memory optimization strategies

• Flash attention is used to reduce memory usage and enhance computational speed through a fused implementation. This includes the use of the FusedSDPA (Scaled Dot Product Attention) applies similar principles to the Intel Gaudi processor environment, optimizing the scaled dot product attention function with reduced memory usage and faster performance while maintaining compatibility with standard PyTorch* functionality.

• Setting epochs = 2; this is enough to ensure that the training loss is below 1.0, running any more epoch is not needed. 

PT_HPU_MAX_COMPOUND_OP_SIZE=10 DEEPSPEED_HPU_ZERO3_SYNC_MARK_STEP_REQUIRED=1 \
python3 ../gaudi_spawn.py --use_deepspeed --world_size 8 run_lora_clm.py \
--model_name_or_path meta-llama/Meta-Llama-3-70B-Instruct \
--deepspeed llama2_ds_zero3_config.json \
--dataset_name tatsu-lab/alpaca \
--bf16 True \
--output_dir ./llama3_fine_tuning_output \
--num_train_epochs 2 \
--max_seq_len 2048 \
--per_device_train_batch_size 10 \
--per_device_eval_batch_size 10 \
--gradient_checkpointing \
--evaluation_strategy epoch \
--eval_delay 2 \
--save_strategy no \
--learning_rate 0.0018 \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--dataset_concatenation \
--attn_softmax_bf16 True \
--do_train \
--do_eval \
--use_habana \
--use_lazy_mode \
--pipelining_fwd_bwd \
--throughput_warmup_steps 3 \
--report_to none \
--lora_rank 4 \
--lora_target_modules "q_proj" "v_proj" "k_proj" "o_proj" \
--validation_split_percentage 4 \
--use_flash_attention True \
--flash_attention_causal_mask True

The result of the run shows that the fine-tuning of the model required only 38 minutes and achieved 2.2 samples (or sentences) per second. 

***** train metrics ***** 
  epoch                       =        2.0 
  max_memory_allocated (GB)   =      94.53 
  memory_allocated (GB)       =      27.15 
  total_flos                  =  1037280GF 
  total_memory_available (GB) =      94.62 
  train_loss                  =     1.1525 
  train_runtime               = 0:38:47.30 
  train_samples_per_second    =      2.221 
  train_steps_per_second      =      0.028 

The output of the run is in the llama3_fine_tuning_output folder. The full model is the adapter_model.safetensors which contains the additional weights generated by the parameter efficient fine-tuning. These weights can used for inference.