Fine-Tuning
Why fine-tune
The story: A general doctor goes back to school to become a heart specialist. They pick up the subtle details of heart conditions, diagnose heart problems more accurately, unlearn some outdated habits from general practice, and work faster in a cardiology ward.
In AI/AWS terms: Fine-tuning tailors a general FM to a specific task or domain. It helps to:
- Increase specificity for domain nuances
- Improve accuracy on specialized tasks
- Reduce biases from the original training data
- Boost efficiency within a specific context
For the exam: Fine-tuning adapts a general model to a specific task or domain.
Fine-tuning approaches
The story: Five ways that doctor could train further:
- Practice with a book of "when a patient says X, respond like Y".
- Learn the basics from a book, then have senior doctors rate each of their decisions, and learn from the ratings.
- Spend a year reading only cardiology journals and patient records.
- Use their general medical knowledge as a head start for the new specialty, rather than starting over.
- Keep reading new research every month for the rest of their career.
In AI/AWS terms:
| Doctor's training | Approach | What it does | Good for |
|---|---|---|---|
| "When X, respond like Y" | Instruction tuning | Retrains on prompts paired with desired outputs | Following commands: chatbots, virtual assistants |
| Senior doctors rate decisions | RLHF | Supervised training first, then reinforcement learning with a reward model built from human feedback | Aligning output with human values and preferences |
| A year of cardiology journals | Domain adaptation | Trains on an industry corpus, such as legal documents or medical records | Relevance and accuracy in one domain |
| General knowledge as a head start | Transfer learning | Reuses a model trained for one task as the starting point for another | Efficiency with less additional training |
| Reading new research every month | Continuous pretraining | Keeps feeding the model new data | Staying current with new vocabulary, trends, and research |
Measuring tools like ROUGE and BLEU and techniques like model pruning are not fine-tuning methods, the same way a stethoscope isn't a training course.
For the exam: Know the five approaches. ROUGE, BLEU, and pruning are not fine-tuning methods.
Pre-training data versus fine-tuning data
The story: Medical school means reading an enormous range of books to learn medicine in general. Specialist training means a small stack of carefully chosen heart cases. For the specialty, ten excellent cases beat a thousand random ones.
In AI/AWS terms:
| Pre-training (medical school) | Fine-tuning (specialist training) |
|---|---|
| Massive, diverse, broad coverage | Focused on the task |
| Built to generalize | Highly relevant to the desired outputs |
| Quantity | Quality over quantity: small, well-curated sets can go a long way |
For the exam: Fine-tuning data: quality over quantity.
Key steps to prepare fine-tuning data
The story: Preparing the specialist's case files: pick only the relevant cases, make sure every diagnosis written on them is correct (a wrong diagnosis teaches the wrong lesson), follow patient privacy law, make sure the cases cover all kinds of patients, and add comments from senior doctors.
In AI/AWS terms:
- Data curation: rigorous selection of only relevant data
- Labeling: accurate, relevant labels are paramount because they drive specialization in the target domain
- Governance and compliance with industry regulations
- Representativeness and bias checking
- Feedback integration: user or expert feedback, especially for RLHF
For the exam: Accurate labels are the most important part of fine-tuning data.