Fine-Tune vs RAG: Choosing the Right Approach for LLMs

Fine-Tune vs RAG: The following question has been coming up a lot in the community. Should I RAG or Fine-tune? I think the question itself can be misleading. RAG and fine-tuning are very different operations. My expectation is they are not interchangeable, I'm glad Jerry is testing.: https://twitter.com/jerryjliu0/status/1701264465977856399 Fine-tuning is typically supported through a frozen model weights and LORA, low rank matrix training. LORA should not be good at storing a lot of new facts, its a small matrix, shorter training runs and smaller dataset examples. If you want to connect your data to an LLM, you should be using RAG. If you want to improve the output of an LLM on a very specific task, say structured data extraction of certain fields, probably Fine-tune is a good pick.