Fixing LLM Hallucinations with Retrieval Augmentation in LangChain #6
Large Language Models (LLMs) have a data freshness problem. Even some of the most powerful models, like ChatGPT's gpt-3.5-turbo and GPT-4, have no idea about recent events. The world, according to LLMs, is frozen in time. They only know the world as it appeared through their training data. So, how do we handle this problem? We can use retrieval augmentation. This technique allows us to retrieve relevant information from an external knowledge base and give that information to our LLM. The external knowledge base is our "window" into the world beyond the LLM's training data. In this video, we will learn all about implementing retrieval augmentation for LLMs using LangChain and the Pinecone vector database. 🌲 Pinecone article: https://pinecone.io/learn/langchain-retrieval-augmentation 📌 Code notebook: https://github.com/pinecone-io/examples/blob/master/learn/generation/langchain/handbook/05-langchain-retrieval-augmentation.ipynb 👋🏼 NLP + LLM Consulting: https://aurelio.ai 👾 Discord: https://discord.gg/c5QtDB9RAP 🎙️ Support me on Patreon: https://patreon.com/JamesBriggs 🤖 70% Discount on the NLP With Transformers in Python course: https://bit.ly/3DFvvY5 🎉 Subscribe for Article and Video Updates! https://jamescalam.medium.com/subscribe https://medium.com/@jamescalam/membership 00:00 Hallucination in LLMs 01:32 Types of LLM Knowledge 03:08 Data Preprocessing with LangChain 09:54 Creating Embeddings with OpenAI's Ada 002 13:14 Creating the Pinecone Vector Database 16:57 Indexing Data into Our Database 20:27 Querying with LangChain 23:07 Generative Question-Answering with LangChain 25:27 Adding Citations to Generated Answers 28:42 Summary of Retrieval Augmentation in LangChain #artificialintelligence #langchain #nlp #openai #deeplearning #chatgpt
