222 lines
12 KiB
Markdown
222 lines
12 KiB
Markdown
# All-in-RAG | Large Model Application Development Practice: RAG Technology Full-Stack Guide
|
||
|
||
<div align='center'>
|
||
<img src="./docs/logo.svg" alt="All-in-RAG Logo" width="70%">
|
||
</div>
|
||
|
||
<div align="center">
|
||
<h2>🔍 Retrieval-Augmented Generation (RAG) Technology Full-Stack Guide</h2>
|
||
<p><em>From theory to practice, from basics to advanced, build your RAG technology system</em></p>
|
||
</div>
|
||
|
||
<div align="center">
|
||
<img src="https://img.shields.io/github/stars/datawhalechina/all-in-rag?style=for-the-badge&logo=github&color=ff6b6b" alt="GitHub stars"/>
|
||
<img src="https://img.shields.io/github/forks/datawhalechina/all-in-rag?style=for-the-badge&logo=github&color=4ecdc4" alt="GitHub forks"/>
|
||
<img src="https://img.shields.io/badge/Python-3.12.7-blue?style=for-the-badge&logo=python&logoColor=white" alt="Python"/>
|
||
</div>
|
||
|
||
<div align="center">
|
||
<a href="https://datawhalechina.github.io/all-in-rag/#/en/">
|
||
<img src="https://img.shields.io/badge/📖_Online_Reading-Get_Started-success?style=for-the-badge&logoColor=white" alt="Online Reading"/>
|
||
</a>
|
||
<a href="README.md">
|
||
<img src="https://img.shields.io/badge/🌍_中文-Version-blue?style=for-the-badge&logoColor=white" alt="Chinese Version"/>
|
||
</a>
|
||
<a href="https://github.com/datawhalechina">
|
||
<img src="https://img.shields.io/badge/💬_Discussion-Join_Us-purple?style=for-the-badge&logoColor=white" alt="Discussion"/>
|
||
</a>
|
||
</div>
|
||
|
||
<div align="center">
|
||
<br>
|
||
<table>
|
||
<tr>
|
||
<td align="center">🎯 <strong>Systematic Learning</strong><br>Complete RAG technology system</td>
|
||
<td align="center">🛠️ <strong>Hands-on Practice</strong><br>Rich project examples</td>
|
||
<td align="center">🚀 <strong>Production Ready</strong><br>Engineering best practices</td>
|
||
<td align="center">📊 <strong>Multimodal Support</strong><br>Text + Image retrieval</td>
|
||
</tr>
|
||
</table>
|
||
</div>
|
||
|
||
## Project Introduction（[中文](README.md) | English）
|
||
|
||
This project is a comprehensive RAG (Retrieval-Augmented Generation) technology full-stack tutorial for large model application developers. It aims to help developers master RAG application development skills based on large language models through systematic learning paths and hands-on practice projects, building production-grade intelligent Q&A and knowledge retrieval systems.
|
||
|
||
**Main content includes:**
|
||
|
||
1. **RAG Technology Fundamentals**: In-depth introduction to RAG core concepts, technical principles, and application scenarios
|
||
2. **Complete Data Processing Pipeline**: From data loading, cleaning to text chunking - the complete data preparation process
|
||
3. **Index Construction and Optimization**: Vector embedding, multimodal embedding, vector database construction and index optimization techniques
|
||
4. **Advanced Retrieval Techniques**: Hybrid retrieval, query construction, Text2SQL and other advanced retrieval technologies
|
||
5. **Generation Integration and Evaluation**: Formatted generation, system evaluation and optimization methods
|
||
6. **Project Practice**: Complete RAG application development practice from basic to advanced
|
||
|
||
## Project Significance
|
||
|
||
With the rapid development of large language models, RAG technology has become the core technology for building intelligent Q&A systems and knowledge retrieval applications. However, existing RAG tutorials are often scattered and lack systematicity, making it difficult for beginners to form a complete understanding of the technical system.
|
||
|
||
Starting from practice and combining the latest RAG technology development trends, this project builds a complete RAG learning system to help developers:
|
||
- Systematically master the theoretical foundation and practical skills of RAG technology
|
||
- Understand the complete architecture of RAG systems and the role of each component
|
||
- Develop the ability to independently develop RAG applications
|
||
- Master evaluation and optimization methods for RAG systems
|
||
|
||
## Target Audience
|
||
|
||
**This project is suitable for the following groups:**
|
||
- Developers with Python programming foundation who are interested in RAG technology
|
||
- AI engineers who want to systematically learn RAG technology
|
||
- Product developers who want to build intelligent Q&A systems
|
||
- Researchers with learning needs for retrieval-augmented generation technology
|
||
|
||
**Prerequisites:**
|
||
- Master Python basic syntax and usage of common libraries
|
||
- Ability to use Docker simply
|
||
- Understanding of basic LLM concepts (recommended but not required)
|
||
- Basic Linux command line operation skills
|
||
|
||
## Project Highlights
|
||
|
||
1. **Systematic Learning Path**: From basic concepts to advanced applications, building a complete RAG technology learning system
|
||
2. **Theory and Practice Combined**: Each chapter includes theoretical explanation and code practice to ensure learning and application
|
||
3. **Multimodal Support**: Covers not only text RAG, but also multimodal embedding and retrieval technologies
|
||
4. **Engineering-Oriented**: Focus on engineering problems in practical applications, including performance optimization, system evaluation, etc.
|
||
5. **Rich Practical Projects**: Provides multiple practical projects from basic to advanced to help consolidate learning outcomes
|
||
|
||
## Content Outline
|
||
|
||
### Part I: RAG Fundamentals
|
||
|
||
**Chapter 1 Unlocking RAG** [📖 View Chapter](./docs/en/chapter1)
|
||
1. [x] [RAG Introduction](./docs/en/chapter1/01_RAG_intro.md) - RAG technology overview and application scenarios
|
||
2. [x] [Preparation](./docs/en/chapter1/02_preparation.md) - Environment configuration and preparation
|
||
3. [x] [Four Steps to Build RAG](./docs/en/chapter1/03_get_start_rag.md) - Quick start with RAG development
|
||
|
||
**Chapter 2 Data Preparation** [📖 View Chapter](./docs/en/chapter2)
|
||
1. [x] [Data Loading](./docs/en/chapter2/04_data_load.md) - Multi-format document processing and loading
|
||
2. [x] [Text Chunking](./docs/en/chapter2/05_text_chunking.md) - Text segmentation strategies and optimization
|
||
|
||
### Part II: Index Construction and Optimization
|
||
|
||
**Chapter 3 Index Construction** [📖 View Chapter](./docs/en/chapter3)
|
||
1. [x] [Vector Embedding](./docs/en/chapter3/06_vector_embedding.md) - Detailed explanation of text vectorization technology
|
||
2. [x] [Multimodal Embedding](./docs/en/chapter3/07_multimodal_embedding.md) - Image-text multimodal vectorization
|
||
3. [x] [Vector Database](./docs/en/chapter3/08_vector_db.md) - Vector storage and retrieval systems
|
||
4. [x] [Milvus Practice](./docs/en/chapter3/09_milvus.md) - Milvus multimodal retrieval practice
|
||
5. [x] [Index Optimization](./docs/en/chapter3/10_index_optimization.md) - Index performance tuning techniques
|
||
|
||
### Part III: Advanced Retrieval Techniques
|
||
|
||
**Chapter 4 Retrieval Optimization** [📖 View Chapter](./docs/en/chapter4)
|
||
1. [x] [Hybrid Search](./docs/en/chapter4/11_hybrid_search.md) - Dense + sparse retrieval fusion
|
||
2. [x] [Query Construction](./docs/en/chapter4/12_query_construction.md) - Intelligent query understanding and construction
|
||
3. [x] [Text2SQL](./docs/en/chapter4/13_text2sql.md) - Natural language to SQL query
|
||
4. [x] [Query Rewriting and Routing](./docs/en/chapter4/14_query_rewriting.md) - Query optimization strategies
|
||
5. [x] [Advanced Retrieval Techniques](./docs/en/chapter4/15_advanced_retrieval_techniques.md) - Advanced retrieval algorithms
|
||
|
||
### Part IV: Generation and Evaluation
|
||
|
||
**Chapter 5 Generation Integration** [📖 View Chapter](./docs/en/chapter5)
|
||
1. [x] [Formatted Generation](./docs/en/chapter5/16_formatted_generation.md) - Structured output and format control
|
||
|
||
**Chapter 6 RAG System Evaluation** [📖 View Chapter](./docs/en/chapter6)
|
||
1. [x] [Evaluation Introduction](./docs/en/chapter6/18_system_evaluation.md) - RAG system evaluation methodology
|
||
2. [x] [Evaluation Tools](./docs/en/chapter6/19_common_tools.md) - Common evaluation tools and metrics
|
||
|
||
### Part V: Advanced Applications and Practice
|
||
|
||
**Chapter 7 Advanced RAG Architecture (Extended Elective)** [📖 View Chapter](./docs/en/chapter7)
|
||
|
||
1. [x] [Knowledge Graph-based RAG](./docs/en/chapter7/20_kg_rag.md)
|
||
|
||
**Chapter 8 Project Practice I (Basic)** [📖 View Chapter](./docs/en/chapter8)
|
||
1. [x] [Environment Configuration and Project Architecture](./docs/en/chapter8/01_env_architecture.md)
|
||
2. [x] [Data Preparation Module Implementation](./docs/en/chapter8/02_data_preparation.md)
|
||
3. [x] [Index Construction and Retrieval Optimization](./docs/en/chapter8/03_index_retrieval.md)
|
||
4. [x] [Generation Integration and System Integration](./docs/en/chapter8/04_generation_sys.md)
|
||
|
||
**Chapter 9 Project Practice I Optimization (Elective)** [📖 View Chapter](./docs/en/chapter9)
|
||
|
||
[🍽️ Project Demo](https://github.com/FutureUnreal/What-to-eat-today)
|
||
1. [x] [Graph RAG Architecture Design](./docs/en/chapter9/01_graph_rag_architecture.md)
|
||
2. [x] [Graph Data Modeling and Preparation](./docs/en/chapter9/02_graph_data_modeling.md)
|
||
3. [x] [Milvus Index Construction](./docs/en/chapter9/03_index_construction.md)
|
||
4. [x] [Intelligent Query Routing and Retrieval Strategy](./docs/en/chapter9/04_intelligent_query_routing.md)
|
||
|
||
**Chapter 10 Project Practice II (Elective)** [📖 View Chapter](./docs/en/chapter10) *In Planning*
|
||
|
||
## Directory Structure
|
||
|
||
```
|
||
all-in-rag/
|
||
├── docs/ # Tutorial documentation
|
||
├── code/ # Code examples
|
||
├── data/ # Sample data
|
||
├── models/ # Pre-trained models
|
||
└── README.md # Project description
|
||
```
|
||
|
||
## Practical Project Showcase
|
||
|
||
### Chapter 8 Project I:
|
||
|
||
|
||
|
||
### Chapter 9 Project I (Graph RAG Optimization):
|
||
|
||
|
||
|
||
### Chapter 10 Project II:
|
||
|
||
## Acknowledgments
|
||
|
||
**Core Contributors**
|
||
- [Yin Dalv - Project Lead](https://github.com/FutureUnreal) (Project initiator and main contributor)
|
||
|
||
### Special Thanks
|
||
- Thanks to [@Sm1les](https://github.com/Sm1les) for help and support on this project
|
||
- Thanks to all developers who contributed to this project
|
||
- Thanks to the open source community for providing excellent tools and framework support
|
||
- Special thanks to the following developers who contributed to the tutorial!
|
||
|
||
[](https://github.com/datawhalechina/all-in-rag/graphs/contributors)
|
||
|
||
*Made with [contrib.rocks](https://contrib.rocks).*
|
||
|
||
## Contributing
|
||
|
||
We welcome all forms of contributions, including but not limited to:
|
||
|
||
- 🚨 **Bug Reports**: Please submit [Issues](https://github.com/datawhalechina/all-in-rag/issues) if you find problems
|
||
- 💭 **Feature Suggestions**: Welcome to discuss good ideas in [Discussions](https://github.com/datawhalechina/all-in-rag/discussions)
|
||
- 📚 **Documentation Improvement**: Help improve documentation content and example code
|
||
- ⚡ **Code Contributions**: Submit [Pull Requests](https://github.com/datawhalechina/all-in-rag/pulls) to improve the project
|
||
|
||
## Star History
|
||
|
||
[](https://datawhalechina.github.io/members-visualization/repo-badge?repo=all-in-rag)
|
||
|
||
<div align="center">
|
||
<p>If this project helps you, please give us a ⭐️</p>
|
||
<p>Let more people discover this project (Food protection? Bring it on!)</p>
|
||
</div>
|
||
|
||
|
||
|
||
## About Datawhale
|
||
|
||
<div align='center'>
|
||
<img src="https://raw.githubusercontent.com/datawhalechina/pumpkin-book/master/res/qrcode.jpeg" alt="Datawhale" width="30%">
|
||
<p>Scan the QR code to follow Datawhale WeChat Official Account for more quality open source content</p>
|
||
</div>
|
||
|
||
---
|
||
|
||
## License
|
||
|
||
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://img.shields.io/badge/license-CC%20BY--NC--SA%204.0-lightgrey" /></a>
|
||
|
||
This work is licensed under a [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-nc-sa/4.0/).
|
||
|
||
--- |