add support for ingesting content from websites, audio files, YouTube, etc. #750
+430
−53
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This update include adding support for multiple input formats, improving documentation, and updating dependencies.
Major Enhancements:
Support for Multiple Input Formats:
ingestion.py
) that supports PDF, website, YouTube, and audio file formats. This module includes classes likePDFIngestor
,WebsiteIngestor
,YouTubeIngestor
, andAudioIngestor
to handle different types of input sources.README.md
with detailed usage examples for ingesting content from various sources, including code snippets for each input type.Documentation Improvements:
recipes/quickstart/NotebookLlama/README.md
with step-by-step instructions, updated terminology, and added links to relevant resources.Dependency Updates:
requirements.txt
to include new dependencies likelangchain
,openai-whisper
, andyoutube-dl
, and updated versions for existing dependencies.Detailed Changes:
Ingestion Module:
ingestion.py
: Added support for multiple input formats with classes for PDF, website, YouTube, and audio ingestion.Documentation:
README.md
: Added sections for supported input formats and usage examples.recipes/quickstart/NotebookLlama/README.md
: Improved the quickstart guide with detailed steps and updated terminology.Dependencies:
requirements.txt
: Updated and added new dependencies to support the new ingestion functionalities.