Integrate Crawl4Ai to scrape web content #383
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Blakeinstein
force-pushed
the
feat/web-crawler
branch
4 times, most recently
from
e740a18 to
cafffba
Compare
mnvsk97
force-pushed
the
feat/web-crawler
branch
from
cafffba to
21fa59c
Compare
Blakeinstein
force-pushed
the
feat/web-crawler
branch
2 times, most recently
from
efc3a15 to
d710426
Compare
Blakeinstein
force-pushed
the
feat/web-crawler
branch
2 times, most recently
from
391fedd to
0728967
Compare
Blakeinstein
force-pushed
the
feat/web-crawler
branch
from
0728967 to
72af101
Compare
mnvsk97
approved these changes
Nov 8, 2024
S1LV3RJ1NX
pushed a commit
that referenced
this pull request
Nov 29, 2024
Integrate Crawl4Ai to scrape web content
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changelog
Summary
Refactor's frontend NewDataSource form breaking it into sub components, fixes form validation on required fields. Cleaner state management and separation of abstractions for different flows.
AbstractDataLoader::load_filtered_data is now async this affects all data loaders.
Update web_loader to now use Crawl4Ai to
Fetch sitemap to get a list of URLs to crawl.
Crawl each of the URL using user provided configs (including using model gateway to extract semantic data from the webpage). I wanted to make crawling pages parallel but looks like doing it async cause all urls to timeout. Can revisit this later perhaps with multi-threading.