Is there any way to specify max crawl depth?

[miraclebakelaser]

I want to restrict my crawler so that it only goes 3 levels deep from the starting URL. Is there a built-in way to set a maximum depth, or what’s the best approach to enforce this depth limit in Crawlee?

[vdusek]

Currently, we do not support such option. You can limit the crawling either by setting an enqueueing strategy, see the Crawl website with relative links. Or your use case cannot be addressed with an enqueueing strategy and you specifically need a max crawl depth?

[miraclebakelaser]

The primary use case I have in mind here is to have crawlee collect content from link aggregators (so desired settings would be strategy='all', max_depth=1), e.g. hacker news, metafilter. Sites like hn have api,s but the goal is to write as little site-specific logic as possible. The current enqueueing strategies help, but don't seem to satisfy the full use case.

Taking a step back, this would help llm crawler agents better plan and refine their crawling strategies without having to write a lot of site-specific include/exclude regex strategies (which still wouldn't work to satisfy the above use case).

[vdusek]

Got it, could you please open an issue for it? You can reference this discussion.

Edit: issue #460