Proposal: Support for Stateless Crawler Execution

[neviaumi]

Background

I'm working on integrating Crawlee with Modal Context Protocol where each tool maps to a specific crawler function. In my case, I need two separate crawlers - one for product search and another for product details.

Current Challenges

1. Storage Management: The default MemoryStorageClient writes results to the local filesystem, which isn't suitable for stateless execution.
2. Shared Global Storage: The default Dataset is globally shared, causing product detail crawls to include product search results due to persistent storage.

Current Workaround

I've implemented a temporary solution using custom datasets and manual cleanup:

async def product_search(keyword: str):
    query = urllib.parse.urlencode({"term": keyword})
    request = Request.from_url(f"{BASE_URL}/search?{query}", label="product search")
    dataset = await Dataset.open(name=request.id)

    crawler = ParselCrawler(
        configure_logging=False,
        request_handler=router,
        http_client=HttpxHttpClient(),
    )

    await crawler.run([request])
    result = [item for item in (await dataset.get_data()).items]
    await dataset.drop()  # Manual cleanup
    crawler.stop()
    return result

Proposed Enhancement

I believe the API could be more intuitive by allowing dataset injection through the crawler constructor:

async def product_search(keyword: str):
    query = urllib.parse.urlencode({"term": keyword})
    request = Request.from_url(f"{BASE_URL}/search?{query}", label="product search")
    dataset = await Dataset.open()

    crawler = ParselCrawler(
        configure_logging=False,
        request_handler=router,
        dataset=dataset,  # Inject custom dataset
        http_client=HttpxHttpClient(),
    )

    await crawler.run([request])
    result = [item for item in (await crawler.get_data()).items]
    crawler.stop()
    return result

Questions

1. Is this the right approach for handling stateless crawler execution?
2. Would it make sense to add dataset injection as a feature?
3. Are there any existing best practices for managing storage in stateless scenarios?

Looking forward to your thoughts and suggestions!

[neviaumi]

I have example repo here
https://github.com/neviaumi/demo-in-memory-crawler/blob/main/main.py

[janbuchar]

Hi @neviaumi and thanks for opening this discussion! The Crawlee storage system is currently undergoing a significant refactor - see #1194. With that, you should be able to easily set up a crawler in a way that each MCP "call" (forgive me for not knowing the correct terminology) has its own non-persistent storage (datasets, key-value stores and request queues).

In fact, you can already configure the MemoryStorageClient to not dump anything in the filesystem - see the highlighted code in https://crawlee.dev/python/docs/deployment/gcp-cloud-run-functions, for example.

[neviaumi]

Hi @janbuchar , thanks for sharing the example code !

It make the challenge one fixed.

Looking forward for #1194 get merged ! But as we have workaround for the problem.

I am happy to close this discussion :) Seem your team is actively working on it.