You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Nov 16, 2023. It is now read-only.
In the configuration for a crawler you should be able to specify patterns that will set the preferred queue for a request based on information in the request. The two scenarios we have so far are:
Changing the priority based on the type of request
Changing the priority based on the URL of the request
For example, you could imaging a configuration that looks like this:
When a request is being queued, if it doesn't already specify a preferred queue in the request, then the priority mappings would be evaluated in order, and the first one where all of the criteria match would define the preferred queue for that request. URL matching must support regular expressions.
The text was updated successfully, but these errors were encountered:
Issue #72 is a prerequisite for this.
In the configuration for a crawler you should be able to specify patterns that will set the preferred queue for a request based on information in the request. The two scenarios we have so far are:
For example, you could imaging a configuration that looks like this:
When a request is being queued, if it doesn't already specify a preferred queue in the request, then the priority mappings would be evaluated in order, and the first one where all of the criteria match would define the preferred queue for that request. URL matching must support regular expressions.
The text was updated successfully, but these errors were encountered: