Skip to content
This repository has been archived by the owner on Nov 16, 2023. It is now read-only.

Allow defaulting the preferred queue based on information in the request #73

Open
iamwillbar opened this issue Mar 16, 2017 · 0 comments

Comments

@iamwillbar
Copy link
Member

Issue #72 is a prerequisite for this.

In the configuration for a crawler you should be able to specify patterns that will set the preferred queue for a request based on information in the request. The two scenarios we have so far are:

  1. Changing the priority based on the type of request
  2. Changing the priority based on the URL of the request

For example, you could imaging a configuration that looks like this:

priorityMappings: [
 {"url": "contoso", "preferredQueue": "later"},
 {"type": "Repo", "preferredQueue": "soon"},
 {"type": "Repo", "url": "(hello|world)", "preferredQueue": "immediate"}
]

When a request is being queued, if it doesn't already specify a preferred queue in the request, then the priority mappings would be evaluated in order, and the first one where all of the criteria match would define the preferred queue for that request. URL matching must support regular expressions.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant