Skip to content
This repository was archived by the owner on Nov 16, 2023. It is now read-only.
This repository was archived by the owner on Nov 16, 2023. It is now read-only.

Allow defaulting the preferred queue based on information in the request #73

Open
@iamwillbar

Description

@iamwillbar

Issue #72 is a prerequisite for this.

In the configuration for a crawler you should be able to specify patterns that will set the preferred queue for a request based on information in the request. The two scenarios we have so far are:

  1. Changing the priority based on the type of request
  2. Changing the priority based on the URL of the request

For example, you could imaging a configuration that looks like this:

priorityMappings: [
 {"url": "contoso", "preferredQueue": "later"},
 {"type": "Repo", "preferredQueue": "soon"},
 {"type": "Repo", "url": "(hello|world)", "preferredQueue": "immediate"}
]

When a request is being queued, if it doesn't already specify a preferred queue in the request, then the priority mappings would be evaluated in order, and the first one where all of the criteria match would define the preferred queue for that request. URL matching must support regular expressions.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions