Allow to select which errors are retryable in the Retry middleware

[xico42]

Feature request

Description

Add a new configuration option to the retry middleware that would make it possible to customize which types of errors are retryable.

Example use case

At the company I work for, we use a custom error package that allows to specify error severities. We have a convention that "retryable errors" are propagated as "Runtime Severity".

It would be great if I could just reuse the existing Retry middleware and have it run a custom function to
check if the returned errors is retryable or not.

A more concrete use case would be to not retry if the input message is in an invalid format, because even if we retry, the result will always be the same once the msg payload is corrupted somehow.

How it can look like in code

// Default implementation
func allErrorsAreRetryable(err error) bool {
	return true
}

// Retry provides a middleware that retries the handler if errors are returned.
// The retry behaviour is configurable, with exponential backoff and maximum elapsed time.
type Retry struct {
        // existing options ...
	
        // RetryableError returns true whenever an error can be retried
        // by default all errors are retryable.
	RetryableError func(error) bool

}

// Middleware returns the Retry middleware.
func (r Retry) Middleware(h message.HandlerFunc) message.HandlerFunc {
	return func(msg *message.Message) ([]*message.Message, error) {
		producedMessages, err := h(msg)
		if err == nil {
			return producedMessages, nil
		}

                // >>> Custom implementation to check if the error is retryable or not
		if ok := r.RetryableError(err); !ok {
                        // If the error is not retryable we completely skip the middleware implementation and return
                        // whatever the inner handler function returns
			return producedMessages, err
		}

		expBackoff := backoff.NewExponentialBackOff()
		expBackoff.InitialInterval = r.InitialInterval
                // existing implementation ...
}

I would be willing to implement this!