[FEATURE REQUEST] Queue Adapters (eg: SQS, GCP Pub/Sub etc)

[yoshdog]

Is your feature request related to a problem? Please describe.
Would be great if we could configure different queue backends like SQS, GCP Pub/Sub etc...

Describe the solution you'd like
Ability to use a different queue backend other then redis.

Additional context
Alternative packages that support this: https://github.com/RichardKnop/machinery

[anupriya17]

A Pubsub Interface like this will help, I just copied the redis pubsub methods:

type Message struct {
	Channel      string
	Payload      string
}

type PubSub interface {
	Subscribe(ctx context.Context, channels ...string) error
	PSubscribe(ctx context.Context, patterns ...string) error
	SSubscribe(ctx context.Context, channels ...string) error
	Unsubscribe(ctx context.Context, channels ...string) error
	PUnsubscribe(ctx context.Context, patterns ...string) error
	SUnsubscribe(ctx context.Context, channels ...string) error
	Ping(ctx context.Context, payload ...string) error
	ReceiveTimeout(ctx context.Context, timeout time.Duration) (interface{}, error)
	Receive(ctx context.Context) (interface{}, error)
	ReceiveMessage(ctx context.Context) (*Message, error)
	Channel(opts ...ChannelOption) <-chan *Message
	Close() error
}

type ChannelOption func(*channelOptions)

type channelOptions struct {
	size        int
	maxInterval int64
}

func WithChannelSize(size int) ChannelOption {
	return func(opts *channelOptions) {
		opts.size = size
	}
}

func WithMaxInterval(maxInterval int64) ChannelOption {
	return func(opts *channelOptions) {
		opts.maxInterval = maxInterval
	}
}

[yoshdog]

ok seems like there is already a broker interface
https://github.com/hibiken/asynq/blob/master/internal/base/base.go#L715-L755

So we would just need to implement one for every type... eg: SQS, RabbitMQ, PUB SUB etc...

[anupriya17]

Thanks. broker interface depends on redis pubsub currently.
https://github.com/hibiken/asynq/blob/master/internal/base/base.go#L751

IMO to decouple pub/sub to its own interface. We don't need all the methods

[anhoder]

Looking forward to this feature👀

[bowenli86]

+1

[zandwang]

It looks like there are already some relevant PRS？
#577

[djragsdale]

Based on #577 and #667, it looks like PRs for other brokers are being declined. What doesn't get asked is why one would choose asynq over "other libraries out there that allow you to bring your own broker for lesser features". I'll summarize the reasons I was hoping to use asynq, even though I would prefer to use SQS as the broker:

• Some alternatives, such as https://github.com/RichardKnop/machinery, are not actively maintained. It had releases in March 2024, but before that had not been updated since June 2021. And per the author in Is this project still maintained? RichardKnop/machinery#790 (comment): "It is not actively maintained anymore."
• Many alternatives do not offer retries. This is an incredible feature, and IMO a must-have.
• Many alternatives do not bring a Web UI. Of course it is possible to develop a Web UI that can read and write to the message broker, but this could be 10x the code written for just the workers.
• Many alternatives do not support queue priorities. For example, I have jobs that are actively created by users, jobs which are scheduled to run overnight, and jobs that involve data cleanup/synchronization. Those are perfect fits for Critical, Default, and Low priority queues.
• Many alternatives do not support timeouts. This feels simple, but a surprising amount of options lack this necessary feature. This is really important to enforce one job doesn't lock up an entire queue, or continue to hold onto memory after an OOM or overflow error.

I honestly haven't seen any option except Asynq that checks all those boxes.

The natural next question is "How does a maintainer with limited availability support other brokers?" Simple: they don't. Decouple. If there is an Asynq feature that Redis offers which can't be achieved with SQS, GCP Pub/Sub, or other brokers, I haven't seen the technical limitation. If there is one, please communicate it. Otherwise, I would happily contribute to the project similar to the aforementioned PRs so that a bring-your-own broker is achievable.