Recurring Tasks questions / workaround for PG/connection.rb "[BUG] Segmentation fault"

[wheee]

Hi all (I guess more specifically @rosa ),

I'm loving the progress being made on SolidQueue over the past year or so!

I had a chance to update SolidQueue and start using recurring tasks as it stands now and I had a few questions and observation.

Observation

When using Recurring Tasks on MacOS with PG, you can/will encounter the following error followed by a huge stacktrace when starting the SolidQueue workers (bin/jobs):

/Users/.../.rbenv/versions/3.3.5/gemsets/.../gems/pg-1.5.9/lib/pg/connection.rb:703: [BUG] Segmentation fault at 0x0000000123be0b04
worker | ruby 3.3.5 (2024-09-03 revision ef084cc8f4) +YJIT [arm64-darwin23]
worker |
worker | -- Crash Report log information --------------------------------------------
worker |    See Crash Report log file in one of the following locations:
worker |      * ~/Library/Logs/DiagnosticReports
worker |      * /Library/Logs/DiagnosticReports
worker |    for more details.
worker | Don't forget to include the above Crash Report log file in bug reports.
worker |
worker | -- Control frame information -----------------------------------------------
worker | c:0117 p:---- s:0691 e:000690 CFUNC  :connect_poll

Weirdly enough, things still work fine after and it seems to happen only upon startup.

After doing some digging, it appears to be a PG bug, see: ged/ruby-pg#538.

The temporary fix is to disable the environment variable "gssencmode", and there was a helpful comment in the above bug that suggests that you add it into your database.yml config.

Mine looks like this:

development:
  primary:
    <<: *default
    <% if RUBY_PLATFORM =~ /darwin/ %>gssencmode: disable<% end %>
    database: app-development
  queue:
    <<: *default
    <% if RUBY_PLATFORM =~ /darwin/ %>gssencmode: disable<% end %>
    database: app-queue-development
    migrations_paths: db/queue_migrate

NOTE! You must have the gssencmode for both the primary and queue configs. I initially only had it under the queue config, and the recurring jobs would fail to run with the following error:

worker | objc[76317]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
worker | objc[76317]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

Question:

I noticed that in the recurring.yml file, if you do not specify the queue to the scheduled item, then the thing-to-be-scheduled goes to the "solid_queue_recurring" queue. If you do not have a worker assigned to the "solid_queue_recurring" queue, then nothing will get scheduled/triggered. Is that the way it's supposed to work?

The documentation didn't make that clear and for a while I was trying to figure out why nothing was being scheduled then triggered - this was also due to the fact that I don't have any workers assigned to '*'.

Question:

I have a use case where I would like to schedule recurring jobs but these jobs need to take in ApplicationRecord model objects that change based on the date/time. I don't think we can do queries in the yml file, so instead, I have configured the recurring.yml to execute a command - and this command will run the query and enqueue the job accordingly. Is this the way to go?

Much thanks!

[rosa]

Hey @wheee, sorry for the delay in replying!

Thanks for including a possible workaround for the PG bug. It's been reported multiple times, and it happens when forking. It's a long-running bug with the pg gem and macOS, unfortunately. Other examples:

• issues when running on macOs #178
• SEGV Fault error when starting worker #145 (comment)
• Terribly long and infinite error logs when crashed #279
• Getting Segmentation fault error during solid queue initialization #458

Question:
I noticed that in the recurring.yml file, if you do not specify the queue to the scheduled item, then the thing-to-be-scheduled goes to the "solid_queue_recurring" queue. If you do not have a worker assigned to the "solid_queue_recurring" queue, then nothing will get scheduled/triggered. Is that the way it's supposed to work?

Yes, this is intended, and it only happens when you use command instead of a class. If you use a job class, the queue would be the one set for your job class. This is documented in the README here:

Question:
I have a use case where I would like to schedule recurring jobs but these jobs need to take in ApplicationRecord model objects that change based on the date/time. I don't think we can do queries in the yml file, so instead, I have configured the recurring.yml to execute a command - and this command will run the query and enqueue the job accordingly. Is this the way to go?

Yes! That'd be the way to go.