How to clear stuck jobs
🤔 Problem
Often times we’ll receive a ticket about a job/process that either got stuck in the middle of the run or it never started.
In the screenshot we can see that this job never started, because the started column was never populated with the timestamp.
🌱 Solution
To solve this problem we have scripts that can fix this issue:
One option, that requires no downtime, is the
batchProcess.reset!rails command.
This command resets the timestamps on the batch process records in the database.
We keep track of status by a handful of different timestamps on the process record to let us know if it has been enqueued, started, finished, etc. When a process starts, it sets thestarted_attimestamp, when a process finishes, it setsfinished_at. But if something breaks, or there is a server issue or the worker dies improperly, sometimes the timestamps will be incorrect.
So if the database tells us the record was started but not finished, the system will stop you from running processes. If we know its really not running anymore,BatchProcess.reset!is a command which resets the timestamps for anything that is stuck, so it can be run againThe other option—and the most common one—is to use the
flush_jobsscript. This is typically run together with therestart_services --rebuildscript, although doing so requires some downtime. Theflush_jobsscript also executes theBatchProcess.reset!Rails command, but it includes additional steps, which is why it's the recommended approach when a job needs to be forcefully unstuck.
Highlight important information in a panel like this one. To edit this panel's color or style, select one of the options in the menu.