Payment processes debugging steps
This documents the various steps that can be taken, to identify the status of a running/stuck process:
RESQUE
Check the Resque queue of the concerned environment, if a process is running there related to payment releases.
Check the worker's process ID (PID) to identify which worker is running the stuck process.
Click on the process to get more info & id.
You can use the
pscommand or theResque.workerobject to get the PID.
Check in the batch processes for the above process id in the SQL server (
batch_processestable)
SQL
Use the SQL Server to run sp.blitzWho & view to identify the details of the actively running queries. You can then use the
sys.dm_exec_requestsview to get more information about the request.Check for blocking session ids if present & if the i/o are increasing or not.
If no blocking ids are present, and I/O read/write operations are increasing, that refers to the process being currently run, and not stuck.
To check further details, get the query & search in the
payment_releaser.rbIf may be a direct query or a rails query.
LOGS
Check for
posting_engine.logto identify where the process is running/stuck currently.Check for performance loggers.
For this to be working,
features.log_performanceshould be enabled inconfig.yml(default value isfalse)
There are few methods in the
payment_releaser.rbprefixed withprofile. These methods make it to theperformance.loglogger, with the entry and exit time, and can help identify where the process might be currently stuck/executing.Figure out the file name from
process_records_into_filesmethod inpayment_releaser.rb. Check if the new file(.dat) has been created with the help of DEVOPS. Note down the size of the file, and check if the size is increasing or not.If it is increasing in size, means the process is not stuck.
If the file size is not increasing even after 3-4 hours, then we can do the following to reset the payment process:
Check with the user, on details of what payments were being released: type, count, accounting date etc.
Identify all the payments that are to be released.
Check if all the payments have been marked released.
If all the payments are marked released, we just need to manually finish the process.
Set the
finished_atinbatch_processesfor the respective processRun
flush_jobs
If (c) is not the case, and we are 100% confirm that the process is stuck, proceed to reset the process
RESET PROCESS
NOTE: This should be treated as extreme measures. Process is slow at times, and RESET should not be considered unless everything above has been verified thoroughly
Check and confirm that no other process than the payment releaser is running.
Kill the resque job via
flush_jobsIf the payment type is
check, we need to reset the check number as well firstNext number of the bank account(where payment type=”Check”) needs to be reset
Identify the minimum payment_number(s) from the affected payments w.r.t the bank accounts
For each bank account, update the next number as the minimum number received from (3.b)
Reset the payments, replacing xxx with the affected payment ids:
UPDATE ap_payments
SET payment_number = NULL,
released = NULL,
status = 'Pending'
WHERE id in (xxx)