Note: Not available for Redis Cluster.
Some processes in Smarty during synchronous execution can cause an avalanche load on the CPU, thereby provoking emergency situations and a long system response.
For example, if you change an active channel through the Smarty or Content API administration panel, all active accounts that access this channel are notified of the need to update the channel list by displaying the appropriate flag, which is automatically requested when the next account status check is made. This action can provoke a large load wave, and as a result, freeze and even complete system failure.
To eliminate this effect, a new RQ module was added to Smarty, which allows you to queue incoming tasks and distribute the load that occurs during their execution by time.
When the RQ mechanism is enabled, flag display from the situation described above will occur on the low priority task queue. This will prevent the panel interface from hanging administration, as well as adjusting the rate of flag display to prevent increased load on the server due to the mass updating of channels by customers (flag speed is regulated by the option) TVMIDDLEWARE_CHANNEL_LIST_MAX_ACCOUNTS_PER_MINUTE(a) the maximum operating time of RQ during this process — RQ_CHANNEL_LIST_CHANGED_JOB_TIMEOUT).
Terms used:
Worker — a process that continually queues for new tasks and, if any, performs them. Usually there are several workmen and they work on the principle: who first got access to the queue, he performs the task.
Planner — a process that sets out tasks with a certain frequency worker in line. They can also be several, collisions between them are solved with the help of locks (the one who first takes possession of the lock flag sets the task).
Inclusion and configuration of RQ
The operation of this mechanism is optional and is included in the parameter SMARTY_USE_RQ in the Smarty configuration file.
Smarty_USE_RQ=TrueIn addition to the enabled option, a running service is required for the mechanism smarty-rq:
systemctl start smarty-rq@where — this is the name of the settings file from /etc/microimpuls/smarty/.
To add the service to the autorun, the command is used:
systemctl enable smarty-rq@The queue uses a separate database in Redis. For the correct operation of the mechanism, it is necessary to provide access to this database from all instances of Smarty. In RQ_QUEUES three lines must be configured: default, high and low.
The settings of the database used are in the parameter RQ_QUEUES. Example:
RQ_QUEUES = {
'default':
'HOST': '10.0.1.2',
'PORT': 6379,
'DB': 10,
'DEFAULT_TIMEOUT': 360,
}
'high':
'HOST': '10.0.1.2',
'PORT': 6379,
'DB': 10,
'DEFAULT_TIMEOUT': 360,
}
'low':
'HOST': '10.0.1.2',
'PORT': 6379,
'DB': 10,
'DEFAULT_TIMEOUT': 3600,
}
}
Viewing the status of the queue
The command is used to check the status of the queue:
smarty_manage rqstats --settings=settings. Example of team withdrawal:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Name | Queued | Active | Deferred | Finished | Workers |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| default 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
| high | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| low | 0 | 0 | 0 | 3 |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Note: the value of 0 in the Workers column indicates that no workman is running at this turn. It is necessary to check the work of the service smarty-rq configuration RQ_QUEUES.
Logistic of queue work
The RQ log file is in the file /var/log/microimpuls/smarty/smarty_rq.logNext to the other log files. For each task logged:
- task name;
- its parameters;
- the result of its implementation (success/failure).
