Installation and configuration

Contentsshow

Primary installation
Update from mistats to mistats-base
Installation of MiStats admin
Logs
- MiStats logs
- ClickHouse Logs

Primary installation

Requirements

Minimum requirements for the MiStats server:

processor: 4 cores with a frequency of more than 3.5 GHz, with support for SSE 4.2;
rAM: 8 GB;
disc: 60GB.

PO versions:

Smarty is not lower than 2.0 for TV statistics and not lower than 2.12 for video statistics;
ClickHouse at least 24.3;
Python at least 3.8;
Debian 11, Debian 12, Ubuntu 24.04, Ubuntu 26.04;
a separate MiStats database on MySQL or PostgreSQL;
Redis.

The time zone of the MiStats server should coincide with the Smarty time zone. Check:

timedatectl | grep 'Time zone'

If the belts are different, you need to ask the same. An example for Moscow:

timedatectl set-timezone Europe/Moscow

Installation of ClickHouse

sudo apt-get install -y apt-transport-https ca-certificates curl gnupg pwgen python3-pip python-is-python3
curl -fsSL 'https://packages.clickhouse.com/rpm/lts/repodata/repomd.xml.key' | sudo gpg --dearmor -o /usr/share/keyrings/clickhouse-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/clickhouse-keyring.gpg] https://packages.clickhouse.com/deb stable main" | sudo tee \
    /etc/apt/sources.list.d/clickhouse.list
sudo apt-get update

sudo apt-get install -y clickhouse-server clickhouse-client

sudo service clickhouse-server start

Next, you need to launch the ClickHouse client. If it is on the same server:

clickhouse-client

If ClickHouse stands alone:

clickhouse-client --host  --secure --port 9940 --user  --password

In the client console, you need to create a database and a user on whose behalf MiStats goes to ClickHouse. The password must be saved: you will need it in the MiStats configuration.

Create Database smarty stats
CREATE USER smarty IDENTIFIED BY '
GRANT ALL ON Smarty stats.* To Smarty;

ClickHouse tables do not need to be created manually at this step. They will be created by migration after setting up MiStats.

MiStats packages

MiStats packages are available for download in the operator’s personal account.

mistat-base - tv viewing and reporting module;
mistats-video video statistics module requires a separate license option.

Versions of both packages in the same installation must match.

sudo dpkg -i mistats-base *.deb
sudo dpkg -i mistats-video *.deb

Together with mistat-base utility mistats manage (/usr/local/bin) for easy control of the instance: migration, scheduler, label compression.

Primary setting

Service configuration file /etc/microimpuls/mistats/configuration.yaml. License - in /etc/microimpuls/mistats/license.yaml.

1. MiStats base

You need to create an empty MySQL or PostgreSQL database and the user. In the MiStats Configuration, section MISTATS DATABASEit is necessary to specify the data for the connection:

HOST
PORT
NAME
USER
PASSWORD
DRIVER MySQL or PostgreSQL driver

Example for MySQL:

MISTATS DATABASE:
HOST: 127.0.0. 1st
PORT: 3306
NAME: mistats
USER: mistats
PASSWORD: .
DRIVER: mysql

2. Connecting to ClickHouse

Same config, section CLICKHOUSEYou need to specify the base created at the step of installing ClickHouse:

HOST, PORT
DATABASE - in the example above this smarty stats
USER, PASSWORD

Example:

CLICKHOUSE:
HOST: 127.0.0. 1st
Port: 9000
DATABASE: smarty stats
USER: smart
PASSWORD:

3. License

The license key is issued in the operator’s personal account and recorded in the /etc/microimpuls/mistats/license.yaml. If there's no key, mistats it doesn't start mistats manage status error from the HW key of the server. This HW key must be specified in the LC when releasing the key. The generated license has the following format:

KEY: "..."
client id:"
client name: "
valid until: 31.12.2026
features
...

4. Redis, API and CORS

Video tags are sent directly from client applications to MiStats. Their pre-processing takes place in Redis, so you need to configure the connection to it configuration.yamlsection MISTATS REDIS: HOST, PORT, DBB.

There you also need to specify the API address for sending tags and requests for reports for adminka, section MISTATS. HOST the address at which the API receives requests, PORT - port. The default port is 8095.

CORS.ENABLED it's on by default. CORS. ALLOWED HOSTS necessary for video statistics tags: it includes the addresses of the client application portal from which requests leave the MiStats API, for example https://tv.example.com.

Example:

MISTATS REDIS:
HOST: 127.0.0. 1st
PORT: 6379
DB: 0
MISTATS:
HOST: 0.0.0. 0
PORT: 8095
CORS:
ENABLED: true
ALLOWED HOSTS:
- https://tv.example.com

5. Environment, migration and launch

Dependencies and migration are created through mistats manage:

sudo systemctl daemon reload
sudo mistats manage venv install
sudo mistats manage requirements
sudo mistats manage migrate clickhouse
sudo mistats manage migrate mistats db
sudo mistats manage start
sudo mistats manage status

Teams mistats managewhich provide the service:

Team	What's he doing
`venv install`	encircles `/usr/share/microimpuls/venvs/mistats`
`requirements`	dependency
`migrate clickhouse`	clickHouse migrations
`migrate mistats db`	the migration of the MiStats base
`start`, `stop`, `restart`, `status`	serviceman `mistats`, `mistats queue` and `mistats scheduler`

Status should be launched mistats, mistats queue and mistats scheduler. If established mistats-videothe scheduler will queue the compression of video viewing labels, a separate cron is not needed for this.

The number of queue processes is configured in the config, section DRAMATIQ. For the first set, the default value does not need to be changed.

Smarty Connection: Television Viewing

IN INSTALLED_APPS smarty configuration should be a module viewstats. The Smarty license must contain this module.

In the Smarty Config, you also need to specify the MiStats API address and the threshold after which Smarty unloads tags from its Redis:

VIEWSTATS MISTATS BASE URL = 'http://127.0.0.1:8095' # host and port listening to the MiStats API
VIEWSTATS BUFFER MAX COUNT = 2500000

Television tags from client devices first enter Smarty, and then unload from there to MiStats through the API /stats/push/which requires authorization with a token. There are 2 Smarty authorization modes in MiStats:

VIEWSTATS MISTATS API TOKEN must be filled in at VIEWSTATS MISTATS USE NEW AUTH = True
VIEWSTATS MISTATS ACCESS TOKEN must be filled in at VIEWSTATS MISTATS USE NEW AUTH = False

Token VIEWSTATS MISTATS API TOKEN must coincide with the record in AUTH.API.TOKENS on the MiStats side. The default values are the same. If the token needs to be changed, it should be done in both configs.

After editing the config, you need to restart uWSGI:

sudo service uwsgi restart

Next, you need to add to cron the unloading of TV viewing tags from Smarty to MiStats. The launch interval should coincide with the value of the portal option push_stat_interval. By default, it is 300,000 milliseconds (5 minutes), below setting the command in cron with the same period:

*/5 * * * * smarty manage push stats --settings=settings;

Smarty's TV viewing label considers complete by option VIEWSTATS MAXIMUM MINUTES BETWEEN INTERVALS. The recommended value is 5 minutes, the same as push_stat_interval by default. If the portal interval has been increased, this value in minutes should also be increased and not left below the new interval:

VIEWSTATS MAXIMUM MINUTES BETWEEN INTERVALS = 5

In order for consoles and applications to start sending TV viewing tags, in the Smarty admin on the “STB and Application Settings” page, you must include the checkbox “Enable the collection of statistical data to the Smarty server”.

Video viewing

Video viewing tags are sent directly to MiStats, bypassing Smarty. Currently, video viewing is sent by the following client interfaces and applications:

Futuristic;
Sibseti;
Titanium;
a mobile app.

Interfaces begin to send labels if:

included checkbox "Enable collection of statistical data on the Smarty server" in the admin for the device;
option added for the device mistats api url with the address of the MiStats API;
portal address added to CORS. ALLOWED HOSTS in the MiStats config.

The first label is sent a minute after the start of the film or series, the next - with an optional interval push_stat_interval.

Checking authkey video viewing tags takes place at the address SMARTY. URL. The default value is calculated for Smarty on the same server and port 8000. In the config you need to specify the actual URL Smarty.

Required options for MiStats config:

VIDEO STATS COMPRESSOR. MAXIMUM MINUTES BETWEEN VIDEO CHECKPOINTS After how many minutes without new labels, the viewing is considered complete. The recommended value is 5. It shouldn't be any less push_stat_intervalif the window is shorter than the pause between the labels, one view is cut by several.
VIDEO STATS COMPRESSOR. CALL INTERVAL SECONS How often the scheduler triggers label compression, in seconds. The recommended value is 600, which is 10 minutes, about twice the closing window.

VIDEO STATS COMPRESSOR:
MAXIMUM MINUTES BETWEEN VIDEO CHECKPOINTS: 5
CALL INTERVAL SECONDS: 600

Based on these statistics, Smarty is able to create a selection of popular films with the help of a team create video election by popularity.

Update from mistats to mistats-base

This instruction is required if the package is already installed mistats (before division into modules) mistat-base and mistats-video):

sudo dpkg -r mistats
sudo dpkg -i mistats-base *.deb

If systemd says that the unit file has changed on the disk, it must be executed sudo systemctl daemon reload.

При переезде у сервиса изменятся пути и формат файлов конфигурации, поэтому нужно будет произвести миграцию текущего конфига на новый формат с помощью специальной команды. При обновлении сборки старая конфигурация не удаляется, а перемещается в /var/lib/mistats/legacy.

После обновления необходимо выполнить:

python3 /usr/share/microimpuls/mistats/backend/commands/migrate mistats configuration.py
  /var/lib/mistats/legacy/configuration.py \
  -d /var/lib/mistats/legacy/mistats_defaults.py \
  -o /etc/microimpuls/mistats/configuration.yaml

The old config is conveyed by the first argument. -d path defaults.py with default values. - o - the way to write configuration.yaml. In the same catalog, the script writes license. No - o YAML is printed in stdout, and license it's not created.

The parameter, which is not in the new scheme, the script does not tolerate and writes a warning. If there is no mandatory parameter in defaults.pyneither configuration.pyThe script ends with an error. The missing value should be added to configuration.yaml initial tuning.

Next mistats manage venv install, mistat manage requirements, mistat manage migrate clickhouse, mistats manage migrate mistats db and mistat manage start. Migration is complemented by existing ClickHouse tables.

Installation of MiStats admin

The section will be expanded.

Logs

MiStats logs

MiStats writes its logs into files. Ways set in /etc/microimpuls/mistats/configuration.yamlsection LOGS.MISTATS.

ERROR FILE PATH - errors;
ACCESS FILE PATH - access to the API;
MISTATS FILE PATH - the main log of the service;
COMMANDS FILE PATH - teams.

Error of the start, including under license, is visible in mistats manage status.

ClickHouse Logs

By default, ClickHouse writes internal logs without storage restrictions. Even without viewing statistics, it quickly takes up the drive.

IN /etc/clickhouse-server/config.xml each system log (blocks with suffix) logneed to be asked ttl:

<clickhouse>
  <query_log>
    <database>system</database>
    <table>query_log</table>
    <partition_by>toYYYMM(event date)</partition_by>
    <ttl>event date + INTERVAL 7 DAY</ttl>
    <flush_interval_milliseconds>7500</flush_interval_milliseconds>
  </query_log>

  <trace_log>
    <database>system</database>
    <table>trace</table>
    <partition_by>toYYYMM(event date)</partition_by>
    <ttl>event date + INTERVAL 1 DAY</ttl>
    <flush_interval_milliseconds>7500</flush_interval_milliseconds>
  </trace_log>

  <metric_log>
    <database>system</database>
    <table>metric_log</table>
    <partition_by>toYYYMM(event date)</partition_by>
    <ttl>event date + INTERVAL 3 DAY</ttl>
    <flush_interval_milliseconds>7500</flush_interval_milliseconds>
  </metric_log>
</clickhouse>

Growing the hardest trace. The storage period can then be extended. Table size:

SELECT table, formatReadableSize(sum(bytes)) AS size
FROM system.parts
WHERE database = 'system'
Group by table
ORDER BY sum(bytes) DESC;

Clean up what has already been accumulated:

TRUNCATE TABLE system.query log
TRUNCATE TABLE system.trace log
TRUNCATE TABLE system.metric log;