Project

General

Profile

Actions

Task #6118

closed

Task #6116: EPIC: Speed up the stats updater daemon (scripts/workflow/update_stats.sh)

Fetch only the missing date tail instead of re-downloading 365 days every round

Added by Redmine Admin about 10 hours ago. Updated about 9 hours ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Start date:
08/26/2026
Due date:
% Done:

100%

Estimated time:

Description

Problem

All three metric commands hardcode a full-year window and re-fetch it from scratch on every iteration of the daemon loop:

  • InitialStatsGatheringCommand.downloadStatsDatadateStart = now - 365d, dateEnd = today
  • PartnersMetricsGatheringCommand.downloadStatsData — same
  • UsersMetricsGatheringCommand.downloadStatsData — same

IikoStatsGatheringService.getMetricValuesFromIikoDateDesc then walks that window day by day (line 117) and issues one OLAP request per day. Nothing consults what is already in stat_slices — the same 364 days are downloaded again on every round, forever, and re-inserted.

Proposed change

Before walking the window, look up the newest slice already stored for the metric name being gathered (the name is composed in IikoOlapV2MetricFetchingService.generateMetricName from prefix + company ids + field, so it is queryable) and start from there instead of from now - 365d.

  • default window = max(storedDateEnd, now - 365d)today
  • always re-fetch a small trailing overlap (the current day, and ideally the previous one) since today's figures are still moving
  • keep the full 365-day backfill available behind an explicit --full / --from option on all three commands, for the initial load and for repairs

Steady state should be ~1–2 days of data per metric per round instead of 365.

Acceptance

  • Second consecutive run of iiko-stats-data-sync over unchanged data issues only the overlap-window requests.
  • --full still performs the complete 365-day backfill.
  • No duplicate slices accumulate for days already stored.

Related issues 1 (0 open1 closed)

Related to Task #6129: Индексы под горячий путь демона: stat_slices вообще без индексовResolved08/26/2026

Actions
Actions

Also available in: Atom PDF