Skip to content
1018

Relabel stored VictoriaMetrics series without duplicating data

Summary

Rewrite labels on series that are already stored in VictoriaMetrics so they match label sets that exist for some other reason (a cluster rename, a Harvest identity change, another scraper). Metric names almost never change. Relabel is applied only to the exported historical samples, by sending them through vmagent, then the series that still have the old labels are deleted. Live scrape config is left alone.

Warning

In VictoriaMetrics a series is __name__ plus every label. Changing a label creates a different series. This procedure copies history onto the new label set and then deletes the old one. It does not change what Harvest or other scrapes write next — if they still emit the old labels, those series will come back. /usr/share/nabox is read-only on Flatcar; overlay vmagent's relabel file under /etc/nabox as in KB-1006. That overlay survives reboot and rewrites everything vmagent receives, so step 6 is part of the procedure, not an optional tidy-up. Keep the export on /data and check df -h /data first.

Why vmagent

On NAbox, Harvest is scraped by VictoriaMetrics. vmagent is the remote-write sidecar (-remoteWrite.url=http://victoria-metrics:8428/vm/api/v1/write and -remoteWrite.relabelConfig=/config/relabel.yml). It is the only place a relabel file is already wired, so the rewrite is: export series that still have the old labels, POST them to vmagent (not to VictoriaMetrics import), let vmagent rewrite the labels, then delete the old series.

VictoriaMetrics listens on 127.0.0.1:8428 with path prefix /vm (no guest access needed from SSH). vmagent listens on 8429 with path prefix /vmagent, but is not published on the host, so step 2 publishes it on loopback for the duration of the import.

The examples below rewrite cluster="old-dc" to cluster="new-dc". Use your own label name and values. If you ever need to rename __name__ itself, the same steps apply with target_label: "__name__".

If series with the new labels already have samples in the same time range you export, those timestamps will be written twice. Restrict the export with start / end to the window the new label set is missing.

Procedure

1. Snapshot

admin@localhost $ curl -s -X POST 'http://127.0.0.1:8428/vm/snapshot/create'

Confirm a new directory under /data/victoria-metrics-data/snapshots.

2. Temporary vmagent relabel overlay

Copy the stock file, keep the existing Harvest instance / job rules, and append the label rewrite. This overlay is only for the import in step 4.

admin@localhost $ sudo mkdir -p /etc/nabox/vmagent
admin@localhost $ sudo cp /usr/share/nabox/vmagent/relabel.yml /etc/nabox/vmagent/

Append to /etc/nabox/vmagent/relabel.yml, same indentation as the existing entries:

  - source_labels: [cluster]
    regex: "old-dc"
    replacement: "new-dc"
    target_label: "cluster"

Regex captures work as usual (regex: "prod-(.*)", replacement: "${1}"). Add one rule per label you need to change. Do not drop labels you want to keep.

Create /etc/nabox/compose-custom.yaml if you do not already have one (do not edit compose.yaml). vmagent is not published on the host by default, so the same override also exposes its port on loopback for the import:

services:
  vmagent:
    volumes:
      - /etc/nabox/vmagent/relabel.yml:/config/relabel.yml
    ports:
      - 127.0.0.1:8429:8429

dc already loads every /etc/nabox/compose*.yaml. Both the rule and the port are removed again in step 6.

Keep the 127.0.0.1: prefix. A bare 8429:8429 would publish vmagent's unauthenticated import endpoint on the LAN, next to the authenticated https://<nabox>/vmagent the reverse proxy serves. Do not run this while a NAbox 3 migration, or any other writer to /vmagent, is in flight: the rule rewrites every sample vmagent receives, not only the export.

admin@localhost $ dc up -d vmagent
admin@localhost $ dc logs --tail 20 vmagent
admin@localhost $ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8429/vmagent/-/healthy

vmagent refuses to start on an invalid relabel file, so a clean log plus a 200 means both the rule and the port took effect.

3. Export series that still have the old labels

admin@localhost $ mkdir -p /data/vm-rename
admin@localhost $ curl -s 'http://127.0.0.1:8428/vm/api/v1/export' \
  -d 'match[]={cluster="old-dc"}' \
  -o /data/vm-rename/old.json
admin@localhost $ chmod 600 /data/vm-rename/old.json
admin@localhost $ ls -lh /data/vm-rename/old.json

Narrow match[] as much as you can (add metric name, svm, volume, or other labels). Add start and end (RFC3339 or unix ms) if the new labels already cover recent time. A full database export will fill the disk.

4. Reimport through vmagent

Do not POST this file to VictoriaMetrics /api/v1/import. That path ignores vmagent relabel and would store a second copy with the old labels.

admin@localhost $ curl -s --data-binary @/data/vm-rename/old.json \
  'http://127.0.0.1:8429/vmagent/api/v1/import'

Wait until the remote-write queue is empty (/data/vmagent-remotewrite-data):

admin@localhost $ curl -s 'http://127.0.0.1:8429/vmagent/metrics' \
  | grep '^vmagent_remotewrite_pending_data_bytes'

Wait until that value is 0. Then confirm history landed on the new labels:

admin@localhost $ curl -sG 'http://127.0.0.1:8428/vm/api/v1/query' \
  --data-urlencode 'query=count({cluster="new-dc"})'

5. Delete series that still have the old labels

Only after step 4 looks right. Same delete API as the FAQ; from SSH, loopback skips guest access.

admin@localhost $ curl -s -X POST \
  'http://127.0.0.1:8428/vm/api/v1/admin/tsdb/delete_series?match[]={cluster="old-dc"}'
admin@localhost $ curl -s 'http://127.0.0.1:8428/vm/internal/force_merge'

match[] must select only the old label set. A query for {cluster="old-dc"} should then be empty. Do not delete by metric name alone — that would drop the rewritten series too.

6. Remove the vmagent overlay

Put vmagent back on the stock relabel file, and un-publish its port, so a later remote-write (NAbox 3 migration, or anything else sending to /vmagent) is not rewritten.

admin@localhost $ sudo rm /etc/nabox/compose-custom.yaml /etc/nabox/vmagent/relabel.yml
admin@localhost $ dc up -d vmagent
admin@localhost $ rm -f /data/vm-rename/old.json

Delete the export even if you abandon the procedure halfway: it is a raw metrics dump. Once you no longer need the rollback, drop the snapshot too — it is a second copy of the whole database:

admin@localhost $ curl -s 'http://127.0.0.1:8428/vm/snapshot/list'
admin@localhost $ curl -s -X POST 'http://127.0.0.1:8428/vm/snapshot/delete?snapshot=<name>'

If compose-custom.yaml also holds other overrides, delete only the vmagent volumes and ports stanzas instead of the whole file.

If something goes wrong

  • New labels already had points in the exported window. Restrict step 3 with start / end, or restore the snapshot from step 1 (dc stop victoria-metrics, restore /data/victoria-metrics-data, dc start victoria-metrics) and redo.
  • Import did not change labels. Overlay missing, or the POST went to port 8428 (VictoriaMetrics) instead of 8429 (vmagent). Re-run the step 2 checks.
  • Old labels return after the delete. A scraper is still writing them. Change that producer; this article does not relabel ingest.
  • Root partition filling. The export belongs on /data. See KB-1012 if / is already tight.