Skip to content

FAQ

Problems and resolutions

Adding 7-mode systems

Check that you enabled TLS with options tls.enable on. Also check the following options : httpd.admin.access, httpd.admin.hostsequiv.enable

Reset Network configuration

If you find yourself in a situation where NAbox has an incorrect network configuration and you cannot connect to it, you wille have to log into the VMware console and manually configure the network :

  1. Login to the console using the adminuser
  2. cd into /etc/systemd/network
  3. Edit 10-nabox.network with vim

Here is an example for a DHCP configuration :

[Match]
Name=ens192

[Network]
LinkLocalAddressing=no
IPv6AcceptRA=no
DHCP=ipv4
Domains=home.lab

Here is an example for a static IP :

[Match]
Name=ens192

[Network]
LinkLocalAddressing=no
IPv6AcceptRA=no
Address=10.1.0.133/24
Gateway=10.1.0.1
Domains=home.lab
DNS=10.0.1.2
  1. After saving the file, run systemctl restart systemd-networkd

General questions

Migrating from older versions

NAbox 3 cannot be upgraded in place to NAbox 4, this is due to a revamped architecture and a change of the base OS.

NAbox 3 was using Alpine linux as a foundation, while NAbox 4 is using Flatcar Container Linux.

  1. Connect to NAbox 3 instance with SSH, and install migrate CLI utility

You will be reminded that the process can take a few hours and it is recommended to run it inside of a screen session.

# curl -o /usr/bin/migrate -k https://<NAbox 4 ip>/migrate && chmod 755 /usr/bin/migrate
# screen -R
# migrate
  1. You will be prompted for NAbox 4 IP address and admin password
  2. The tool will go through the following steps :
    1. Verify and exclude already configured systems
    2. Confirm which systems should be configured (all by defaut)
    3. Add your systems to NAbox 4
    4. Enable Prometheus admin API
    5. Take a snapshot of Prometheus data
    6. Disable Prometheus admin API
    7. Copy Prometheus data to NAbox 4

Example session :

~ # migrate
┃ NAbox 4 IP Address
┃ > 10.3.0.157

  NAbox 4 Password 
  > ******

┃ The folowing systems are already at the destination and have been ignored :
┃   - cluster2   
┃         
┃ Select systems to configure in NAbox 4 :
┃ > ✓ Webscale Demo

Restarting Prometheus with admin API enabled... OK
Waiting for Prometheus.. OK
Taking snapshot... OK
Restarting Prometheus with admin API disabled... OK
Waiting for Prometheus.. OK
Prometheus import mode
Prometheus snapshot stats:
  blocks found: 14;
  blocks skipped by time filter: 0;
  min time: 1700636426825 (2023-11-22T07:00:26Z);
  max time: 1714855433776 (2024-05-04T20:43:53Z);
  samples: 3628977545;
  series: 310991.
VM worker 0:↓ 375376 samples/s                                                                                                     
VM worker 1:↓ 350265 samples/s                                                                                                     
Processing blocks: 0 / 14 [▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒] 0.00%

The data migration can take several hours, this is why it is strongly recommended to run it in a screen session, so if the terminal session is interrupted, the migration will continue.

If your session is interrupted, you can restore it by running screen -R

Manual Migration

If the migration tool doesn't work for some reason, the manual steps are described below. Note that it will only migrate Prometheus data and not import the systems configuration to NAbox 4.

  1. Restart Prometheus with admin API
$ cd /usr/local/nabox/docker-compose
$ WEB_ENABLE_ADMIN_API=--web.enable-admin-api docker compose --env-file .env --env-file .env.custom up -d prometheus
  1. Make a Prometheus data snapshot using API
$ curl -k -X POST https://localhost/prometheus/api/v1/admin/tsdb/snapshot
{"status":"success","data":{"name":"20240404T061544Z-00b8acf6e99be9be"}}
$ export SNAPSHOT_PATH=/prometheus/data/snapshots/20240404T061544Z-00b8acf6e99be9be

Set the SNAPSHOT_PATH environment variable with the path to the created snapshot

  1. Remove admin API from Prometheus
$ docker compose --env-file .env --env-file .env.custom up -d prometheus
  1. Migrate prometheus data to Victoria Metrics
$ NABOX_IP=<nabox_ip>
$ docker run --rm -v /data/prometheus:/prometheus -it --network docker-compose_default victoriametrics/vmctl:heads-master-0-g93a29fce4e prometheus --prom-snapshot=$SNAPSHOT_PATH --vm-addr=https://$NABOX_IP/vm/ --vm-insecure-skip-verify -s

Root access

At your own risk, for troubleshooting or customizing purpose, you can access the virtual appliance root shell using sudo after logging in as admin through ssh

Reset password

TODO

Managing metrics

Delete data

Metrics can be deleted using API calls to Victoria Metrics. Note that un-authenticated acces to Victoria Metrics is disabled by default and can be enabled in NAbox preferences.

  1. Delete metrics for a given cluster

    # Delete metrics for cluster2
    curl -k -s \
      'https://nabox_ip/vm/api/v1/admin/tsdb/delete_series?match[]=\{cluster="cluster2"\}'
    
    # Purge data from disk
    curl -s 'https://nabox_ip/vm/internal/force_merge'
    
  2. Delete metrics for a given volume

    # Delete metrics for volume `myvolume`
    curl -k -s \
      'https://nabox_ip/vm/api/v1/admin/tsdb/delete_series?match[]=\{volume="myvolume"\}'
    # Purge data from disk
    curl -s 'https://nabox_ip/vm/internal/force_merge'
    

Don't forget to turn back off guest access to metrics if you had to turn it on.

Delete old data

TODO

Change default retention

TODO

If you find that 2 years of data retention is not appropriate in your environment, you can override the default by editing /etc/nabox/.env.custom and add/change the following line

DOCKER_SUBNET=192.0.0.1/24
GF_AUTH_LDAP_ENABLED=true
GF_AUTH_LDAP_CONFIG_FILE=
GF_AUTH_ANONYMOUS_ENABLED=true
GF_SERVER_DOMAIN=10.1.0.133
GF_SMTP_ENABLED=true
GF_SMTP_HOST=smtp.mail.me.com:587
GF_SMTP_USER=ybizeul@me.com
GF_SMTP_PASSWORD=cxiu-oefj-aapz-ssjd
GF_SMTP_FROM_ADDRESS=yann@tynsoe.org
PROMETHEUS_NABOX_AUTH=
retentionPeriod=2y

After the change, you have to restart services :

# dc up -d

Customize Harvest

NAbox lets you customize harvest templates as described in Harvest docs.

Customization must be done in CLI in the /etc/nabox/harvest/user folder.

/etc/nabox/harvest/user                       # User customization directory
├─ /zapi                                      # Collector specific customization directory
│  ├─ /cdot/9.8/                              # Version aware template directory
│  │  └─ custom_my_ignore_list_template.yaml  # Template file
│  └─ custom.yaml                             # Customization file
└─ /zapiperf
   ├─ /cdot/9.8/
   │  └─custom_my_ignore_list_template.yaml
   └─ custom.yaml

Extending templates is only supported by the Zapi / ZapiPerf collectors

The Harvest REST collector only supports replacing one of the builtin templates with one of your own.

The REST collector does not support template extending, so if you choose to prefer Rest over Zapi in Preferences make sure that you do your customization based on the templates in /data/packages/harvest/conf/rest/9.x.0/ (most template files reside in the 9.12.0 directory).

Common practice is to copy the needed file(s) and do the necessary modifications, it will override the default Harvest file for the object definition.

Note about built-in customizations

NAbox defines its own customizations in /etc/nabox/harvest/nabox.

Symbolic links are created pointing to /etc/nabox/harvest/nabox.available/ directories.

/etc/nabox/harvest
├─ /nabox
│  ├─ /collect-workloads -> ../nabox.available/collect-workloads
│  └─ /exclude-transient-volumes -> ../nabox.available/exclude-transient-volumes
└─ /nabox.available/
   ├─ /collect-workloads
   └─ /exclude-transient-volumes

exclude-transient-volumes is always enabled, while collect-workloads is only enabled according to user preferences in NAbox web ui.

During start up, NAbox merges all customizations into /etc/nabox/harvest/active

/etc/nabox/harvest/active
├─ /zapiperf
│  ├─ /custom.yaml
│  └─ /cdot/9.8.0
│     ├─/custom_my_ignore_list_template.yaml
│     ├─/exclude_transient_volumes.yaml
└─ /zapi
   ├─ /custom.yaml
   └─ /cdot/9.8.0
      ├─/custom_my_ignore_list_template.yaml
      ├─/exclude_transient_volumes.yaml

zapiperf/custom.yaml content :

objects:
    Volume: exclude_transient_volumes.yaml,custom_my_ignore_list_template.yaml
    Workload: workload.yaml,exclude_transient_volumes.yaml,custom_my_ignore_list_template.yaml
    WorkloadDetail: workload_detail.yaml,exclude_transient_volumes.yaml,custom_my_ignore_list_template.yaml
    WorkloadDetailVolume: workload_detail_volume.yaml,exclude_transient_volumes.yaml,custom_my_ignore_list_template.yaml
    WorkloadVolume: workload_volume.yaml,exclude_transient_volumes.yaml,custom_my_ignore_list_template.yaml

zapi/custom.yaml content :

objects:
    Volume: exclude_transient_volumes.yaml,custom_my_ignore_list_template.yaml

Example : exclude collection based on volume name

To ignore volumes that you don't want to be collected by Harvest performance and/or capacity pollers, you can follow these steps.

  1. Create /etc/nabox/harvest/user/zapiperf/custom.yaml

    objects:
      Volume: custom_my_ignore_list_template.yaml
      Workload: custom_my_ignore_list_template.yaml
      WorkloadDetail: custom_my_ignore_list_template.yaml
      WorkloadDetailVolume: custom_my_ignore_list_template.yaml
      WorkloadVolume: custom_my_ignore_list_template.yaml
    
  2. Create /etc/nabox/harvest/user/zapiperf/cdot/9.8.0/custom_my_ignore_list_template.yaml

    plugins:
      LabelAgent:
        exclude_regex:
          - volume `Test_volume.*`
          - volume `Temp_volume.*`
    
  3. We need to do the same for Zapi poller. Create /etc/nabox/harvest/user/zapi/custom.yaml

    objects:
      Volume: custom_my_ignore_list_template.yaml
    
  4. Create /etc/nabox/harvest/user/zapi/cdot/9.8.0/custom_my_ignore_list_template.yaml

    plugins:
      LabelAgent:
        exclude_regex:
          - volume `Test_volume.*`
          - volume `Temp_volume.*`
    
  5. Restart Harvest

    dc restart havrest