Troubleshooting

Collecting support bundle

Please see the Help section to find out how to get help with NAbox.

In most case, you will be asked for a support bundle. To collect it, simply open the About box in NAbox web ui and click Download Support Bundle button

About

NAbox Architecture

Most of NAbox components are delivered as containers inside the virtual appliance.

Containers are managed by Docker Compose, you can find the files for the docker-compose environment in /etc/nabox/.

File	Description
`compose.yaml`	Main configuration file, it should not be modified because it will be overwritten with each update
`.env`	This file should never be modified, it contains default values for internal NAbox settings
`.env.custom`	Environment in this files overrides default values from `.env` and is managed by NAbox web ui

You can interact with the compose stack with the dc command line, which is an alias to docker compose with the right parameters.

admin@localhost ~ $ dc ps
NAME               IMAGE              COMMAND                  SERVICE            CREATED              STATUS                    PORTS
alertmanager       alertmanager       "/bin/alertmanager -…"   alertmanager       3 weeks ago          Up 57 seconds             9093/tcp
cadvisor           cadvisor           "/usr/bin/cadvisor -…"   cadvisor           3 weeks ago          Up 57 seconds (healthy)   8080/tcp
grafana            grafana-oss        "/run.sh"                grafana            2 weeks ago          Up 57 seconds             127.0.0.1:3000->3000/tcp
havrest            havrest:latest     "/havrest"               havrest            About a minute ago   Up 57 seconds             8080/tcp
node-exporter      node-exporter      "/bin/node_exporter …"   node-exporter      3 weeks ago          Up 57 seconds             9100/tcp
traefik            traefik:v3.0       "/entrypoint.sh trae…"   traefik            8 days ago           Up 57 seconds             0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp
victoria-metrics   victoria-metrics   "/victoria-metrics-p…"   victoria-metrics   2 weeks ago          Up 57 seconds             127.0.0.1:8428->8428/tcp
vmagent            vmagent            "/vmagent-prod -remo…"   vmagent            4 days ago           Up 57 seconds             8429/tcp
vmalert            vmalert            "/vmalert-prod -rule…"   vmalert            3 weeks ago          Up 57 seconds             8880/tcp

$ dc logs -f --tail 10 havrest
Attaching to nabox-api
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=82 bytesRx=3393 calcMs=0 collector=ZapiPerf:HostAdapter exportMs=0 instances=9 instancesExported=9 metrics=18 metricsExported=18 numCalls=1 parseMs=1 pluginMs=0 pollMs=83 skips=0 zBegin=1711821143313
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=79 bytesRx=19418 calcMs=0 collector=ZapiPerf:Disk exportMs=1 instances=10 instancesExported=50 metrics=180 metricsExported=213 numCalls=1 parseMs=4 pluginMs=1 pollMs=85 skips=0 zBegin=1711821143313
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=87 bytesRx=693 calcMs=0 collector=ZapiPerf:NFSv4Node exportMs=0 instances=1 instancesExported=1 metrics=4 metricsExported=4 numCalls=1 parseMs=0 pluginMs=0 pollMs=87 skips=0 zBegin=1711821143312
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=86 bytesRx=1969 calcMs=0 collector=ZapiPerf:SystemNode exportMs=0 instances=1 instancesExported=1 metrics=21 metricsExported=20 numCalls=1 parseMs=0 pluginMs=0 pollMs=87 skips=0 zBegin=1711821143312
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=87 bytesRx=1097 calcMs=0 collector=ZapiPerf:NFSv3Node exportMs=0 instances=1 instancesExported=1 metrics=9 metricsExported=6 numCalls=1 parseMs=0 pluginMs=0 pollMs=88 skips=0 zBegin=1711821143312
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=85 bytesRx=992 calcMs=0 collector=ZapiPerf:CIFSNode exportMs=0 instances=1 instancesExported=1 metrics=15 metricsExported=14 numCalls=1 parseMs=0 pluginMs=0 pollMs=86 skips=0 zBegin=1711821143314
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=86 bytesRx=5801 calcMs=0 collector=ZapiPerf:VolumeNode exportMs=0 instances=2 instancesExported=2 metrics=72 metricsExported=68 numCalls=1 parseMs=1 pluginMs=0 pollMs=88 skips=0 zBegin=1711821143312

$ sudo journalctl -fu naboxd
Mar 30 17:24:53 nabox naboxd[363264]: 2024/03/30 17:24:53 "GET http://10.1.0.133/assets/index-CJn9ZU7D.css HTTP/1.1" from 192.0.0.3:49140 - 200 190602B in 309.852µs
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.444Z level=DEBUG source=auth.go:105 msg=authenticated middleware=FakeAuthMiddleware
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.444Z level=DEBUG source=auth.go:180 msg=authenticated middleware=BasicAuthMiddleware
Mar 30 17:24:58 nabox naboxd[363264]: 2024/03/30 17:24:58 "GET http://10.1.0.133/api/2.0/health HTTP/1.1" from 192.0.0.3:49130 - 200 16B in 158.652µs
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.471Z level=DEBUG source=auth.go:44 msg="not authenticated" message="FakeAuthMiddleware: no credentials"
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.471Z level=DEBUG source=auth.go:44 msg="not authenticated" message="BasicAuthMiddleware: unable to read credentials"
Mar 30 17:24:58 nabox naboxd[363264]: 2024/03/30 17:24:58 "GET http://10.1.0.133/api/2.0/health HTTP/1.1" from 192.0.0.3:49130 - 200 16B in 148.552µs
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.505Z level=DEBUG source=auth.go:44 msg="not authenticated" message="FakeAuthMiddleware: no credentials"
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.505Z level=DEBUG source=auth.go:44 msg="not authenticated" message="BasicAuthMiddleware: unable to read credentials"
Mar 30 17:25:05 nabox naboxd[363264]: 2024/03/30 17:25:05 "GET http://10.1.0.133/api/2.0/system/version HTTP/1.1" from 192.0.0.3:49130 - 200 88B in 7.18837696s

Containers

The following containers are deployed with NAbox :

Container	Description
`alertmanager`	Alert Manager manages email notifications for various rules declared by NetApp Harvest
`cadvisor`	Exports containers usage statistics for NAbox specific dashboard
`grafana`	Grafana Dashboards
`havrest`	NetApp Harvest 2.x container
`node-exporter`	Exports NAbox host usage statistics for NAbox dashboard
`traefik`	Reverse proxy for all NAbox http services
`victoria-metrics`	Victoria metrics stores metrics coming from NetApp Harvest 2.x
`vmagent`	vmagent is used to transform metrics coming during migration from NAbox 3
`vmalert`	Interface to the various alerts provided in standard Harvest dashboards deployed in Grafana

Collecting logs

Most issues can be troubleshoot by looking at logs for naboxd process and havrest container :

$ sudo journalctl -fu naboxd
Mar 30 17:24:53 nabox naboxd[363264]: 2024/03/30 17:24:53 "GET http://10.1.0.133/assets/index-CJn9ZU7D.css HTTP/1.1" from 192.0.0.3:49140 - 200 190602B in 309.852µs
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.444Z level=DEBUG source=auth.go:105 msg=authenticated middleware=FakeAuthMiddleware
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.444Z level=DEBUG source=auth.go:180 msg=authenticated middleware=BasicAuthMiddleware
Mar 30 17:24:58 nabox naboxd[363264]: 2024/03/30 17:24:58 "GET http://10.1.0.133/api/2.0/health HTTP/1.1" from 192.0.0.3:49130 - 200 16B in 158.652µs
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.471Z level=DEBUG source=auth.go:44 msg="not authenticated" message="FakeAuthMiddleware: no credentials"
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.471Z level=DEBUG source=auth.go:44 msg="not authenticated" message="BasicAuthMiddleware: unable to read credentials"
Mar 30 17:24:58 nabox naboxd[363264]: 2024/03/30 17:24:58 "GET http://10.1.0.133/api/2.0/health HTTP/1.1" from 192.0.0.3:49130 - 200 16B in 148.552µs
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.505Z level=DEBUG source=auth.go:44 msg="not authenticated" message="FakeAuthMiddleware: no credentials"
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.505Z level=DEBUG source=auth.go:44 msg="not authenticated" message="BasicAuthMiddleware: unable to read credentials"
Mar 30 17:25:05 nabox naboxd[363264]: 2024/03/30 17:25:05 "GET http://10.1.0.133/api/2.0/system/version HTTP/1.1" from 192.0.0.3:49130 - 200 88B in 7.18837696s

$ dc logs -f --tail 10 havrest
Attaching to nabox-api
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=82 bytesRx=3393 calcMs=0 collector=ZapiPerf:HostAdapter exportMs=0 instances=9 instancesExported=9 metrics=18 metricsExported=18 numCalls=1 parseMs=1 pluginMs=0 pollMs=83 skips=0 zBegin=1711821143313
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=79 bytesRx=19418 calcMs=0 collector=ZapiPerf:Disk exportMs=1 instances=10 instancesExported=50 metrics=180 metricsExported=213 numCalls=1 parseMs=4 pluginMs=1 pollMs=85 skips=0 zBegin=1711821143313
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=87 bytesRx=693 calcMs=0 collector=ZapiPerf:NFSv4Node exportMs=0 instances=1 instancesExported=1 metrics=4 metricsExported=4 numCalls=1 parseMs=0 pluginMs=0 pollMs=87 skips=0 zBegin=1711821143312
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=86 bytesRx=1969 calcMs=0 collector=ZapiPerf:SystemNode exportMs=0 instances=1 instancesExported=1 metrics=21 metricsExported=20 numCalls=1 parseMs=0 pluginMs=0 pollMs=87 skips=0 zBegin=1711821143312
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=87 bytesRx=1097 calcMs=0 collector=ZapiPerf:NFSv3Node exportMs=0 instances=1 instancesExported=1 metrics=9 metricsExported=6 numCalls=1 parseMs=0 pluginMs=0 pollMs=88 skips=0 zBegin=1711821143312
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=85 bytesRx=992 calcMs=0 collector=ZapiPerf:CIFSNode exportMs=0 instances=1 instancesExported=1 metrics=15 metricsExported=14 numCalls=1 parseMs=0 pluginMs=0 pollMs=86 skips=0 zBegin=1711821143314
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=86 bytesRx=5801 calcMs=0 collector=ZapiPerf:VolumeNode exportMs=0 instances=2 instancesExported=2 metrics=72 metricsExported=68 numCalls=1 parseMs=1 pluginMs=0 pollMs=88 skips=0 zBegin=1711821143312

You can save those logs in a file like so :

{ sudo journalctl -S "yesterday" -u nabox; sudo journalctl -S "yesterday" -u naboxd; } | gzip > nabox-logs-`date +%Y-%m-%d_%H:%M:%S`.gz

Collecting Harvest Configuration

To collect a redacted copy of harvest.yml, run the following command :

dc exec -w /harvest -e HARVEST_CONF=/harvest-conf  havrest /harvest/bin/harvest doctor --print

Using Victoria Metrics

Victoria Metrics is exposed on /vm/ URL so you can browse metrics and test queries.

Just point your browser to https://<nabox ip>/vm/.

Metrics

Configuration files

NAbox stores most configuration files in /etc/nabox/ directory :

Directory	Description
`alertmanager`	AlertManager configuration contains email notification settings
`harvest`	Harvest 2.x configuration directory.
`secrets`	This directory contains critical authentication token for communication with Grafana, Havrest container and SSL certificates

harvest

In this directory you will find the main configuration file for Harvest. Normally, this file is only managed by NAbox Web UI and shouldn't be modified externally.

It is possible though, to copy this file from one NAbox to another when migrating, or generate this file with external script, as long as you take extra care putting all the required properties for a poller to function properly. Specifically, prometheus_port is required and must be unique across pollers in the file.

When you modify that file, you should see the changes reported in the Web UI.

Note

In the Web UI, only systems having Zapi, ZapiPerf or StorageGrid collectors will be displayed.

Harvest directory contains a few folders

File	Description
`active`	This directory contains the Harvest configuration templates compiled from standard NAbox templates and user defined templates
`user`	User template directory where you can add your own customizations (see faq)
`nabox`	This directory contains default NAbox customizations and user-enabled options like workloads collections

secrets

The secrets directory maintains the tokens used for communication between the different products in NAbox

File	Description
`api-tokens`	Contains internal and user-generated tokens
`grafana-secret`	Grafana API token used when adding datasources or customizing dashboards properties from Web UI
`havrest-secret`	Harvest REST frontend authentication token (Documentation on https://<nabox ip>/havrest/ui/)
`jwt-secret`	JWT seed, unique for each NAbox deployment, used to generate Web UI session token
`ssl`	Certificate files for NAbox HTTP server