Troubleshooting
Collecting support bundle
Please see the Help section to find out how to get help with NAbox.
In most case, you will be asked for a support bundle. To collect it, simply open the About box in NAbox web ui and click Download Support Bundle button
NAbox Architecture
Most of NAbox components are delivered as containers inside the virtual appliance.
Containers are managed by Docker Compose, you can find the files for the docker-compose environment in /etc/nabox/.
| File | Description |
|---|---|
compose.yaml |
Main configuration file, it should not be modified because it will be overwritten with each update |
.env |
This file should never be modified, it contains default values for internal NAbox settings |
.env.custom |
Environment in this files overrides default values from .env and is managed by NAbox web ui |
You can interact with the compose stack with the dc command line, which is an alias to docker compose with the right parameters.
admin@localhost ~ $ dc ps
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
alertmanager alertmanager "/bin/alertmanager -…" alertmanager 3 weeks ago Up 57 seconds 9093/tcp
cadvisor cadvisor "/usr/bin/cadvisor -…" cadvisor 3 weeks ago Up 57 seconds (healthy) 8080/tcp
grafana grafana-oss "/run.sh" grafana 2 weeks ago Up 57 seconds 127.0.0.1:3000->3000/tcp
havrest havrest:latest "/havrest" havrest About a minute ago Up 57 seconds 8080/tcp
node-exporter node-exporter "/bin/node_exporter …" node-exporter 3 weeks ago Up 57 seconds 9100/tcp
traefik traefik:v3.0 "/entrypoint.sh trae…" traefik 8 days ago Up 57 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp
victoria-metrics victoria-metrics "/victoria-metrics-p…" victoria-metrics 2 weeks ago Up 57 seconds 127.0.0.1:8428->8428/tcp
vmagent vmagent "/vmagent-prod -remo…" vmagent 4 days ago Up 57 seconds 8429/tcp
vmalert vmalert "/vmalert-prod -rule…" vmalert 3 weeks ago Up 57 seconds 8880/tcp
$ dc logs -f --tail 10 havrest
Attaching to nabox-api
havrest | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=82 bytesRx=3393 calcMs=0 collector=ZapiPerf:HostAdapter exportMs=0 instances=9 instancesExported=9 metrics=18 metricsExported=18 numCalls=1 parseMs=1 pluginMs=0 pollMs=83 skips=0 zBegin=1711821143313
havrest | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=79 bytesRx=19418 calcMs=0 collector=ZapiPerf:Disk exportMs=1 instances=10 instancesExported=50 metrics=180 metricsExported=213 numCalls=1 parseMs=4 pluginMs=1 pollMs=85 skips=0 zBegin=1711821143313
havrest | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=87 bytesRx=693 calcMs=0 collector=ZapiPerf:NFSv4Node exportMs=0 instances=1 instancesExported=1 metrics=4 metricsExported=4 numCalls=1 parseMs=0 pluginMs=0 pollMs=87 skips=0 zBegin=1711821143312
havrest | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=86 bytesRx=1969 calcMs=0 collector=ZapiPerf:SystemNode exportMs=0 instances=1 instancesExported=1 metrics=21 metricsExported=20 numCalls=1 parseMs=0 pluginMs=0 pollMs=87 skips=0 zBegin=1711821143312
havrest | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=87 bytesRx=1097 calcMs=0 collector=ZapiPerf:NFSv3Node exportMs=0 instances=1 instancesExported=1 metrics=9 metricsExported=6 numCalls=1 parseMs=0 pluginMs=0 pollMs=88 skips=0 zBegin=1711821143312
havrest | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=85 bytesRx=992 calcMs=0 collector=ZapiPerf:CIFSNode exportMs=0 instances=1 instancesExported=1 metrics=15 metricsExported=14 numCalls=1 parseMs=0 pluginMs=0 pollMs=86 skips=0 zBegin=1711821143314
havrest | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=86 bytesRx=5801 calcMs=0 collector=ZapiPerf:VolumeNode exportMs=0 instances=2 instancesExported=2 metrics=72 metricsExported=68 numCalls=1 parseMs=1 pluginMs=0 pollMs=88 skips=0 zBegin=1711821143312
$ sudo journalctl -fu naboxd
Mar 30 17:24:53 nabox naboxd[363264]: 2024/03/30 17:24:53 "GET http://10.1.0.133/assets/index-CJn9ZU7D.css HTTP/1.1" from 192.0.0.3:49140 - 200 190602B in 309.852µs
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.444Z level=DEBUG source=auth.go:105 msg=authenticated middleware=FakeAuthMiddleware
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.444Z level=DEBUG source=auth.go:180 msg=authenticated middleware=BasicAuthMiddleware
Mar 30 17:24:58 nabox naboxd[363264]: 2024/03/30 17:24:58 "GET http://10.1.0.133/api/2.0/health HTTP/1.1" from 192.0.0.3:49130 - 200 16B in 158.652µs
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.471Z level=DEBUG source=auth.go:44 msg="not authenticated" message="FakeAuthMiddleware: no credentials"
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.471Z level=DEBUG source=auth.go:44 msg="not authenticated" message="BasicAuthMiddleware: unable to read credentials"
Mar 30 17:24:58 nabox naboxd[363264]: 2024/03/30 17:24:58 "GET http://10.1.0.133/api/2.0/health HTTP/1.1" from 192.0.0.3:49130 - 200 16B in 148.552µs
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.505Z level=DEBUG source=auth.go:44 msg="not authenticated" message="FakeAuthMiddleware: no credentials"
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.505Z level=DEBUG source=auth.go:44 msg="not authenticated" message="BasicAuthMiddleware: unable to read credentials"
Mar 30 17:25:05 nabox naboxd[363264]: 2024/03/30 17:25:05 "GET http://10.1.0.133/api/2.0/system/version HTTP/1.1" from 192.0.0.3:49130 - 200 88B in 7.18837696s
Containers
The following containers are deployed with NAbox :
| Container | Description |
|---|---|
alertmanager |
Alert Manager manages email notifications for various rules declared by NetApp Harvest |
cadvisor |
Exports containers usage statistics for NAbox specific dashboard |
grafana |
Grafana Dashboards |
havrest |
NetApp Harvest 2.x container |
node-exporter |
Exports NAbox host usage statistics for NAbox dashboard |
traefik |
Reverse proxy for all NAbox http services |
victoria-metrics |
Victoria metrics stores metrics coming from NetApp Harvest 2.x |
vmagent |
vmagent is used to transform metrics coming during migration from NAbox 3 |
vmalert |
Interface to the various alerts provided in standard Harvest dashboards deployed in Grafana |
Collecting logs
Most issues can be troubleshoot by looking at logs for naboxd process and havrest container :
$ sudo journalctl -fu naboxd
Mar 30 17:24:53 nabox naboxd[363264]: 2024/03/30 17:24:53 "GET http://10.1.0.133/assets/index-CJn9ZU7D.css HTTP/1.1" from 192.0.0.3:49140 - 200 190602B in 309.852µs
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.444Z level=DEBUG source=auth.go:105 msg=authenticated middleware=FakeAuthMiddleware
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.444Z level=DEBUG source=auth.go:180 msg=authenticated middleware=BasicAuthMiddleware
Mar 30 17:24:58 nabox naboxd[363264]: 2024/03/30 17:24:58 "GET http://10.1.0.133/api/2.0/health HTTP/1.1" from 192.0.0.3:49130 - 200 16B in 158.652µs
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.471Z level=DEBUG source=auth.go:44 msg="not authenticated" message="FakeAuthMiddleware: no credentials"
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.471Z level=DEBUG source=auth.go:44 msg="not authenticated" message="BasicAuthMiddleware: unable to read credentials"
Mar 30 17:24:58 nabox naboxd[363264]: 2024/03/30 17:24:58 "GET http://10.1.0.133/api/2.0/health HTTP/1.1" from 192.0.0.3:49130 - 200 16B in 148.552µs
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.505Z level=DEBUG source=auth.go:44 msg="not authenticated" message="FakeAuthMiddleware: no credentials"
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.505Z level=DEBUG source=auth.go:44 msg="not authenticated" message="BasicAuthMiddleware: unable to read credentials"
Mar 30 17:25:05 nabox naboxd[363264]: 2024/03/30 17:25:05 "GET http://10.1.0.133/api/2.0/system/version HTTP/1.1" from 192.0.0.3:49130 - 200 88B in 7.18837696s
$ dc logs -f --tail 10 havrest
Attaching to nabox-api
havrest | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=82 bytesRx=3393 calcMs=0 collector=ZapiPerf:HostAdapter exportMs=0 instances=9 instancesExported=9 metrics=18 metricsExported=18 numCalls=1 parseMs=1 pluginMs=0 pollMs=83 skips=0 zBegin=1711821143313
havrest | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=79 bytesRx=19418 calcMs=0 collector=ZapiPerf:Disk exportMs=1 instances=10 instancesExported=50 metrics=180 metricsExported=213 numCalls=1 parseMs=4 pluginMs=1 pollMs=85 skips=0 zBegin=1711821143313
havrest | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=87 bytesRx=693 calcMs=0 collector=ZapiPerf:NFSv4Node exportMs=0 instances=1 instancesExported=1 metrics=4 metricsExported=4 numCalls=1 parseMs=0 pluginMs=0 pollMs=87 skips=0 zBegin=1711821143312
havrest | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=86 bytesRx=1969 calcMs=0 collector=ZapiPerf:SystemNode exportMs=0 instances=1 instancesExported=1 metrics=21 metricsExported=20 numCalls=1 parseMs=0 pluginMs=0 pollMs=87 skips=0 zBegin=1711821143312
havrest | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=87 bytesRx=1097 calcMs=0 collector=ZapiPerf:NFSv3Node exportMs=0 instances=1 instancesExported=1 metrics=9 metricsExported=6 numCalls=1 parseMs=0 pluginMs=0 pollMs=88 skips=0 zBegin=1711821143312
havrest | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=85 bytesRx=992 calcMs=0 collector=ZapiPerf:CIFSNode exportMs=0 instances=1 instancesExported=1 metrics=15 metricsExported=14 numCalls=1 parseMs=0 pluginMs=0 pollMs=86 skips=0 zBegin=1711821143314
havrest | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=86 bytesRx=5801 calcMs=0 collector=ZapiPerf:VolumeNode exportMs=0 instances=2 instancesExported=2 metrics=72 metricsExported=68 numCalls=1 parseMs=1 pluginMs=0 pollMs=88 skips=0 zBegin=1711821143312
You can save those logs in a file like so :
{ sudo journalctl -S "yesterday" -u nabox; sudo journalctl -S "yesterday" -u naboxd; } | gzip > nabox-logs-`date +%Y-%m-%d_%H:%M:%S`.gz
Collecting Harvest Configuration
To collect a redacted copy of harvest.yml, run the following command :
dc exec -w /harvest -e HARVEST_CONF=/harvest-conf havrest /harvest/bin/harvest doctor --print
Using Victoria Metrics
Victoria Metrics is exposed on /vm/ URL so you can browse
metrics and test queries.
Just point your browser to https://<nabox ip>/vm/.
Configuration files
NAbox stores most configuration files in /etc/nabox/ directory :
| Directory | Description |
|---|---|
alertmanager |
AlertManager configuration contains email notification settings |
harvest |
Harvest 2.x configuration directory. |
secrets |
This directory contains critical authentication token for communication with Grafana, Havrest container and SSL certificates |
harvest
In this directory you will find the main configuration file for Harvest. Normally, this file is only managed by NAbox Web UI and shouldn't be modified externally.
It is possible though, to copy this file from one NAbox to another when
migrating, or generate this file with external script, as long as you take
extra care putting all the required properties for a poller to function
properly. Specifically, prometheus_port is required and must
be unique across pollers in the file.
When you modify that file, you should see the changes reported in the Web UI.
Note
In the Web UI, only systems having Zapi, ZapiPerf or
StorageGrid collectors will be displayed.
Harvest directory contains a few folders
| File | Description |
|---|---|
active |
This directory contains the Harvest configuration templates compiled from standard NAbox templates and user defined templates |
user |
User template directory where you can add your own customizations (see faq) |
nabox |
This directory contains default NAbox customizations and user-enabled options like workloads collections |
secrets
The secrets directory maintains the tokens used for communication between the
different products in NAbox
| File | Description |
|---|---|
grafana-secret |
Grafana API token used when adding datasources or customizing dashboards properties from Web UI |
havrest-secret |
Harvest REST frontend authentication token (Documentation on https://<nabox ip>/havrest/ui/) |
jwt-secret |
JWT seed, unique for each NAbox deployment, used to generate Web UI session token |
ssl |
Certificate files for NAbox HTTP server |