Troubleshooting

Collecting support bundle

Please see the Help section to find out how to get help with NAbox.

In most case, you will be asked for a support bundle. To collect it, simply open the About box in NAbox web ui and click Download Support Bundle button

About

Collecting logs

Most issues can be troubleshoot by looking at logs for naboxd process and havrest container :

$ sudo journalctl -fu naboxd
Mar 30 17:24:53 nabox naboxd[363264]: 2024/03/30 17:24:53 "GET http://10.1.0.133/assets/index-CJn9ZU7D.css HTTP/1.1" from 192.0.0.3:49140 - 200 190602B in 309.852µs
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.444Z level=DEBUG source=auth.go:105 msg=authenticated middleware=FakeAuthMiddleware
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.444Z level=DEBUG source=auth.go:180 msg=authenticated middleware=BasicAuthMiddleware
Mar 30 17:24:58 nabox naboxd[363264]: 2024/03/30 17:24:58 "GET http://10.1.0.133/api/2.0/health HTTP/1.1" from 192.0.0.3:49130 - 200 16B in 158.652µs
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.471Z level=DEBUG source=auth.go:44 msg="not authenticated" message="FakeAuthMiddleware: no credentials"
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.471Z level=DEBUG source=auth.go:44 msg="not authenticated" message="BasicAuthMiddleware: unable to read credentials"
Mar 30 17:24:58 nabox naboxd[363264]: 2024/03/30 17:24:58 "GET http://10.1.0.133/api/2.0/health HTTP/1.1" from 192.0.0.3:49130 - 200 16B in 148.552µs
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.505Z level=DEBUG source=auth.go:44 msg="not authenticated" message="FakeAuthMiddleware: no credentials"
Mar 30 17:24:58 nabox naboxd[363264]: time=2024-03-30T17:24:58.505Z level=DEBUG source=auth.go:44 msg="not authenticated" message="BasicAuthMiddleware: unable to read credentials"
Mar 30 17:25:05 nabox naboxd[363264]: 2024/03/30 17:25:05 "GET http://10.1.0.133/api/2.0/system/version HTTP/1.1" from 192.0.0.3:49130 - 200 88B in 7.18837696s

$ dc logs -f --tail 10 havrest
Attaching to nabox-api
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=82 bytesRx=3393 calcMs=0 collector=ZapiPerf:HostAdapter exportMs=0 instances=9 instancesExported=9 metrics=18 metricsExported=18 numCalls=1 parseMs=1 pluginMs=0 pollMs=83 skips=0 zBegin=1711821143313
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=79 bytesRx=19418 calcMs=0 collector=ZapiPerf:Disk exportMs=1 instances=10 instancesExported=50 metrics=180 metricsExported=213 numCalls=1 parseMs=4 pluginMs=1 pollMs=85 skips=0 zBegin=1711821143313
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=87 bytesRx=693 calcMs=0 collector=ZapiPerf:NFSv4Node exportMs=0 instances=1 instancesExported=1 metrics=4 metricsExported=4 numCalls=1 parseMs=0 pluginMs=0 pollMs=87 skips=0 zBegin=1711821143312
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=86 bytesRx=1969 calcMs=0 collector=ZapiPerf:SystemNode exportMs=0 instances=1 instancesExported=1 metrics=21 metricsExported=20 numCalls=1 parseMs=0 pluginMs=0 pollMs=87 skips=0 zBegin=1711821143312
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=87 bytesRx=1097 calcMs=0 collector=ZapiPerf:NFSv3Node exportMs=0 instances=1 instancesExported=1 metrics=9 metricsExported=6 numCalls=1 parseMs=0 pluginMs=0 pollMs=88 skips=0 zBegin=1711821143312
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=85 bytesRx=992 calcMs=0 collector=ZapiPerf:CIFSNode exportMs=0 instances=1 instancesExported=1 metrics=15 metricsExported=14 numCalls=1 parseMs=0 pluginMs=0 pollMs=86 skips=0 zBegin=1711821143314
havrest  | 2024-03-30T17:52:23Z INF collector/collector.go:608 > Collected Poller=7-mode apiMs=86 bytesRx=5801 calcMs=0 collector=ZapiPerf:VolumeNode exportMs=0 instances=2 instancesExported=2 metrics=72 metricsExported=68 numCalls=1 parseMs=1 pluginMs=0 pollMs=88 skips=0 zBegin=1711821143312

You can save those logs in a file like so :

{ sudo journalctl -S "yesterday" -u nabox; sudo journalctl -S "yesterday" -u naboxd; } | gzip > nabox-logs-`date +%Y-%m-%d_%H:%M:%S`.gz

Collecting Harvest Configuration

To collect a redacted copy of harvest.yml, run the following command :

dc exec -w /harvest -e HARVEST_CONF=/harvest-conf  havrest /harvest/bin/harvest doctor --print

Using Victoria Metrics

Victoria Metrics is exposed on /vm/ URL so you can browse metrics and test queries.

Just point your browser to https://<nabox ip>/vm/.

Metrics