Troubleshooting
NAbox Architecture
Most of NAbox components are delivered as containers inside the virtual appliance.
Containers are managed by docker-compose, you can find the files for the docker-compose environment in /usr/local/nabox/docker-compose/.
| File | Description |
|---|---|
docker-compose.yaml |
Main configuration file, it should not be modified because it will be overwritten with each update |
docker-compose.override.yaml |
Override file, you can customize docker-compose environement by making changes here |
.env |
Environment file, some critical environment variables are defined here |
You can interact with the compose stack with the dc command line, which is an alias to docker-compose with the right arguments.
nabox:~# dc ps
Name Command State Ports
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
container-exporter /bin/container-exporter -- ... Up 9104/tcp
go-carbon /init/run.sh Up (healthy) 2003/tcp, 2003/udp, 2004/tcp, 7002/tcp, 7003/tcp, 7007/tcp, 8080/tcp
grafana /run.sh Up 3000/tcp
graphite /bin/sh -c /run.sh Up 80/tcp
nabox-admin /docker-entrypoint.sh ngin ... Up 80/tcp
nabox-api python api.py Up 5000/tcp
nabox-harvest python3 harvest.py Up 5000/tcp
nabox-harvest2 python3 harvest.py Up 5000/tcp
prometheus /bin/prometheus --config.f ... Up 9090/tcp
traefik /entrypoint.sh --api.insec ... Up 0.0.0.0:2003->2003/tcp,:::2003->2003/tcp, 0.0.0.0:443->443/tcp,:::443->443/tcp,
nabox-api:~# dc logs -f --tail 10 nabox-api
Attaching to nabox-api
nabox-api | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:14] "GET /api/1.0/system/disks HTTP/1.1" 200 -
nabox-api | 172.99.0.9 - - [01/Apr/2022 10:01:19] "GET /api/1.0/dashboards/nullpointmode HTTP/1.1" 200 -
nabox-api | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:19] "GET /api/1.0/dashboards/nullpointmode HTTP/1.1" 200 -
nabox-api | 172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/upgrade HTTP/1.1" 200 -
nabox-api | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/upgrade HTTP/1.1" 200 -
nabox-api | 172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/version HTTP/1.1" 200 -
nabox-api | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/version HTTP/1.1" 200 -
nabox-api | WARNING:root:{'text': 'harvest version 22.02.0-4 (commit 9484df6) (build date 2022-02-15T09:11:03-0500) linux/amd64', 'version': '22.02.0-4'}
nabox-api | 172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/packages HTTP/1.1" 200 -
nabox-api | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/packages HTTP/1.1" 200 -
Containers
The following containers are deployed with NAbox :
| Container | Description |
|---|---|
container-exporter |
Exports usage statistics for NAbox dashboard |
go-carbon |
Ingests metrics coming from NetApp Harvest 1.x |
grafana |
Grafana Dashboards |
graphite |
Presents metrics coming from NetApp Harvest 1.x to Grafana |
nabox-admin |
Web interface |
nabox-api |
NAbox API server |
nabox-harvest |
NetApp Harvest 1.x |
nabox-harvest2 |
NetApp Harvest 2.x |
prometheus |
Prometheus stores metrics coming from NetApp Harvest 2.x |
traefik |
Routes HTTPS trafic to different containers |
Collecting logs
Most issues can be troubleshoot by looking at logs for nabox-api and nabox-harvest2 containers :
nabox:~# dc logs -f --tail 10 nabox-api
Attaching to nabox-api
nabox-api | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:14] "GET /api/1.0/system/disks HTTP/1.1" 200 -
nabox-api | 172.99.0.9 - - [01/Apr/2022 10:01:19] "GET /api/1.0/dashboards/nullpointmode HTTP/1.1" 200 -
nabox-api | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:19] "GET /api/1.0/dashboards/nullpointmode HTTP/1.1" 200 -
nabox-api | 172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/upgrade HTTP/1.1" 200 -
nabox-api | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/upgrade HTTP/1.1" 200 -
nabox-api | 172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/version HTTP/1.1" 200 -
nabox-api | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/version HTTP/1.1" 200 -
nabox-api | WARNING:root:{'text': 'harvest version 22.02.0-4 (commit 9484df6) (build date 2022-02-15T09:11:03-0500) linux/amd64', 'version': '22.02.0-4'}
nabox-api | 172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/packages HTTP/1.1" 200 -
nabox-api | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/packages HTTP/1.1" 200 -
nabox:~# dc logs -f --tail 10 nabox-harvest2
Attaching to nabox-harvest2
nabox-harvest2 | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [files-used-pct-file-limit] instances on this SVM_NAS.fg_oss_1605688496.bucket03 Poller=AFF plugin=Zapi:Qtree
nabox-harvest2 | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [files-used-pct-soft-file-limit] instances on this SVM_NAS.fg_oss_1605688496. Poller=AFF plugin=Zapi:Qtree
nabox-harvest2 | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [disk-used-pct-disk-limit] instances on this SVM_NAS.fg_oss_1605688496. Poller=AFF plugin=Zapi:Qtree
nabox-harvest2 | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [disk-used-pct-soft-disk-limit] instances on this SVM_NAS.fg_oss_1605688496. Poller=AFF plugin=Zapi:Qtree
nabox-harvest2 | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [disk-used-pct-threshold] instances on this SVM_NAS.fg_oss_1605688496. Poller=AFF plugin=Zapi:Qtree
nabox-harvest2 | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [files-used-pct-file-limit] instances on this SVM_NAS.fg_oss_1605688496. Poller=AFF plugin=Zapi:Qtree
nabox-harvest2 | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [files-used-pct-soft-file-limit] instances on this NAS_SVM_MC1.VOL_DEMO_CIFS.qtree1 Poller=ClusterA plugin=Zapi:Qtree
nabox-harvest2 | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [disk-used-pct-soft-disk-limit] instances on this NAS_SVM_MC1.VOL_DEMO_CIFS.qtree1 Poller=ClusterA plugin=Zapi:Qtree
nabox-harvest2 | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [disk-used-pct-threshold] instances on this NAS_SVM_MC1.VOL_DEMO_CIFS.qtree1 Poller=ClusterA plugin=Zapi:Qtree
nabox-harvest2 | 8:05AM ERR goharvest2/cmd/poller/collector/collector.go:375 > plugin [Qtree]: error="no instances => no quota instances found" Poller=ClusterB collector=Zapi:Qtree stack=[{"func":"New","line":"35","source":"errors.go"},{"func":"(*Quota).Run","line":"156","source":"quota.go"},{"func":"(*AbstractCollector).Start","line":"374","source":"collector.go"},{"func":"goexit","line":"1581","source":"asm_amd64.s"}]
You can save those logs in a file and create an archive like so :
dc logs nabox-api > nabox-api.log; dc logs nabox-harvest2 > nabox-harvest2.log;\
tar -czf nabox-logs-`date +%Y-%m-%d_%H:%M:%S`.tgz *
Using Prometheus and Graphite
Both Prometheus and Graphite are exposed to the outside so you can browse metrics and test queries.
Point your browser to https://nabox/prometheus/ or https://nabox/graphite/ accordingly
