Skip to content

Troubleshooting

NAbox Architecture

Most of NAbox components are delivered as containers inside the virtual appliance.

Containers are managed by docker-compose, you can find the files for the docker-compose environment in /usr/local/nabox/docker-compose/.

File Description
docker-compose.yaml Main configuration file, it should not be modified because it will be overwritten with each update
docker-compose.override.yaml Override file, you can customize docker-compose environement by making changes here
.env Environment file, some critical environment variables are defined here

You can interact with the compose stack with the dc command line, which is an alias to docker-compose with the right arguments.

nabox:~# dc ps
       Name                     Command                  State                                                     Ports                                              
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
container-exporter   /bin/container-exporter -- ...   Up             9104/tcp                                                                                         
go-carbon            /init/run.sh                     Up (healthy)   2003/tcp, 2003/udp, 2004/tcp, 7002/tcp, 7003/tcp, 7007/tcp, 8080/tcp                             
grafana              /run.sh                          Up             3000/tcp                                                                                         
graphite             /bin/sh -c /run.sh               Up             80/tcp                                                                                           
nabox-admin          /docker-entrypoint.sh ngin ...   Up             80/tcp                                                                                           
nabox-api            python api.py                    Up             5000/tcp                                                                                         
nabox-harvest        python3 harvest.py               Up             5000/tcp                                                                                         
nabox-harvest2       python3 harvest.py               Up             5000/tcp                                                                                         
prometheus           /bin/prometheus --config.f ...   Up             9090/tcp                                                                                         
traefik              /entrypoint.sh --api.insec ...   Up             0.0.0.0:2003->2003/tcp,:::2003->2003/tcp, 0.0.0.0:443->443/tcp,:::443->443/tcp,                  

nabox-api:~# dc logs -f --tail 10 nabox-api
Attaching to nabox-api
nabox-api             | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:14] "GET /api/1.0/system/disks HTTP/1.1" 200 -
nabox-api             | 172.99.0.9 - - [01/Apr/2022 10:01:19] "GET /api/1.0/dashboards/nullpointmode HTTP/1.1" 200 -
nabox-api             | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:19] "GET /api/1.0/dashboards/nullpointmode HTTP/1.1" 200 -
nabox-api             | 172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/upgrade HTTP/1.1" 200 -
nabox-api             | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/upgrade HTTP/1.1" 200 -
nabox-api             | 172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/version HTTP/1.1" 200 -
nabox-api             | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/version HTTP/1.1" 200 -
nabox-api             | WARNING:root:{'text': 'harvest version 22.02.0-4 (commit 9484df6) (build date 2022-02-15T09:11:03-0500) linux/amd64', 'version': '22.02.0-4'}
nabox-api             | 172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/packages HTTP/1.1" 200 -
nabox-api             | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/packages HTTP/1.1" 200 -

Containers

The following containers are deployed with NAbox :

Container Description
container-exporter Exports usage statistics for NAbox dashboard
go-carbon Ingests metrics coming from NetApp Harvest 1.x
grafana Grafana Dashboards
graphite Presents metrics coming from NetApp Harvest 1.x to Grafana
nabox-admin Web interface
nabox-api NAbox API server
nabox-harvest NetApp Harvest 1.x
nabox-harvest2 NetApp Harvest 2.x
prometheus Prometheus stores metrics coming from NetApp Harvest 2.x
traefik Routes HTTPS trafic to different containers

Collecting logs

Most issues can be troubleshoot by looking at logs for nabox-api and nabox-harvest2 containers :

nabox:~# dc logs -f --tail 10 nabox-api
Attaching to nabox-api
nabox-api             | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:14] "GET /api/1.0/system/disks HTTP/1.1" 200 -
nabox-api             | 172.99.0.9 - - [01/Apr/2022 10:01:19] "GET /api/1.0/dashboards/nullpointmode HTTP/1.1" 200 -
nabox-api             | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:19] "GET /api/1.0/dashboards/nullpointmode HTTP/1.1" 200 -
nabox-api             | 172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/upgrade HTTP/1.1" 200 -
nabox-api             | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/upgrade HTTP/1.1" 200 -
nabox-api             | 172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/version HTTP/1.1" 200 -
nabox-api             | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/version HTTP/1.1" 200 -
nabox-api             | WARNING:root:{'text': 'harvest version 22.02.0-4 (commit 9484df6) (build date 2022-02-15T09:11:03-0500) linux/amd64', 'version': '22.02.0-4'}
nabox-api             | 172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/packages HTTP/1.1" 200 -
nabox-api             | INFO:werkzeug:172.99.0.9 - - [01/Apr/2022 10:01:27] "GET /api/1.0/system/packages HTTP/1.1" 200 -

nabox:~# dc logs -f --tail 10 nabox-harvest2
Attaching to nabox-harvest2
nabox-harvest2        | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [files-used-pct-file-limit] instances on this SVM_NAS.fg_oss_1605688496.bucket03 Poller=AFF plugin=Zapi:Qtree
nabox-harvest2        | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [files-used-pct-soft-file-limit] instances on this SVM_NAS.fg_oss_1605688496. Poller=AFF plugin=Zapi:Qtree
nabox-harvest2        | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [disk-used-pct-disk-limit] instances on this SVM_NAS.fg_oss_1605688496. Poller=AFF plugin=Zapi:Qtree
nabox-harvest2        | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [disk-used-pct-soft-disk-limit] instances on this SVM_NAS.fg_oss_1605688496. Poller=AFF plugin=Zapi:Qtree
nabox-harvest2        | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [disk-used-pct-threshold] instances on this SVM_NAS.fg_oss_1605688496. Poller=AFF plugin=Zapi:Qtree
nabox-harvest2        | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [files-used-pct-file-limit] instances on this SVM_NAS.fg_oss_1605688496. Poller=AFF plugin=Zapi:Qtree
nabox-harvest2        | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [files-used-pct-soft-file-limit] instances on this NAS_SVM_MC1.VOL_DEMO_CIFS.qtree1 Poller=ClusterA plugin=Zapi:Qtree
nabox-harvest2        | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [disk-used-pct-soft-disk-limit] instances on this NAS_SVM_MC1.VOL_DEMO_CIFS.qtree1 Poller=ClusterA plugin=Zapi:Qtree
nabox-harvest2        | 8:05AM WRN goharvest2/cmd/collectors/zapi/plugins/quota/quota.go:180 > no [disk-used-pct-threshold] instances on this NAS_SVM_MC1.VOL_DEMO_CIFS.qtree1 Poller=ClusterA plugin=Zapi:Qtree
nabox-harvest2        | 8:05AM ERR goharvest2/cmd/poller/collector/collector.go:375 > plugin [Qtree]:  error="no instances => no quota instances found" Poller=ClusterB collector=Zapi:Qtree stack=[{"func":"New","line":"35","source":"errors.go"},{"func":"(*Quota).Run","line":"156","source":"quota.go"},{"func":"(*AbstractCollector).Start","line":"374","source":"collector.go"},{"func":"goexit","line":"1581","source":"asm_amd64.s"}]

You can save those logs in a file and create an archive like so :

dc logs nabox-api > nabox-api.log; dc logs nabox-harvest2 > nabox-harvest2.log;\
  tar -czf nabox-logs-`date +%Y-%m-%d_%H:%M:%S`.tgz *

Using Prometheus and Graphite

Both Prometheus and Graphite are exposed to the outside so you can browse metrics and test queries.

Point your browser to https://nabox/prometheus/ or https://nabox/graphite/ accordingly

Prometheus