Skip to content

FAQ

Problems and resolutions

Adding 7-mode systems

Check that you enabled TLS with options tls.enable on. Also check the following options : httpd.admin.access, httpd.admin.hostsequiv.enable

Grafana Alerting

Grafana 4 supports alerting only for queries that do not make use of variables, so if you edit the panel and you see anything with a '$' sign in the query used to fetch the metrics, alerting will not work. What you can do, is create your own panels without tempate variables and use it to alert.

Network conflict (Docker)

It might happen that NAbox internal network for docker containers overlaps one of your own networks.

When that happens, you notice that you cannot get to NAbox web site from those overlapping networks.

To fix this, you need to log into NAbox with root, and edit /usr/local/nabox/docker-compose/docker-compose.override.yaml.

There is a sample configuration ready to use, you just have to un-comment the whole networks section like so :

# Use this file to customize docker-compose configuration.
# For example, you can change default network CIDR like this :
#
networks:                                                         
  default:                                                            
    driver: bridge                                               
    name: docker-compose_default                                         
    ipam:                                                                 
      driver: default 
      config:                                               
        - subnet: 172.98.0.0/16
          gateway: 172.98.0.1
version: "3.7"

Feel free to change subnetand gateway to values that suits you.

After doing modification to this file, you need to stop the containers :

# dc down
Stopping graphite           ... done
Stopping nabox-admin        ... done
Stopping nabox-harvest      ... done
Stopping nabox-api          ... done
Stopping nabox-harvest2     ... done
Stopping grafana            ... done
Stopping prometheus         ... done
Stopping traefik            ... done
Stopping container-exporter ... done
Stopping go-carbon          ... done
Removing graphite           ... done
Removing nabox-admin        ... done
Removing nabox-harvest      ... done
Removing nabox-api          ... done
Removing nabox-harvest2     ... done
Removing grafana            ... done
Removing prometheus         ... done
Removing traefik            ... done
Removing container-exporter ... done
Removing go-carbon          ... done
Removing network docker-compose_default

After a few seconds, NAbox will restart everything with the right network configuration.

Reset Network configuration

If you find yourself stuck in a situation where NAbox has no IP address configured, and you have to manually reconfigure from the console, you can use one of those two methods :

Simply edit /etc/network/interfaces, and use the following for static configuration :

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0

iface eth0 inet static
address 192.168.0.101
netmask 255.255.255.0
gateway 192.168.0.1

Or this file if you use DHCP :

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0

iface eth0 inet dhcp

Then restart network :

service networking restart

There is a much more convoluted way to change IP configuration, indicated here more as a proof of concept/demonstration than for actual practical method :

# For static IP configuration
curl -X POST -uadmin:Netapp01 -H "Content-type: application/json" -d '{
  "hostname": "nabox",
  "ip": {
    "dns": [
      "192.168.0.100"
    ],
    "domain": "company.com",
    "gateway": "192.168.0.1",
    "ip_address": "192.168.0.100",
    "netmask": "255.255.255.0"
  },
  "use_dhcp": false
}' http://`docker inspect nabox-api|jq -r '.[0].NetworkSettings.Networks["docker-compose_default"].IPAddress'`:5000/api/1.0/system/network-config

# For DHCP configuration
curl -X POST -uadmin:Netapp01 -H "Content-type: application/json" -d '{
  "hostname": "nabox",
  "use_dhcp": true
}' http://`docker inspect nabox-api|jq -r '.[0].NetworkSettings.Networks["docker-compose_default"].IPAddress'`:5000/api/1.0/system/network-config

General questions

Migrating from older versions

While there is no direct upgrade path from NAbox 2.x to 3.x, there are options to make your migration easier.

Your fastest and easiest way to move to NAbox 3 is to run both NAbox side-by-side, and wait for NAbox 2.6 data to age out and be not relevant for you.

If you absolutely need to run a single virtual appliance, know that you can actually run Harvest 1.6 in NAbox 3, and have both Harvest 1.6 and Harvest 2.x running at the same time, the former storing into Graphite, the later into Prometheus.

Migrating configurations

Again, NAbox 3 does not directly import cluster configurations from NAbox 2, so it will have to be manual. You can leverage the full API provided with NAbox 3 to make the switch easier, for example using a script reading Harvest 1.6 configuration file, and executing REST calls to configure the systems into NAbox 3.

Here is an example to get you started : you can extract configuration of all the clusters configured into NAbox 2.x with this command line :

sed -E -n "/^\[/,/^$/{s/^[^=]*= *//g;H;/^$/{x;s/\n/,/g;p;x;h;};}" /opt/conf/harvest/netapp-harvest.conf |grep -E -v ",\[(global|default)\]"
,[AFF],aff8040,Paris,admin,sillyP@ssWd,60,default,cdot-policy-group.conf,
,[ClusterA],mcca,LAB,wfa,sillyP@ssWd,60,default,cdot-policy-group.conf,
,[ClusterB],mccb,LAB,wfa,sillyP@ssWd,60,default,cdot-policy-group.conf,

Root access

At your own risk, for troubleshooting or customizing purpose, you can access the virtual appliance as root with the following password : NetappGrafanaVA. You can change the root password from the web interface.

Managing metrics

Delete data

Metrics can be delete from Prometheus using the admin API, which is disabled by default for security reasons.

  1. Enable Prometheus Admin API

    • Login to NAbox using SSH and edit /usr/local/nabox/docker-compose/docker-compose.yaml
    • Around line 228, right after - --storage.tsdb.retention.time=2y add - --web.enable-admin-api
    prometheus:                                                                                                                                                                                       
        image: prom/prometheus:latest                                        
        container_name: prometheus                             
        hostname: prometheus                                           
        command:                                                                                        
        - --config.file=/etc/prometheus/prometheus.yml
        - --web.route-prefix=/        
        - --web.external-url=http://example.com/prometheus
        - --storage.tsdb.retention.time=2y                                
        - --web.enable-admin-api                                           
    
  2. Restart Prometheus

    • dc up -d
  3. Delete metrics for a given cluster

    # Delete metrics for cluster2
    curl -k -L -X POST -g \
      'https://localhost/prometheus/api/v1/admin/tsdb/delete_series?match[]={cluster="cluster2"}'
    
    # Purge data from disk
    curl -k -L -X POST http://localhost/prometheus/api/v1/admin/tsdb/clean_tombstones
    

  4. Restore Prometheus configuration

    • Don't forget to remove --web.enable-admin-apifrom arguments in docker-compose.yaml and run dc up -d again.

If you want to manually delete metrics from old systems or just because you need the space, you can safely remove any file or directory in the /opt/graphite/storage/whisper/ directory. Anything you remove here will then disappear from grafana itself, there is no other database or registry. The basic structure is :

/data/graphite/storage/whisper/netapp/perf/<Site>/<Cluster>/node/<Node> /data/graphite/storage/whisper/netapp/perf/<Site>/<Cluster>/svm/<SVM> /data/graphite/storage/whisper/netapp/capacity/<Site>/<Cluster>/node/<Node> /data/graphite/storage/whisper/netapp/capacity/<Site>/<Cluster>/svm/<SVM>

Inside these various directories you will find sub-directories for volumes, lifs, aggregates, etc... You can use rm -rf to remove them.

Delete Old Data

Metrics can be delete from Prometheus using the admin API, which is disabled by default for security reasons.

  1. Enable Prometheus Admin API

    • Login to NAbox using SSH and edit /usr/local/nabox/docker-compose/docker-compose.yaml
    • Around line 228, right after - --storage.tsdb.retention.time=2y add - --web.enable-admin-api
    prometheus:                                                                                                                                                                                       
        image: prom/prometheus:latest                                        
        container_name: prometheus                             
        hostname: prometheus                                           
        command:                                                                                        
        - --config.file=/etc/prometheus/prometheus.yml
        - --web.route-prefix=/        
        - --web.external-url=http://example.com/prometheus
        - --storage.tsdb.retention.time=2y                                
        - --web.enable-admin-api                                           
    
  2. Restart Prometheus

    • dc up -d
  3. Delete metrics older than 6 months

    # Delete metrics older than 6 months (180 days)
    DAYS=180
    curl -k -L -X POST -g \
      'https://localhost/prometheus/api/v1/admin/tsdb/delete_series?match[]={job="harvest2"}&end='$(( `date +%s` - $DAYS * 3600 * 24 ))
    
    # Purge data from disk
    curl -k -L -X POST http://localhost/prometheus/api/v1/admin/tsdb/clean_tombstones
    
  4. Restore Prometheus configuration

    • Don't forget to remove --web.enable-admin-apifrom arguments in docker-compose.yaml and run dc up -d again.

To manually purge old metrics, you can run these command lines directly :

find /data/graphite/storage/whisper -type f -mtime +120 -name \*.wsp -delete
find /data/graphite/storage/whisper -depth -type d -empty -delete

Change default retention

If you find that 2 years of data retention is not appropriate in your environment, you can override the default by editing /usr/local/nabox/docker-compose/docker-compose.override.yaml and implement the following block :

services:
  prometheus:
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --web.route-prefix=/
      - --web.external-url=http://example.com/prometheus
      - --storage.tsdb.retention.time=1y

After the change, you have to restart prometheus :

# dc up -d

Customize Harvest 2

Note

This is a NAbox 3.1.1 feature

Customizing Harvest 2 configuration can be a little bit tricky, and is documented here.

That documentation is mostly valid for NAbox, except for the poller custom configuraton file.

NAbox needs to control the content of custom.yaml because it implements its own set of rules for the default ignore list, so you cannot modify that file (and if you do, it will be overwritten!).

  1. You have to implement customizations in your own file in /opt/harvest2-conf/conf/<poller>/ with a name that starts with custom_, i.e. custom_mycustom_conf.yaml.
  2. The object template file parameter must be a list. So instead of doing this like Harvest docuemntation says :
objects:
  Aggregate: custom_aggr.yaml

you do this :

objects:
  Aggregate:
    - custom_aggr.yaml

That way the different custom_* will be nicely merged together with the stock files provided with NAbox.

Exclude volumes

To ignore volumes that you don't want to be collected by Harvest performance and/or capacity pollers, you can follow these steps :

  1. Create /opt/harvest2-conf/conf/zapiperf/custom_my_ignore_list.yaml

    objects:
      Volume:
        - custom_my_ignore_list_template.yaml
    
  2. Create /opt/harvest2-conf/conf/zapiperf/cdot/9.8.0/custom_my_ignore_list_template.yaml

    plugins:
      LabelAgent:
        exclude_regex:
          - volume `Test_volume.*`
          - volume `Temp_volume.*`
    
  3. We need to do the same for Zapi poller. Create /opt/harvest2-conf/conf/zapi/custom_my_ignore_list.yaml

    objects:
      Volume:
        - custom_my_ignore_list_template.yaml
    
  4. Create /opt/harvest2-conf/conf/zapi/cdot/9.8.0/custom_my_ignore_list_template.yaml

    plugins:
      LabelAgent:
        exclude_regex:
          - volume `Test_volume.*`
          - volume `Temp_volume.*`
    
  5. Restart Harvest

    dc restart nabox-harvest2
    

Customize Grafana

Configure email server

  1. Edit /usr/local/nabox/docker-compose/docker-compose.override.yaml

  2. Configure custom environment variables for grafanaservice.

    services:
      grafana:
        environment:
          - GF_SMTP_ENABLED=true
          - GF_SMTP_HOST=mysmtpserver.mydomain.com:25
          - GF_SMTP_FROM_ADDRESS=grafana@mydomain.com
          - GF_SERVER_DOMAIN=nabox.mydomain.com
          - GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s/grafana/
          - GF_SERVER_SERVE_FROM_SUBATH=true
    

  3. Restart Grafana
    dc up -d grafana