various alerts numbers

[ltrilety]

The number of cluster alerts does not always correspond to the other available alerts numbers. Moreover those numbers not always correspond with actual state of the cluster.

E.g.

• I had 2 as number of alerts on Clusters page, whilst related Volumes and Hosts shows all 0. The same for the alert list.
• In another case I had 1 alert on the Clusters, related Volumes and Hosts shows all 0 and Alerts list shows 3 active alerts.

In both cases the cluster was completely up and running.

Reproduction steps:

1. kill some bricks, let the tendrl notice (especially there should be alert about degraded state of a volume)
2. stop glusterd services on all nodes and wait few seconds
3. start glusterd services on all nodes so all bricks are running again
4. If needed repeat previous steps (in my case just stopping and starting glusterd was enough)

Tendrl version:
tendrl-commons-1.6.1-1.el7.centos.noarch
tendrl-api-1.6.1-1.el7.centos.noarch
tendrl-ui-1.6.1-1.el7.centos.noarch
tendrl-grafana-selinux-1.5.4-2.el7.centos.noarch
tendrl-ansible-1.6.1-1.el7.centos.noarch
tendrl-notifier-1.6.0-1.el7.centos.noarch
tendrl-node-agent-1.6.1-1.el7.centos.noarch
tendrl-api-httpd-1.6.1-1.el7.centos.noarch
tendrl-selinux-1.5.4-2.el7.centos.noarch
tendrl-grafana-plugins-1.6.1-1.el7.centos.noarch
tendrl-monitoring-integration-1.6.1-1.el7.centos.noarch

[GowthamShanmugam]

@ltrilety I can't reproduce with the latest master, need more info

[GowthamShanmugam]

If you are able to reproduce it please share screenshot also

[cybernth]

@ltrilety , Please provide the information requested. If not reproducible, close the issue

[GowthamShanmugam]

@ltrilety need more info

[ltrilety]

@GowthamShanmugam I tried it once more. You're right the scenario didn't bring the issue right away. However when I let the tendrl run (about 4 days) the issue was hit again.
I was doing the same steps, but moreover I stopped one of available volumes and start it again after a while. The result is as follows:
clusters page - 3 alerts
hosts page - 0 alerts
volumes page - 4 alerts
alerts side window - 1 alert

Checked version:
tendrl-ansible-1.6.1-2.el7rhgs.noarch
tendrl-commons-1.6.1-1.el7rhgs.noarch
tendrl-ui-1.6.1-1.el7rhgs.noarch
tendrl-selinux-1.5.4-2.el7rhgs.noarch
tendrl-api-1.6.1-1.el7rhgs.noarch
tendrl-api-httpd-1.6.1-1.el7rhgs.noarch
tendrl-monitoring-integration-1.6.1-1.el7rhgs.noarch
tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch
tendrl-notifier-1.6.0-1.el7rhgs.noarch
tendrl-node-agent-1.6.1-1.el7rhgs.noarch
tendrl-grafana-plugins-1.6.1-1.el7rhgs.noarch

[GowthamShanmugam]

@ltrilety have you unmanaged cluster in that 4 days, Why i am asking this means I feel during unmanage cluster we are missing something
@shtripat ^

[ltrilety]

@GowthamShanmugam nope, not this time. I didn't perform any un-manage of the cluster.

[GowthamShanmugam]

@ltrilety @r0h4n @shtripat problem is when glusterd stopped for some time then TTL for volume object is deleting entire volume details, including an alert count for that volume also. So when glusterd is started then again alert count is initialized by zero but still some warning alert present for that particular volume. So alert count in the cluster is not matching with volume alert count. Entire alert count calculation goes wrong here.

shall we put TTL only for deleted volumes?

[shtripat]

I understand that alert counter is a different object from volume. Even if volume gets deleted due to ttl feel alert counter you should still retain and that should work out. What you say?

[GowthamShanmugam]

@r0h4n @shirshendu i had discuss with @shtripat about this problem, So we are incrementing alert count when any warning alert came and decrementing any info alert came, Problem is any one time increment goes wrong then entire alert calculation will go wrong, So @shtripat suggested me like instead of storing alert count in etcd, we will calculate the alert count in API and send it to UI for the particular cluster will solve this issue.

suggestion please

[shtripat]

@r0h4n @shirshendu @nthomas-redhat why I say/suggest so is that maintaining counters could always be problematic with TTLs in place. What I suggest is that maintain the alerts for volumes and bricks as well at location as below in etcd

/alerting/alerts/volumes/{vol-id}
/alerting/alerts/bricks/{brick-path}

No need to maintain the counters at individual entities level. The object model which REST uses to return details in GET output for these entities can have additional fields for alert counters and during GET call API layer can actually make two get calls to etcd. One for getting the entity and one for counting the no of alerts for the entities by looking at /alerting/alerts dir. Then merge the two details and return details to API client.

@shirshendu thoughts??