At first I thought maybe zabbix server was busy, but only one host is having issue.
My second thought: maybe that host was busy and zabbix agent couldn't return value. But all other items are working fine, only item "product_diff" is broken.
I tried zabbix_get manually, I could get value without problem.
$ zabbix_get -s centos-1.local.vb -k product_diff
-398
$ zabbix_get -s centos-1.local.vb -k product_diffAnd in the zabbix_server.log, I saw entries like this:
-205
31168:20160322:095648.251 item "centos-1.local.vb:product_diff" became not supported: Received value [-299] is not suitable for value type [Numeric (unsigned)] and data type [Decimal]
So centos-1.local.vb returned negative values while zabbix server was expecting Numeric (unassigned).
It turns out that, we configured zabbix to monitor product_diff on centos-1.local.vb, initially we were only interested in the absolute difference. But recently production team felt the real difference makes more sense to them. So they replaced the script on centos-1.local.vb.
But inside zabbix server, the item configuration was not changed
In item configuration, product_diff is configured like this:
I updated "Type of Information" from "Numeric (unassigned)" to "Numeric (float)"
Monitoring should work now, but it doesn't, and I still see errors in zabbix_server.log
In zabbix, all the configurations are kept in database, so maybe the changes made on web UI didn't update database.
Logging in to zabbix database, check the item table:
MYSQL> select key_, value_type from items where key_ = 'product_diff';
+---------------------------+
| key_ | value_type |
+---------------------------+
| product_diff | 3 |
+--------------+------------+
In zabbix value_type 3 represents Numeric (unassigned), 0 represents Numeric (float). So we need to update this table, setting value_type to 0.
MYSQL> update items set value_type = 0 key_ = 'product_diff';
After that, monitoring started working again!
