Active items
Active items
Passive Zabbix items are fine if you can connect to all of the monitored hosts from the Zabbix server, but what if you can't allow incoming connections to the monitored hosts because of security or network topology reasons?
This is where active items come into play. As opposed to passive items, for active items, it's the agent that connects to the server; the server never connects to the agent. When connecting, the agent downloads a list of items to check and then reports the new data to the server periodically. Let's create an active item, but this time, we'll try to use some help when selecting the item key:
- Go to Configuration | Hosts
- Click on Items next to Another host
- Click on Create item
For now, use these values:
- Name: Incoming traffic on interface $1
- Type: Zabbix agent (active)
- Update interval: 60s
- History storage period: 7d
Click on the Select button and, in the upcoming dialog that we saw before, click on net.if.in[if,<mode>]. This will fill in the chosen string, as follows:
If your system has a different network interface name, use that here instead of eth0. You can find out the interface names with the ifconfig or ip addr show commands. In many modern distributions, the standard ethX naming scheme has been changed to one that will result in various different interface names such as enp0s3 and em1. Further, replace any occurrences of eth0 with the correct interface name:
Well, it doesn't look like they have. You could wait a bit to be completely sure, but most likely, no data will appear for this new active item, which means we're in for another troubleshooting session.
First, we should test basic network connectivity. Remember, active agents connect to the server, so we have to know which port they use (by default, it's port 10051). So, let's start by testing whether the remotely monitored machine can connect to the Zabbix server:
$ telnet <Zabbix server IP or DNS name> 10051
This should produce output similar to the following:
Trying <Zabbix server IP>... Connected to <Zabbix server IP or DNS name>. Escape character is '^]'.
Press Ctrl + ] and enter quit in the resulting prompt:
telnet> quit Connection closed.
Such a sequence indicates that the network connection is working properly. If it isn't, verify possible network configuration issues, including network firewalls and the local firewall on the Zabbix server. Make sure to allow incoming connections on port 10051:
# To check you local firewall rules run
# For iptables
iptables -S
# For firewalld
$ firewall-cmd --list-all
So, there might be something wrong with the agent; let's take a closer look. We could try to look at the agent daemon's log file, so find the LogFile configuration parameter. If you're using the default configuration files from the source archive, it should be set to log to /tmp/zabbix_agentd.log. If you installed from packages, it is likely to be in /var/log/zabbix or similar. Open this log file and look for any interesting messages regarding active checks. Each line will be prefixed with PID and timestamp in the syntax,?PID:YYYYMMDD:HHMMSS. You'll probably see lines similar to these:
15794:20141230:153731.992 active check configuration update from [127.0.0.1:10051] started to fail (cannot connect to [[127.0.0.1]:10051]: [111] Connection refused)
The agent is trying to request the active check list, but the connection fails. The attempt seems to be wrong—our Zabbix server should be on a different system than the localhost. Let's see how we can fix this. On the remote machine, open the zabbix_agentd.conf configuration file and check the ServerActive parameter. (This file can probably be located under /etc/zabbix/) The default configuration file will have a line like this:
ServerActive=127.0.0.1
This parameter tells the agent where it should connect to for active items. In our case, the localhost will not work as the Zabbix server is on a remote machine, so we should modify this. Replace 127.0.0.1 with the IP address or DNS name of the Zabbix server, and then restart the agent either using a?systemd script or the manual method:?killall.
While you have the configuration file open, take a look at another parameter there—StartAgents. This parameter controls how many processes are handling incoming connections for passive items. If set to 0, it will prevent the agent from listening on incoming connections from the server. This enables you to customize agents to support either or both of the methods. Disabling passive items can be better from a security perspective, but they are very handy for testing and debugging various problems. Also, some items will only work as passive items. Active items can be disabled by not specifying (commenting out) ServerActive. Disabling both active and passive items won't work; the agent daemon will complain and refuse to start up?and it's correct—starting with both disabled would be a pointless thing to do. Take a look:
zabbix-agentd [16208]: ERROR: either active or passive checks must be enabled
We could wait for values to appear on the frontend again, but again, they would not. Let's return to the agent daemon log file and see whether there is any hint about what's wrong:
15938:20141230:154544.559 no active checks on server [192.168.1.3:10051]: host [Zabbix server] not monitored
If we carefully read the entry, we will notice that the agent is reporting its hostname as Zabbix server, but that is the hostname of the default host, which we decided not to use and left disabled. The log message agrees: it says that the host is not monitored.
If we look at the startup messages, there's even another line mentioning this:
15931:20141230:154544.552 Starting Zabbix Agent [Zabbix server]. Zabbix 4.0.0 (revision 85308)
As that is not the hostname we want to use, let's check the agent daemon configuration file again. There's a parameter named Hostname, which currently reads Zabbix server. Given that the comment for this parameter says Required for active checks and must match hostname as configured on the server, it has to be what we're after. Change the agent configuration parameter to Another host, save and close the configuration file, and then restart the Zabbix agent daemon. Check for new entries in the zabbix_agentd.log file; there should be no more errors.
While we're at it, let's update the agent configuration on A test host as well. Modify zabbix_agentd.conf and set the?Hostname=A test host and restart the agent daemon.
If there still are errors about the host not being found on the server, double-check that the hostname in the Zabbix frontend host properties and agent daemon configuration file (the one we just changed) match.
It's now time to return to the frontend and see whether data has started flowing in at the Monitoring | Latest data section:
- If the decimal value arrived from a Zabbix agent source such as a system.cpu.load item, the item will turn up unsupported
- If Zabbix received an integer but further calculations resulted in a decimal number appearing, like with our network item, the decimal part will be discarded
This behavior is depicted in the following diagram:
Keep in mind that, in the worst case scenario, configuration changes might take up to three minutes to propagate to the Zabbix agent—one minute to get into the server configuration cache and two minutes until the agent refreshes its own item list. On top of this delay, this item is different from the others we created—it needs to gather two values to compute per second, one of which we are interested in; hence, we will also have to wait for whatever the item interval is before the first value appears in the frontend.
That's better; Zabbix now automatically calculates the change between every two checks (that's what the delta is for) and stores it, but the values still don't seem to be too user friendly. Maybe they're better in the graph—let's click on the Graph link to find out:
Still, looking at the Y axis in that graph, we see the incoming values being represented as a number without any explanation of what it is, and larger values get K, M, and other multiplier identifiers applied. It would be so much better if Zabbix knew how to calculate it in bytes or a similar unit:
- Navigate to Configuration | Hosts.
- Click on Items next to Another host.
- Click on the Incoming traffic on the enp0s3 (or whatever your interface is)?interface in the Name?column. Edit the Units field and enter Bps
- Click on Update.
Let's check whether there's any improvement in the Monitoring | Latest data:
One parameter that we set, the update interval, could have been smaller, hence resulting in a better-looking graph. But it is important to remember that the smaller the intervals you have on your items, the more data Zabbix has to retrieve and, each second, more data has to be inserted into the database and more calculations have to be performed when displaying this data. While it would have made no notable difference on our test system; you should try to keep intervals as large as possible.
So far, we have created items that gathered numeric data—either integers or decimal values. Let's create another one, a bit different this time:
- As usual, go to Configuration | Hosts.
- Click on Items next to Another host. Before continuing with item creation, let's look at what helpful things are available in the configuration section, particularly for items. If we look above the item list, we can see the navigation and information bar.
This area provides quick and useful information about the currently selected host—the hostname, whether the host is monitored, and its availability. Even more importantly, on the right-hand side, it provides quick shortcuts back to the host list and other elements associated with the current host—applications, items, triggers, graphs, discovery rules, and web scenarios. This is a handy way to switch between element categories for a single host without going through the host list all the time. But that's not all yet.
- Click on the Filter button to open the filter we got thrown in our face before. The sophisticated filter appears again:
- Clear out the Host field
- Choose Linux servers from the Host group field
- Click on the Apply button below the filter
Now, look right below the main item filter—that is a Subfilter, which, as its header informs, only affects data already filtered by the main filter.
The entries in the subfilter work like toggles—if we switch one on, it works as a filter on the data in addition to all other toggled subfilter controls. Let's click on Zabbix agent (active) now. Notice how the item list now contains only one item; this is what the number 1 represented next to this Subfilter toggle. But the subfilter itself now also looks different:
When you have many different hosts monitored by Zabbix, it's quite easy to forget which version of the Zabbix agent daemon each host has, and even if you have automated software deploying in place, it is nice to be able to see which version each host is at, all in one place.
Use the following values:
- Name: Enter Zabbix agent version
- Type: Select Zabbix agent (active) (we're still creating active items)
- Key: Click on Select and then choose the third entry from the list—agent.version
- Type of information: Choose Character
- Update interval: Enter 86400s
When done! Click on the Add button. There are two notable things we did. Firstly, we set the information type to Character, which reloaded the form, slightly changing available options. Most notably, fields that are relevant for numeric information were hidden, such as units, multiplier, and trends.
Secondly, we entered a very large update interval, 86400, which is equivalent to 24 hours. While this might seem excessive, remember what we will be monitoring here, the Zabbix agent version, so it probably (hopefully) won't be changing several times per day. Depending on your needs, you might set it to even larger values, such as a week.
To check out the results of our work, go to Monitoring | Latest data.
If you don't see the data, wait a while; it should appear eventually. When it does, you should see the version of the Zabbix agent installed on the listed remote machine, and it might be a higher number than displayed here, as newer versions of Zabbix have been released. Notice one minor difference—while all the items we added previously have links named Graph on the right-hand side, the last one has one called History. The reason is simple—for textual items, graphs can't be drawn, so Zabbix does not even attempt to do that.
Now, about that waiting—why did we have to wait for the data to appear? Well, remember how active items work? The agent queries the server for the item list it should report on and then sends in data periodically, but this checking of the item list is also done periodically. To find out how often, open the zabbix_agentd.conf configuration file on the remote machine and look for the RefreshActiveChecks parameter. The default is two minutes, which is configured in seconds, hence listing 120 seconds.
So, in the worst case, you might have had to wait for nearly three minutes to see any data as opposed to normal or passive items, where the server would have queried the agent as soon as the configuration change was available in its cache. In a production environment with many agents using active items, it might be a good idea to increase this value. Usually, item parameters aren't changed that often.