1 # <a id="monitoring-remote-systems"></a> Monitoring Remote Systems
3 There are multiple ways you can monitor remote clients. Be it using [agent-less](8-monitoring-remote-systems.md#agent-less-checks)
4 or [agent-based](agent-based-checks-addons) using additional addons & tools.
6 Icinga 2 uses its own unique and secure communitication protol amongst instances.
7 Be it an High-Availability cluster setup, distributed load-balanced setup or just a single
8 agent [monitoring a remote client](8-monitoring-remote-systems.md#icinga2-remote-client-monitoring).
10 All communication is secured by TLS with certificates, and fully supports IPv4 and IPv6.
12 If you are planning to use the native Icinga 2 cluster feature for distributed
13 monitoring and high-availability, please continue reading in
14 [this chapter](8-monitoring-remote-systems.md#distributed-monitoring-high-availability).
18 > Don't panic - there are CLI commands available, including setup wizards for easy installation
19 > with SSL certificates.
20 > If you prefer to use your own CA (for example Puppet) you can do that as well.
22 ## <a id="agent-less-checks"></a> Agent-less Checks
24 If the remote service is available using a network protocol and port,
25 and a [check plugin](2-getting-started.md#setting-up-check-plugins) is available, you don't
26 necessarily need a local client installed. Rather choose a plugin and
27 configure all parameters and thresholds. The [Icinga 2 Template Library](6-icinga-template-library.md#icinga-template-library)
28 already ships various examples like
30 * [ping4](6-icinga-template-library.md#plugin-check-command-ping4), [ping6](6-icinga-template-library.md#plugin-check-command-ping6),
31 [fping4](6-icinga-template-library.md#plugin-check-command-fping4), [fping6](6-icinga-template-library.md#plugin-check-command-fping6), [hostalive](6-icinga-template-library.md#plugin-check-command-hostalive)
32 * [tcp](6-icinga-template-library.md#plugin-check-command-tcp), [udp](6-icinga-template-library.md#plugin-check-command-udp), [ssl](6-icinga-template-library.md#plugin-check-command-ssl)
33 * [http](6-icinga-template-library.md#plugin-check-command-http), [ftp](6-icinga-template-library.md#plugin-check-command-ftp)
34 * [smtp](6-icinga-template-library.md#plugin-check-command-smtp), [ssmtp](6-icinga-template-library.md#plugin-check-command-ssmtp),
35 [imap](6-icinga-template-library.md#plugin-check-command-imap), [simap](6-icinga-template-library.md#plugin-check-command-simap),
36 [pop](6-icinga-template-library.md#plugin-check-command-pop), [spop](6-icinga-template-library.md#plugin-check-command-spop)
37 * [ntp_time](6-icinga-template-library.md#plugin-check-command-ntp-time)
38 * [ssh](6-icinga-template-library.md#plugin-check-command-ssh)
39 * [dns](6-icinga-template-library.md#plugin-check-command-dns), [dig](6-icinga-template-library.md#plugin-check-command-dig), [dhcp](6-icinga-template-library.md#plugin-check-command-dhcp)
41 There are numerous check plugins contributed by community members available
42 on the internet. If you found one for your requirements, [integrate them into Icinga 2](3-monitoring-basics.md#command-plugin-integration).
46 * [Icinga Exchange](https://exchange.icinga.org)
47 * [Icinga Wiki](https://wiki.icinga.org)
49 An example is provided in the sample configuration in the getting started
50 section shipped with Icinga 2 ([hosts.conf](4-configuring-icinga-2.md#hosts-conf), [services.conf](4-configuring-icinga-2.md#services-conf)).
52 ## <a id="icinga2-remote-client-monitoring"></a> Monitoring Icinga 2 Remote Clients
54 First, you should decide which role the remote client has:
56 * a single host with local checks and configuration
57 * a remote satellite checking other hosts (for example in your DMZ)
58 * a remote command execution client (similar to NRPE, NSClient++, etc)
60 Later on, you will be asked again and told how to proceed with these
61 different [roles](8-monitoring-remote-systems.md#icinga2-remote-monitoring-client-roles).
65 > If you are planning to build an Icinga 2 distributed setup using the cluster feature, please skip
66 > the following instructions and jump directly to the
67 > [cluster setup instructions](8-monitoring-remote-systems.md#distributed-monitoring-high-availability).
71 > Remote instances are independent Icinga 2 instances which schedule
72 > their checks and just synchronize them back to the defined master zone.
74 ## <a id="icinga2-remote-monitoring-master"></a> Master Setup for Remote Monitoring
76 If you are planning to use the [remote Icinga 2 clients](8-monitoring-remote-systems.md#icinga2-remote-monitoring-client)
77 you'll first need to update your master setup.
79 Your master setup requires the following
81 * SSL CA and signed certificate for the master
82 * Enabled API feature, and a local Endpoint and Zone object configuration
83 * Firewall ACLs for the communication port (default 5665)
85 You can use the [CLI command](7-cli-commands.md#cli-command-node) `node wizard` for setting up a new node
86 on the master. The command must be run as root, all Icinga 2 specific files
87 will be updated to the icinga user the daemon is running as (certificate files
90 Make sure to answer the first question with `n` (no).
94 Welcome to the Icinga 2 Setup Wizard!
96 We'll guide you through all required configuration details.
98 If you have questions, please consult the documentation at http://docs.icinga.org
99 or join the community support channels at https://support.icinga.org
102 Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: n
103 Starting the Master setup routine...
104 Please specifiy the common name (CN) [icinga2m]:
105 information/base: Writing private key to '/var/lib/icinga2/ca/ca.key'.
106 information/base: Writing X509 certificate to '/var/lib/icinga2/ca/ca.crt'.
107 information/cli: Initializing serial file in '/var/lib/icinga2/ca/serial.txt'.
108 information/cli: Generating new CSR in '/etc/icinga2/pki/icinga2m.csr'.
109 information/base: Writing private key to '/etc/icinga2/pki/icinga2m.key'.
110 information/base: Writing certificate signing request to '/etc/icinga2/pki/icinga2m.csr'.
111 information/cli: Signing CSR with CA and writing certificate to '/etc/icinga2/pki/icinga2m.crt'.
112 information/cli: Copying CA certificate to '/etc/icinga2/pki/ca.crt'.
113 information/cli: Dumping config items to file '/etc/icinga2/zones.conf'.
114 Please specify the API bind host/port (optional):
117 information/cli: Enabling the APIlistener feature.
118 information/cli: Updating constants.conf.
119 information/cli: Updating constants file '/etc/icinga2/constants.conf'.
120 information/cli: Updating constants file '/etc/icinga2/constants.conf'.
121 information/cli: Edit the constants.conf file '/etc/icinga2/constants.conf' and set a secure 'TicketSalt' constant.
124 Now restart your Icinga 2 daemon to finish the installation!
126 If you encounter problems or bugs, please do not hesitate to
127 get in touch with the community at https://support.icinga.org
130 The setup wizard will do the following:
132 * Generate a local CA in `/var/lib/icinga2/ca` or use the existing one
133 * Generate a new CSR, sign it with the local CA and copying it into `/etc/icinga2/pki`
134 * Generate a local zone and endpoint configuration for this master based on FQDN
135 * Enabling the API feature, and setting optional `bind_host` and `bind_port`
136 * Setting the `NodeName` and `TicketSalt` constants in [constants.conf](4-configuring-icinga-2.md#constants-conf)
138 The setup wizard does not automatically restart Icinga 2.
143 > This setup wizard will install a standalone master, HA cluster scenarios are currently
148 ## <a id="icinga2-remote-monitoring-client"></a> Client Setup for Remote Monitoring
150 Icinga 2 can be installed on Linux/Unix and Windows. While
151 [Linux/Unix](8-monitoring-remote-systems.md#icinga2-remote-monitoring-client-linux) will be using the [CLI command](7-cli-commands.md#cli-command-node)
152 `node wizard` for a guided setup, you will need to use the
153 graphical installer for Windows based client setup.
155 Your client setup requires the following
157 * A ready configured and installed [master node](8-monitoring-remote-systems.md#icinga2-remote-monitoring-master)
158 * SSL signed certificate for communication with the master (Use [CSR auto-signing](certifiates-csr-autosigning)).
159 * Enabled API feature, and a local Endpoint and Zone object configuration
160 * Firewall ACLs for the communication port (default 5665)
164 ### <a id="icinga2-remote-monitoring-client-linux"></a> Linux Client Setup for Remote Monitoring
166 #### <a id="csr-autosigning-requirements"></a> Requirements for CSR Auto-Signing
168 If your remote clients are capable of connecting to the central master, Icinga 2
169 supports CSR auto-signing.
171 First you'll need to define a secure ticket salt in the [constants.conf](4-configuring-icinga-2.md#constants-conf).
172 The [setup wizard for the master setup](8-monitoring-remote-systems.md#icinga2-remote-monitoring-master) will create
175 # grep TicketSalt /etc/icinga2/constants.conf
177 The client setup wizard will ask you to generate a valid ticket number using its CN.
178 If you already know your remote client's Common Names (CNs) - usually the FQDN - you
179 can generate all ticket numbers on-demand.
181 This is also reasonable if you are not capable of installing the remote client, but
182 a colleague of yours, or a customer.
184 Example for a client notebook:
186 # icinga2 pki ticket --cn nbmif.int.netways.de
190 > You can omit the `--salt` parameter using the `TicketSalt` constant from
191 > [constants.conf](4-configuring-icinga-2.md#constants-conf) if already defined and Icinga 2 was
192 > reloaded after the master setup.
194 #### <a id="certificates-manual-creation"></a> Manual SSL Certificate Generation
196 This is described separately in the [cluster setup chapter](8-monitoring-remote-systems.md#manual-certificate-generation).
200 > If you're using [CSR Auto-Signing](8-monitoring-remote-systems.md#csr-autosigning-requirements), skip this step.
203 #### <a id="icinga2-remote-monitoring-client-linux-setup"></a> Linux Client Setup Wizard for Remote Monitoring
205 Install Icinga 2 from your distribution's package repository as described in the
206 general [installation instructions](2-getting-started.md#setting-up-icinga2).
208 Please make sure that either [CSR Auto-Signing](8-monitoring-remote-systems.md#csr-autosigning-requirements) requirements
209 are fulfilled, or that you're using [manual SSL certificate generation](8-monitoring-remote-systems.md#manual-certificate-generation).
213 > You don't need any features (DB IDO, Livestatus) or user interfaces on the remote client.
214 > Install them only if you're planning to use them.
216 Once the package installation succeeded, use the `node wizard` CLI command to install
217 a new Icinga 2 node as client setup.
219 You'll need the following configuration details:
221 * The client common name (CN). Defaults to FQDN.
222 * The client's local zone name. Defaults to FQDN.
223 * The master endpoint name. Look into your master setup `zones.conf` file for the proper name.
224 * The master endpoint connection information. Your master's IP address and port (defaults to 5665)
225 * The [request ticket number](8-monitoring-remote-systems.md#csr-autosigning-requirements) generated on your master
227 * Bind host/port for the Api feature (optional)
229 The command must be run as root, all Icinga 2 specific files will be updated to the icinga
230 user the daemon is running as (certificate files for example).
233 # icinga2 node wizard
235 Welcome to the Icinga 2 Setup Wizard!
237 We'll guide you through all required configuration details.
239 If you have questions, please consult the documentation at http://docs.icinga.org
240 or join the community support channels at https://support.icinga.org
243 Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]:
244 Starting the Node setup routine...
245 Please specifiy the common name (CN) [nbmif.int.netways.de]:
246 Please specifiy the local zone name [nbmif.int.netways.de]:
247 Please specify the master endpoint(s) this node should connect to:
248 Master Common Name (CN from your master setup, defaults to FQDN): icinga2m
249 Please fill out the master connection information:
250 Master endpoint host (required, your master's IP address or FQDN): 192.168.33.100
251 Master endpoint port (optional) []:
252 Add more master endpoints? [y/N]
253 Please specify the master connection for CSR auto-signing (defaults to master endpoint host):
254 Host [192.168.33.100]:
256 information/base: Writing private key to '/var/lib/icinga2/ca/ca.key'.
257 information/base: Writing X509 certificate to '/var/lib/icinga2/ca/ca.crt'.
258 information/cli: Initializing serial file in '/var/lib/icinga2/ca/serial.txt'.
259 information/base: Writing private key to '/etc/icinga2/pki/nbmif.int.netways.de.key'.
260 information/base: Writing X509 certificate to '/etc/icinga2/pki/nbmif.int.netways.de.crt'.
261 information/cli: Generating self-signed certifiate:
262 information/cli: Fetching public certificate from master (192.168.33.100, 5665):
264 information/cli: Writing trusted certificate to file '/etc/icinga2/pki/trusted-master.crt'.
265 information/cli: Stored trusted master certificate in '/etc/icinga2/pki/trusted-master.crt'.
267 Please specify the request ticket generated on your Icinga 2 master.
268 (Hint: '# icinga2 pki ticket --cn nbmif.int.netways.de'):
269 2e070405fe28f311a455b53a61614afd718596a1
270 information/cli: Processing self-signed certificate request. Ticket '2e070405fe28f311a455b53a61614afd718596a1'.
272 information/cli: Writing signed certificate to file '/etc/icinga2/pki/nbmif.int.netways.de.crt'.
273 information/cli: Writing CA certificate to file '/var/lib/icinga2/ca/ca.crt'.
274 Please specify the API bind host/port (optional):
277 information/cli: Disabling the Notification feature.
278 Disabling feature notification. Make sure to restart Icinga 2 for these changes to take effect.
279 information/cli: Enabling the Apilistener feature.
280 information/cli: Generating local zones.conf.
281 information/cli: Dumping config items to file '/etc/icinga2/zones.conf'.
282 information/cli: Updating constants.conf.
283 information/cli: Updating constants file '/etc/icinga2/constants.conf'.
286 Now restart your Icinga 2 daemon to finish the installation!
288 If you encounter problems or bugs, please do not hesitate to
289 get in touch with the community at https://support.icinga.org
292 The setup wizard will do the following:
294 * Generate a new self-signed certificate and copy it into `/etc/icinga2/pki`
295 * Store the master's certificate as trusted certificate for requesting a new signed certificate
296 (manual step when using `node setup`).
297 * Request a new signed certificate from the master and store updated certificate and master CA in `/etc/icinga2/pki`
298 * Generate a local zone and endpoint configuration for this client and the provided master information
300 * Disabling the notification feature for this client
301 * Enabling the API feature, and setting optional `bind_host` and `bind_port`
302 * Setting the `NodeName` constant in [constants.conf](4-configuring-icinga-2.md#constants-conf)
304 The setup wizard does not automatically restart Icinga 2.
306 If you are getting an error when requesting the ticket number, please check the following:
308 * Is the CN the same (from pki ticket on the master and setup node on the client)
309 * Is the ticket expired
312 ### <a id="icinga2-remote-monitoring-client-windows"></a> Windows Client Setup for Remote Monitoring
314 Download the MSI-Installer package from [http://packages.icinga.org/windows/](http://packages.icinga.org/windows/).
317 * [Microsoft .NET Framework 2.0](http://www.microsoft.com/de-de/download/details.aspx?id=1639) if not already installed.
319 The setup wizard will install Icinga 2 and then continue with SSL certificate generation,
320 CSR-Autosigning and configuration setup.
322 You'll need the following configuration details:
324 * The client common name (CN). Defaults to FQDN.
325 * The client's local zone name. Defaults to FQDN.
326 * The master endpoint name. Look into your master setup `zones.conf` file for the proper name.
327 * The master endpoint connection information. Your master's IP address and port (defaults to 5665)
328 * The [request ticket number](8-monitoring-remote-systems.md#csr-autosigning-requirements) generated on your master
330 * Bind host/port for the Api feature (optional)
332 Once install is done, Icinga 2 is automatically started as a Windows service.
334 ### <a id="icinga2-remote-monitoring-client-roles"></a> Remote Monitoring Client Roles
336 Icinga 2 allows you to use two separate ways of defining a client (or: `agent`) role:
338 * execute commands remotely, but host/service configuration happens on the master.
339 * schedule remote checks on remote satellites with their local configuration.
341 Depending on your scenario, either one or both combined with a cluster setup
342 could be build and put together.
345 ### <a id="icinga2-remote-monitoring-client-command-execution"></a> Remote Client for Command Execution
347 This scenario allows you to configure the checkable objects (hosts, services) on
348 your Icinga 2 master or satellite, and only send commands remotely.
351 * Exact same [CheckCommand](5-object-types.md#objecttype-checkcommand) (and
352 [EventCommand](5-object-types.md#objecttype-eventcommand)) configuration objects
353 on the master and the remote client(s).
354 * Installed plugin scripts on the remote client (`PluginDir` constant can be locally modified)
355 * `Zone` and `Endpoint` configuration for the client on the master
356 * `command_endpoint` attribute configured for host/service objects pointing to the configured
359 `CheckCommand` objects are already shipped with the Icinga 2 ITL
360 as [plugin check commands](6-icinga-template-library.md#plugin-check-commands). If you are
361 using your own configuration definitions for example in
362 [commands.conf](4-configuring-icinga-2.md#commands-conf) make sure to copy/sync it
363 on your remote client.
365 #### <a id="icinga2-remote-monitoring-client-command-execution-client"></a> Client Configuration Remote Client for Command Execution
369 > Remote clients must explicitely accept commands in a similar
370 > fashion as cluster nodes [accept configuration]#i(cluster-zone-config-sync).
371 > This is due to security reasons.
373 Edit the `api` feature configuration in `/etc/icinga2/features-enabled/api.conf`
374 and set `accept_commands` to `true`.
376 object ApiListener "api" {
377 cert_path = SysconfDir + "/icinga2/pki/" + NodeName + ".crt"
378 key_path = SysconfDir + "/icinga2/pki/" + NodeName + ".key"
379 ca_path = SysconfDir + "/icinga2/pki/ca.crt"
380 accept_commands = true
383 #### <a id="icinga2-remote-monitoring-client-command-execution-master"></a> Master Configuration Remote Client for Command Execution
385 Add an `Endpoint` and `Zone` configuration object for the remote client
386 in [zones.conf](#zones-conf) and define a trusted master zone as `parent`.
388 object Endpoint "remote-client1" {
389 host = "192.168.33.20"
392 object Zone "remote-client1" {
393 endpoints = [ "remote-client1" ]
398 * [configure endpoints](8-monitoring-remote-systems.md#configure-cluster-endpoints)
399 * [configure zones](8-monitoring-remote-systems.md#configure-cluster-zones)
402 Configuration example for host and service objects running commands on the remote endpoint `remote-client1`:
404 object Host "host-remote" {
405 import "generic-host"
407 address = "127.0.0.1"
412 vars.remote_client = "remote-client1"
414 /* host specific check arguments */
415 vars.users_wgreater = 10
416 vars.users_cgreater = 20
419 apply Service "users-remote" {
420 import "generic-service"
422 check_command = "users"
423 command_endpoint = host.vars.remote_client
425 /* override (remote) command arguments with host settings */
426 vars.users_wgreater = host.vars.users_wgreater
427 vars.users_cgreater = host.vars.users_cgreater
429 /* assign where a remote client is set */
430 assign where host.vars.remote_client
434 That way you can also execute the `icinga` check remotely
435 thus verifying the health of your remote client(s). As a bonus
436 you'll also get the running Icinga 2 version and may
437 schedule client updates in your management tool (e.g. Puppet).
441 > [Event commands](3-monitoring-basics.md#event-commands) are executed on the
442 > remote command endpoint as well. You do not need
443 > an additional transport layer such as SSH or similar.
446 > You cannot add any Icinga 2 features like DB IDO on the remote
447 > clients. There are no local configured objects available.
449 > If you require this, please install a full-featured
450 > [local client](8-monitoring-remote-systems.md#icinga2-remote-monitoring-client-local-config).
452 ### <a id="icinga2-remote-monitoring-client-local-config"></a> Remote Client with Local Configuration
454 This is considered as independant satellite using a local scheduler, configuration
455 and the possibility to add Icinga 2 features on demand.
457 Local configured checks are transferred to the central master and helped
458 with discovery CLI commands.
460 Please follow the instructions closely in order to deploy your fully featured
461 client, or `agent` as others might call it.
463 #### <a id="icinga2-remote-monitoring-client-configuration"></a> Client Configuration for Remote Monitoring
465 There is no difference in the configuration syntax on clients to any other Icinga 2 installation.
467 The following convention applies to remote clients:
469 * The hostname in the default host object should be the same as the Common Name (CN) used for SSL setup
470 * Add new services and check commands locally
472 The default setup routine will install a new host based on your FQDN in `repository.d/hosts` with all
473 services in separate configuration files a directory underneath.
475 The repository can be managed using the CLI command `repository`.
479 > The CLI command `repository` only supports basic configuration manipulation (add, remove). Future
480 > versions will support more options (set, etc.). Please check the Icinga 2 development roadmap
483 You can also use additional features like notifications directly on the remote client, if you are
484 required to. Basically everything a single Icinga 2 instance provides by default.
487 #### <a id="icinga2-remote-monitoring-master-discovery"></a> Discover Client Services on the Master
489 Icinga 2 clients will sync their locally defined objects to the defined master node. That way you can
490 list, add, filter and remove nodes based on their `node`, `zone`, `host` or `service` name.
492 List all discovered nodes (satellites, agents) and their hosts/services:
497 #### <a id="icinga2-remote-monitoring-master-discovery-manual"></a> Manually Discover Clients on the Master
499 Add a to-be-discovered client to the master:
501 # icinga2 node add my-remote-client
503 Set the connection details, and the Icinga 2 master will attempt to connect to this node and sync its
506 # icinga2 node set my-remote-client --host 192.168.33.101 --port 5665
508 You can control that by calling the `node list` command:
511 Node 'my-remote-client' (host: 192.168.33.101, port: 5665, log duration: 1 day, last seen: Sun Nov 2 17:46:29 2014)
513 #### <a id="icinga2-remote-monitoring-master-discovery-remove"></a> Remove Discovered Clients
515 If you don't require a connected agent, you can manually remove it and its discovered hosts and services
516 using the following CLI command:
518 # icinga2 node remove my-discovered-agent
522 > Better use [blacklists and/or whitelists](8-monitoring-remote-systems.md#icinga2-remote-monitoring-master-discovery-blacklist-whitelist)
523 > to control which clients and hosts/services are integrated into your master configuration repository.
525 ### <a id="icinga2-remote-monitoring-master-discovery-generate-config"></a> Generate Icinga 2 Configuration for Client Services on the Master
527 There is a dedicated Icinga 2 CLI command for updating the client services on the master,
528 generating all required configuration.
530 # icinga2 node update-config
532 The generated configuration of all nodes is stored in the `repository.d/` directory.
534 By default, the following additional configuration is generated:
535 * add `Endpoint` and `Zone` objects for the newly added node
536 * add `cluster-zone` health check for the master host detecting if the remote node died
537 * use the default templates `satellite-host` and `satellite-service` defined in `/etc/icinga2/conf.d/satellite.conf`
538 * apply a dependency for all other hosts on the remote satellite prevening failure checks/notifications
542 > If there are existing hosts/services defined or modified, the CLI command will not overwrite these (modified)
543 > configuration files.
545 > If hosts or services disappeared from the client discovery, it will remove the existing configuration objects
546 > from the config repository.
548 The `update-config` CLI command will fail, if there are uncommitted changes for the
549 configuration repository.
550 Please review these changes manually, or clear the commit and try again. This is a
551 safety hook to prevent unwanted manual changes to be committed by a updating the
552 client discovered objects only.
554 # icinga2 repository commit --simulate
556 # icinga2 repository clear-changes
558 # icinga2 repository commit
560 After updating the configuration repository, make sure to reload Icinga 2.
562 # service icinga2 reload
565 # systemctl reload icinga2
568 #### <a id="icinga2-remote-monitoring-master-discovery-blacklist-whitelist"></a> Blacklist/Whitelist for Clients on the Master
570 It's sometimes necessary to `blacklist` an entire remote client, or specific hosts or services
571 provided by this client. While it's reasonable for the local admin to configure for example an
572 additional ping check, you're not interested in that on the master sending out notifications
573 and presenting the dashboard to your support team.
575 Blacklisting an entire set might not be sufficient for excluding several objects, be it a
576 specific remote client with one ping servie you're interested in. Therefore you can `whitelist`
577 clients, hosts, services in a similar manner
579 Example for blacklisting all `ping*` services, but allowing only `probe` host with `ping4`:
581 # icinga2 node blacklist add --zone "*" --host "*" --service "ping*"
582 # icinga2 node whitelist add --zone "*" --host "probe" --service "ping*"
584 You can `list` and `remove` existing blacklists:
586 # icinga2 node blacklist list
587 Listing all blacklist entries:
588 blacklist filter for Node: '*' Host: '*' Service: 'ping*'.
590 # icinga2 node whitelist list
591 Listing all whitelist entries:
592 whitelist filter for Node: '*' Host: 'probe' Service: 'ping*'.
597 > The `--zone` and `--host` arguments are required. A zone is always where the remote client is in.
598 > If you are unsure about it, set a wildcard (`*`) for them and filter only by host/services.
601 ### <a id="icinga2-remote-monitoring-master-manual-add-endpoint-zone"></a> Manually add Client Endpoint and Zone Objects on the Master
603 Define a [Zone](5-object-types.md#objecttype-zone) with a new [Endpoint](5-object-types.md#objecttype-endpoint) similar to the cluster setup.
605 * [configure the node name](8-monitoring-remote-systems.md#configure-nodename)
606 * [configure the ApiListener object](8-monitoring-remote-systems.md#configure-apilistener-object)
607 * [configure cluster endpoints](8-monitoring-remote-systems.md#configure-cluster-endpoints)
608 * [configure cluster zones](8-monitoring-remote-systems.md#configure-cluster-zones)
610 on a per remote client basis. If you prefer to synchronize the configuration to remote
611 clients, you can also use the cluster provided [configuration sync](8-monitoring-remote-systems.md#cluster-zone-config-sync)
615 ### <a id="agent-based-checks-addon"></a> Agent-based Checks using additional Software
617 If the remote services are not directly accessible through the network, a
618 local agent installation exposing the results to check queries can
621 ### <a id="agent-based-checks-snmp"></a> SNMP
623 The SNMP daemon runs on the remote system and answers SNMP queries by plugin
624 binaries. The [Monitoring Plugins package](2-getting-started.md#setting-up-check-plugins) ships
625 the `check_snmp` plugin binary, but there are plenty of [existing plugins](9-addons-plugins.md#plugins)
626 for specific use cases already around, for example monitoring Cisco routers.
628 The following example uses the [SNMP ITL](6-icinga-template-library.md#plugin-check-command-snmp) `CheckCommand` and just
629 overrides the `snmp_oid` custom attribute. A service is created for all hosts which
630 have the `snmp-community` custom attribute.
632 apply Service "uptime" {
633 import "generic-service"
635 check_command = "snmp"
636 vars.snmp_oid = "1.3.6.1.2.1.1.3.0"
638 assign where host.vars.snmp_community != ""
641 Additional SNMP plugins are available using the [Manubulon SNMP Plugins](6-icinga-template-library.md#snmp-manubulon-plugin-check-commands).
643 ### <a id="agent-based-checks-ssh"></a> SSH
645 Calling a plugin using the SSH protocol to execute a plugin on the remote server fetching
646 its return code and output. The `by_ssh` command object is part of the built-in templates and
647 requires the `check_by_ssh` check plugin which is available in the [Monitoring Plugins package](2-getting-started.md#setting-up-check-plugins).
649 object CheckCommand "by_ssh_swap" {
652 vars.by_ssh_command = "/usr/lib/nagios/plugins/check_swap -w $by_ssh_swap_warn$ -c $by_ssh_swap_crit$"
653 vars.by_ssh_swap_warn = "75%"
654 vars.by_ssh_swap_crit = "50%"
657 object Service "swap" {
658 import "generic-service"
660 host_name = "remote-ssh-host"
662 check_command = "by_ssh_swap"
664 vars.by_ssh_logname = "icinga"
667 ### <a id="agent-based-checks-nrpe"></a> NRPE
669 [NRPE](http://docs.icinga.org/latest/en/nrpe.html) runs as daemon on the remote client including
670 the required plugins and command definitions.
671 Icinga 2 calls the `check_nrpe` plugin binary in order to query the configured command on the
676 > The NRPE protocol is considered insecure and has multiple flaws in its
677 > design. Upstream is not willing to fix these issues.
679 > In order to stay safe, please use the native [Icinga 2 client](8-monitoring-remote-systems.md#icinga2-remote-monitoring-master)
682 The NRPE daemon uses its own configuration format in nrpe.cfg while `check_nrpe`
683 can be embedded into the Icinga 2 `CheckCommand` configuration syntax.
685 You can use the `check_nrpe` plugin from the NRPE project to query the NRPE daemon.
686 Icinga 2 provides the [nrpe check command](6-icinga-template-library.md#plugin-check-command-nrpe) for this:
690 object Service "users" {
691 import "generic-service"
693 host_name = "remote-nrpe-host"
695 check_command = "nrpe"
696 vars.nrpe_command = "check_users"
701 command[check_users]=/usr/local/icinga/libexec/check_users -w 5 -c 10
703 If you are planning to pass arguments to NRPE using the `-a`
704 command line parameter, make sure that your NRPE daemon has them
705 supported and enabled.
709 > Enabling command arguments in NRPE is considered harmful
710 > and exposes a security risk allowing attackers to execute
711 > commands remotely. Details at [seclists.org](http://seclists.org/fulldisclosure/2014/Apr/240).
713 The plugin check command `nrpe` provides the `nrpe_arguments` custom
714 attribute which expects either a single value or an array of values.
718 object Service "nrpe-disk-/" {
719 import "generic-service"
721 host_name = "remote-nrpe-host"
723 check_command = "nrpe"
724 vars.nrpe_command = "check_disk"
725 vars.nrpe_arguments = [ "20%", "10%", "/" ]
728 Icinga 2 will execute the nrpe plugin like this:
730 /usr/lib/nagios/plugins/check_nrpe -H <remote-nrpe-host> -c 'check_disk' -a '20%' '10%' '/'
732 NRPE expects all additional arguments in an ordered fashion
733 and interprets the first value as `$ARG1$` macro, the second
734 value as `$ARG2$`, and so on.
738 command[check_disk]=/usr/local/icinga/libexec/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
740 Using the above example with `nrpe_arguments` the command
741 executed by the NRPE daemon looks similar to that:
743 /usr/local/icinga/libexec/check_disk -w 20% -c 10% -p /
745 You can pass arguments in a similar manner to [NSClient++](8-monitoring-remote-systems.md#agent-based-checks-nsclient)
746 when using its NRPE supported check method.
748 ### <a id="agent-based-checks-nsclient"></a> NSClient++
750 [NSClient++](http://nsclient.org) works on both Windows and Linux platforms and is well
751 known for its magnificent Windows support. There are alternatives like the WMI interface,
752 but using `NSClient++` will allow you to run local scripts similar to check plugins fetching
753 the required output and performance counters.
755 You can use the `check_nt` plugin from the Monitoring Plugins project to query NSClient++.
756 Icinga 2 provides the [nscp check command](6-icinga-template-library.md#plugin-check-command-nscp) for this:
760 object Service "disk" {
761 import "generic-service"
763 host_name = "remote-windows-host"
765 check_command = "nscp"
767 vars.nscp_variable = "USEDDISKSPACE"
768 vars.nscp_params = "c"
773 For details on the `NSClient++` configuration please refer to the [official documentation](http://www.nsclient.org/nscp/wiki/doc/configuration/0.4.x).
775 ### <a id="agent-based-checks-nsca-ng"></a> NSCA-NG
777 [NSCA-ng](http://www.nsca-ng.org) provides a client-server pair that allows the
778 remote sender to push check results into the Icinga 2 `ExternalCommandListener`
783 > This addon works in a similar fashion like the Icinga 1.x distributed model. If you
784 > are looking for a real distributed architecture with Icinga 2, scroll down.
786 ### <a id="agent-based-checks-snmp-traps"></a> Passive Check Results and SNMP Traps
788 SNMP Traps can be received and filtered by using [SNMPTT](http://snmptt.sourceforge.net/)
789 and specific trap handlers passing the check results to Icinga 2.
791 Following the SNMPTT [Format](http://snmptt.sourceforge.net/docs/snmptt.shtml#SNMPTT.CONF-FORMAT)
792 documentation and the Icinga external command syntax found [here](17-appendix.md#external-commands-list-detail)
793 we can create generic services that can accommodate any number of hosts for a given scenario.
795 #### <a id="simple-traps"></a> Simple SNMP Traps
797 A simple example might be monitoring host reboots indicated by an SNMP agent reset.
798 Building the event to auto reset after dispatching a notification is important.
799 Setup the manual check parameters to reset the event from an initial unhandled
800 state or from a missed reset event.
802 Add a directive in `snmptt.conf`
804 EVENT coldStart .1.3.6.1.6.3.1.1.5.1 "Status Events" Normal
805 FORMAT Device reinitialized (coldStart)
806 EXEC echo "[$@] PROCESS_SERVICE_CHECK_RESULT;$A;Coldstart;2;The snmp agent has reinitialized." >> /var/run/icinga2/cmd/icinga2.cmd
808 A coldStart trap signifies that the SNMPv2 entity, acting
809 in an agent role, is reinitializing itself and that its
810 configuration may have been altered.
813 1. Define the `EVENT` as per your need.
814 2. Construct the `EXEC` statement with the service name matching your template
815 applied to your _n_ hosts. The host address inferred by SNMPTT will be the
816 correlating factor. You can have snmptt provide host names or ip addresses to
817 match your Icinga convention.
819 Add an `EventCommand` configuration object for the passive service auto reset event.
821 object EventCommand "coldstart-reset-event" {
822 import "plugin-event-command"
824 command = [ SysconfDir + "/icinga2/conf.d/custom/scripts/coldstart_reset_event.sh" ]
827 "-i" = "$service.state_id$"
829 "-s" = "$service.name$"
833 Create the `coldstart_reset_event.sh` shell script to pass the expanded variable
834 data in. The `$service.state_id$` is important in order to prevent an endless loop
835 of event firing after the service has been reset.
846 Usage: ${0##*/} [-h] -n HOST_NAME -s SERVICE_NAME
847 Writes a coldstart reset event to the Icinga command pipe.
849 -h Display this help and exit.
850 -i SERVICE_STATE_ID The associated service state id.
851 -n HOST_NAME The associated host name.
852 -s SERVICE_NAME The associated service name.
856 while getopts "hi:n:s:" opt; do
863 SERVICE_STATE_ID=$OPTARG
878 if [ -z "$SERVICE_STATE_ID" ]; then
880 printf "\n Error: -i required.\n"
884 if [ -z "$HOST_NAME" ]; then
886 printf "\n Error: -n required.\n"
890 if [ -z "$SERVICE_NAME" ]; then
892 printf "\n Error: -s required.\n"
896 if [ "$SERVICE_STATE_ID" -gt 0 ]; then
897 echo "[`date +%s`] PROCESS_SERVICE_CHECK_RESULT;$HOST_NAME;$SERVICE_NAME;0;Auto-reset (`date +"%m-%d-%Y %T"`)." >> /var/run/icinga2/cmd/icinga2.cmd
900 Finally create the `Service` and assign it:
902 apply Service "Coldstart" {
903 import "generic-service-custom"
905 check_command = "dummy"
906 event_command = "coldstart-reset-event"
908 enable_notifications = 1
909 enable_active_checks = 0
910 enable_passive_checks = 1
916 vars.dummy_text = "Manual reset."
920 assign where (host.vars.os == "Linux" || host.vars.os == "Windows")
923 #### <a id="complex-traps"></a> Complex SNMP Traps
925 A more complex example might be passing dynamic data from a traps varbind list
926 for a backup scenario where the backup software dispatches status updates. By
927 utilizing active and passive checks, the older freshness concept can be leveraged.
929 By defining the active check as a hard failed state, a missed backup can be reported.
930 As long as the most recent passive update has occurred, the active check is bypassed.
932 Add a directive in `snmptt.conf`
934 EVENT enterpriseSpecific <YOUR OID> "Status Events" Normal
935 FORMAT Enterprise specific trap
936 EXEC echo "[$@] PROCESS_SERVICE_CHECK_RESULT;$A;$1;$2;$3" >> /var/run/icinga2/cmd/icinga2.cmd
938 An enterprise specific trap.
939 The varbinds in order denote the Icinga service name, state and text.
942 1. Define the `EVENT` as per your need using your actual oid.
943 2. The service name, state and text are extracted from the first three varbinds.
944 This has the advantage of accommodating an unlimited set of use cases.
946 Create a `Service` for the specific use case associated to the host. If the host
947 matches and the first varbind value is `Backup`, SNMPTT will submit the corresponding
948 passive update with the state and text from the second and third varbind:
950 object Service "Backup" {
951 import "generic-service-custom"
953 host_name = "host.domain.com"
954 check_command = "dummy"
956 enable_notifications = 1
957 enable_active_checks = 1
958 enable_passive_checks = 1
961 max_check_attempts = 1
962 check_interval = 87000
967 vars.dummy_text = "No passive check result received."
971 ## <a id="distributed-monitoring-high-availability"></a> Distributed Monitoring and High Availability
973 Building distributed environments with high availability included is fairly easy with Icinga 2.
974 The cluster feature is built-in and allows you to build many scenarios based on your requirements:
976 * [High Availability](8-monitoring-remote-systems.md#cluster-scenarios-high-availability). All instances in the `Zone` elect one active master and run as Active/Active cluster.
977 * [Distributed Zones](8-monitoring-remote-systems.md#cluster-scenarios-distributed-zones). A master zone and one or more satellites in their zones.
978 * [Load Distribution](8-monitoring-remote-systems.md#cluster-scenarios-load-distribution). A configuration master and multiple checker satellites.
980 You can combine these scenarios into a global setup fitting your requirements.
982 Each instance got their own event scheduler, and does not depend on a centralized master
983 coordinating and distributing the events. In case of a cluster failure, all nodes
984 continue to run independently. Be alarmed when your cluster fails and a Split-Brain-scenario
985 is in effect - all alive instances continue to do their job, and history will begin to differ.
989 > Before you start, make sure to read the [requirements](#distributed-monitoring-requirements).
992 ### <a id="cluster-requirements"></a> Cluster Requirements
994 Before you start deploying, keep the following things in mind:
996 * Your [SSL CA and certificates](#certificate-authority-certificates) are mandatory for secure communication
997 * Get pen and paper or a drawing board and design your nodes and zones!
998 * all nodes in a cluster zone are providing high availability functionality and trust each other
999 * cluster zones can be built in a Top-Down-design where the child trusts the parent
1000 * communication between zones happens bi-directional which means that a DMZ-located node can still reach the master node, or vice versa
1001 * Update firewall rules and ACLs
1002 * Decide whether to use the built-in [configuration syncronization](8-monitoring-remote-systems.md#cluster-zone-config-sync) or use an external tool (Puppet, Ansible, Chef, Salt, etc) to manage the configuration deployment
1007 > If you're looking for troubleshooting cluster problems, check the general
1008 > [troubleshooting](12-troubleshooting.md#troubleshooting-cluster) section.
1011 ### <a id="manual-certificate-generation"></a> Manual SSL Certificate Generation
1013 Icinga 2 ships [CLI commands](7-cli-commands.md#cli-command-pki) assisting with CA and node certificate creation
1014 for your Icinga 2 distributed setup.
1018 > You're free to use your own method to generated a valid ca and signed client
1021 The first step is the creation of the certificate authority (CA) by running the
1024 # icinga2 pki new-ca
1026 Now create a certificate and key file for each node running the following command
1027 (replace `icinga2a` with the required hostname):
1029 # icinga2 pki new-cert --cn icinga2a --key icinga2a.key --csr icinga2a.csr
1030 # icinga2 pki sign-csr --csr icinga2a.csr --cert icinga2a.crt
1032 Repeat the step for all nodes in your cluster scenario.
1034 Save the CA key in a secure location in case you want to set up certificates for
1035 additional nodes at a later time.
1037 Navigate to the location of your newly generated certificate files, and manually
1038 copy/transfer them to `/etc/icinga2/pki` in your Icinga 2 configuration folder.
1042 > The certificate files must be readable by the user Icinga 2 is running as. Also,
1043 > the private key file must not be world-readable.
1045 Each node requires the following files in `/etc/icinga2/pki` (replace `fqdn-nodename` with
1049 * <fqdn-nodename>.crt
1050 * <fqdn-nodename>.key
1054 #### <a id="cluster-naming-convention"></a> Cluster Naming Convention
1056 The SSL certificate common name (CN) will be used by the [ApiListener](5-object-types.md#objecttype-apilistener)
1057 object to determine the local authority. This name must match the local [Endpoint](5-object-types.md#objecttype-endpoint)
1062 # icinga2 pki new-cert --cn icinga2a --key icinga2a.key --csr icinga2a.csr
1063 # icinga2 pki sign-csr --csr icinga2a.csr --cert icinga2a.crt
1067 object Endpoint "icinga2a" {
1068 host = "icinga2a.icinga.org"
1071 The [Endpoint](5-object-types.md#objecttype-endpoint) name is further referenced as `endpoints` attribute on the
1072 [Zone](5-object-types.md#objecttype-zone) object.
1074 object Endpoint "icinga2b" {
1075 host = "icinga2b.icinga.org"
1078 object Zone "config-ha-master" {
1079 endpoints = [ "icinga2a", "icinga2b" ]
1082 Specifying the local node name using the [NodeName](8-monitoring-remote-systems.md#configure-nodename) variable requires
1083 the same name as used for the endpoint name and common name above. If not set, the FQDN is used.
1085 const NodeName = "icinga2a"
1088 ### <a id="cluster-configuration"></a> Cluster Configuration
1090 The following section describe which configuration must be updated/created
1091 in order to get your cluster running with basic functionality.
1093 * [configure the node name](8-monitoring-remote-systems.md#configure-nodename)
1094 * [configure the ApiListener object](8-monitoring-remote-systems.md#configure-apilistener-object)
1095 * [configure cluster endpoints](8-monitoring-remote-systems.md#configure-cluster-endpoints)
1096 * [configure cluster zones](8-monitoring-remote-systems.md#configure-cluster-zones)
1098 Once you're finished with the basic setup the following section will
1099 describe how to use [zone configuration synchronisation](8-monitoring-remote-systems.md#cluster-zone-config-sync)
1100 and configure [cluster scenarios](8-monitoring-remote-systems.md#cluster-scenarios).
1102 #### <a id="configure-nodename"></a> Configure the Icinga Node Name
1104 Instead of using the default FQDN as node name you can optionally set
1105 that value using the [NodeName](15-language-reference.md#constants) constant.
1109 > Skip this step if your FQDN already matches the default `NodeName` set
1110 > in `/etc/icinga2/constants.conf`.
1112 This setting must be unique for each node, and must also match
1113 the name of the local [Endpoint](5-object-types.md#objecttype-endpoint) object and the
1114 SSL certificate common name as described in the
1115 [cluster naming convention](8-monitoring-remote-systems.md#cluster-naming-convention).
1117 vim /etc/icinga2/constants.conf
1119 /* Our local instance name. By default this is the server's hostname as returned by `hostname --fqdn`.
1120 * This should be the common name from the API certificate.
1122 const NodeName = "icinga2a"
1125 Read further about additional [naming conventions](8-monitoring-remote-systems.md#cluster-naming-convention).
1127 Not specifying the node name will make Icinga 2 using the FQDN. Make sure that all
1128 configured endpoint names and common names are in sync.
1130 #### <a id="configure-apilistener-object"></a> Configure the ApiListener Object
1132 The [ApiListener](5-object-types.md#objecttype-apilistener) object needs to be configured on
1133 every node in the cluster with the following settings:
1135 A sample config looks like:
1137 object ApiListener "api" {
1138 cert_path = SysconfDir + "/icinga2/pki/" + NodeName + ".crt"
1139 key_path = SysconfDir + "/icinga2/pki/" + NodeName + ".key"
1140 ca_path = SysconfDir + "/icinga2/pki/ca.crt"
1141 accept_config = true
1144 You can simply enable the `api` feature using
1146 # icinga2 feature enable api
1148 Edit `/etc/icinga2/features-enabled/api.conf` if you require the configuration
1149 synchronisation enabled for this node. Set the `accept_config` attribute to `true`.
1153 > The certificate files must be readable by the user Icinga 2 is running as. Also,
1154 > the private key file must not be world-readable.
1156 #### <a id="configure-cluster-endpoints"></a> Configure Cluster Endpoints
1158 `Endpoint` objects specify the `host` and `port` settings for the cluster nodes.
1159 This configuration can be the same on all nodes in the cluster only containing
1160 connection information.
1162 A sample configuration looks like:
1165 * Configure config master endpoint
1168 object Endpoint "icinga2a" {
1169 host = "icinga2a.icinga.org"
1172 If this endpoint object is reachable on a different port, you must configure the
1173 `ApiListener` on the local `Endpoint` object accordingly too.
1175 #### <a id="configure-cluster-zones"></a> Configure Cluster Zones
1177 `Zone` objects specify the endpoints located in a zone. That way your distributed setup can be
1178 seen as zones connected together instead of multiple instances in that specific zone.
1180 Zones can be used for [high availability](8-monitoring-remote-systems.md#cluster-scenarios-high-availability),
1181 [distributed setups](8-monitoring-remote-systems.md#cluster-scenarios-distributed-zones) and
1182 [load distribution](8-monitoring-remote-systems.md#cluster-scenarios-load-distribution).
1184 Each Icinga 2 `Endpoint` must be put into its respective `Zone`. In this example, you will
1185 define the zone `config-ha-master` where the `icinga2a` and `icinga2b` endpoints
1186 are located. The `check-satellite` zone consists of `icinga2c` only, but more nodes could
1189 The `config-ha-master` zone acts as High-Availability setup - the Icinga 2 instances elect
1190 one active master where all features are running on (for example `icinga2a`). In case of
1191 failure of the `icinga2a` instance, `icinga2b` will take over automatically.
1193 object Zone "config-ha-master" {
1194 endpoints = [ "icinga2a", "icinga2b" ]
1197 The `check-satellite` zone is a separated location and only sends back their checkresults to
1198 the defined parent zone `config-ha-master`.
1200 object Zone "check-satellite" {
1201 endpoints = [ "icinga2c" ]
1202 parent = "config-ha-master"
1206 ### <a id="cluster-zone-config-sync"></a> Zone Configuration Synchronisation
1208 By default all objects for specific zones should be organized in
1210 /etc/icinga2/zones.d/<zonename>
1212 on the configuration master.
1214 Your child zones and endpoint members **must not** have their config copied to `zones.d`.
1215 The built-in configuration synchronisation takes care of that if your nodes accept
1216 configuration from the parent zone. You can define that in the
1217 [ApiListener](8-monitoring-remote-systems.md#configure-apilistener-object) object by configuring the `accept_config`
1218 attribute accordingly.
1220 You should remove the sample config included in `conf.d` by commenting the `recursive_include`
1221 statement in [icinga2.conf](4-configuring-icinga-2.md#icinga2-conf):
1223 //include_recursive "conf.d"
1225 Better use a dedicated directory name like `cluster` or similar, and include that
1226 one if your nodes require local configuration not being synced to other nodes. That's
1227 useful for local [health checks](8-monitoring-remote-systems.md#cluster-health-check) for example.
1231 > In a [high availability](8-monitoring-remote-systems.md#cluster-scenarios-high-availability)
1232 > setup only one assigned node can act as configuration master. All other zone
1233 > member nodes **must not** have the `/etc/icinga2/zones.d` directory populated.
1235 These zone packages are then distributed to all nodes in the same zone, and
1236 to their respective target zone instances.
1238 Each configured zone must exist with the same directory name. The parent zone
1239 syncs the configuration to the child zones, if allowed using the `accept_config`
1240 attribute of the [ApiListener](8-monitoring-remote-systems.md#configure-apilistener-object) object.
1242 Config on node `icinga2a`:
1244 object Zone "master" {
1245 endpoints = [ "icinga2a" ]
1248 object Zone "checker" {
1249 endpoints = [ "icinga2b" ]
1253 /etc/icinga2/zones.d
1260 Config on node `icinga2b`:
1262 object Zone "master" {
1263 endpoints = [ "icinga2a" ]
1266 object Zone "checker" {
1267 endpoints = [ "icinga2b" ]
1271 /etc/icinga2/zones.d
1272 EMPTY_IF_CONFIG_SYNC_ENABLED
1274 If the local configuration is newer than the received update Icinga 2 will skip the synchronisation
1279 > `zones.d` must not be included in [icinga2.conf](4-configuring-icinga-2.md#icinga2-conf). Icinga 2 automatically
1280 > determines the required include directory. This can be overridden using the
1281 > [global constant](15-language-reference.md#constants) `ZonesDir`.
1283 #### <a id="zone-global-config-templates"></a> Global Configuration Zone for Templates
1285 If your zone configuration setup shares the same templates, groups, commands, timeperiods, etc.
1286 you would have to duplicate quite a lot of configuration objects making the merged configuration
1287 on your configuration master unique.
1291 > Only put templates, groups, etc into this zone. DO NOT add checkable objects such as
1292 > hosts or services here. If they are checked by all instances globally, this will lead
1293 > into duplicated check results and unclear state history. Not easy to troubleshoot too -
1294 > you've been warned.
1296 That is not necessary by defining a global zone shipping all those templates. By setting
1297 `global = true` you ensure that this zone serving common configuration templates will be
1298 synchronized to all involved nodes (only if they accept configuration though).
1300 Config on configuration master:
1302 /etc/icinga2/zones.d
1312 In this example, the global zone is called `global-templates` and must be defined in
1313 your zone configuration visible to all nodes.
1315 object Zone "global-templates" {
1321 > If the remote node does not have this zone configured, it will ignore the configuration
1322 > update, if it accepts synchronized configuration.
1324 If you don't require any global configuration, skip this setting.
1326 #### <a id="zone-config-sync-permissions"></a> Zone Configuration Synchronisation Permissions
1328 Each [ApiListener](5-object-types.md#objecttype-apilistener) object must have the `accept_config` attribute
1329 set to `true` to receive configuration from the parent `Zone` members. Default value is `false`.
1331 object ApiListener "api" {
1332 cert_path = SysconfDir + "/icinga2/pki/" + NodeName + ".crt"
1333 key_path = SysconfDir + "/icinga2/pki/" + NodeName + ".key"
1334 ca_path = SysconfDir + "/icinga2/pki/ca.crt"
1335 accept_config = true
1338 If `accept_config` is set to `false`, this instance won't accept configuration from remote
1339 master instances anymore.
1343 > Look into the [troubleshooting guides](12-troubleshooting.md#troubleshooting-cluster-config-sync) for debugging
1344 > problems with the configuration synchronisation.
1347 ### <a id="cluster-health-check"></a> Cluster Health Check
1349 The Icinga 2 [ITL](6-icinga-template-library.md#icinga-template-library) ships an internal check command checking all configured
1350 `EndPoints` in the cluster setup. The check result will become critical if
1351 one or more configured nodes are not connected.
1355 object Service "cluster" {
1356 check_command = "cluster"
1360 host_name = "icinga2a"
1363 Each cluster node should execute its own local cluster health check to
1364 get an idea about network related connection problems from different
1367 Additionally you can monitor the connection from the local zone to the remote
1370 Example for the `checker` zone checking the connection to the `master` zone:
1372 object Service "cluster-zone-master" {
1373 check_command = "cluster-zone"
1376 vars.cluster_zone = "master"
1378 host_name = "icinga2b"
1382 ### <a id="cluster-scenarios"></a> Cluster Scenarios
1384 All cluster nodes are full-featured Icinga 2 instances. You only need to enabled
1385 the features for their role (for example, a `Checker` node only requires the `checker`
1386 feature enabled, but not `notification` or `ido-mysql` features).
1388 #### <a id="cluster-scenarios-security"></a> Security in Cluster Scenarios
1390 While there are certain capabilities to ensure the safe communication between all
1391 nodes (firewalls, policies, software hardening, etc) the Icinga 2 cluster also provides
1392 additional security itself:
1394 * [SSL certificates](#certificate-authority-certificates) are mandatory for cluster communication.
1395 * Child zones only receive event updates (check results, commands, etc) for their configured updates.
1396 * Zones cannot influence/interfere other zones. Each checked object is assigned to only one zone.
1397 * All nodes in a zone trust each other.
1398 * [Configuration sync](8-monitoring-remote-systems.md#zone-config-sync-permissions) is disabled by default.
1400 #### <a id="cluster-scenarios-features"></a> Features in Cluster Zones
1402 Each cluster zone may use all available features. If you have multiple locations
1403 or departments, they may write to their local database, or populate graphite.
1404 Even further all commands are distributed amongst connected nodes. For example, you could
1405 re-schedule a check or acknowledge a problem on the master, and it gets replicated to the
1406 actual slave checker node.
1408 DB IDO on the left, graphite on the right side - works (if you disable
1409 [DB IDO HA](8-monitoring-remote-systems.md#high-availability-db-ido)).
1410 Icinga Web 2 on the left, checker and notifications on the right side - works too.
1411 Everything on the left and on the right side - make sure to deal with
1412 [load-balanced notifications and checks](8-monitoring-remote-systems.md#high-availability-features) in a
1413 [HA zone](8-monitoring-remote-systems.md#cluster-scenarios-high-availability).
1414 configure-cluster-zones
1415 #### <a id="cluster-scenarios-distributed-zones"></a> Distributed Zones
1417 That scenario fits if your instances are spread over the globe and they all report
1418 to a master instance. Their network connection only works towards the master master
1419 (or the master is able to connect, depending on firewall policies) which means
1420 remote instances won't see each/connect to each other.
1422 All events (check results, downtimes, comments, etc) are synced to the master node,
1423 but the remote nodes can still run local features such as a web interface, reporting,
1424 graphing, etc. in their own specified zone.
1426 Imagine the following example with a master node in Nuremberg, and two remote DMZ
1427 based instances in Berlin and Vienna. Additonally you'll specify
1428 [global templates](8-monitoring-remote-systems.md#zone-global-config-templates) available in all zones.
1430 The configuration tree on the master instance `nuremberg` could look like this:
1443 The configuration deployment will take care of automatically synchronising
1444 the child zone configuration:
1446 * The master node sends `zones.d/berlin` to the `berlin` child zone.
1447 * The master node sends `zones.d/vienna` to the `vienna` child zone.
1448 * The master node sends `zones.d/global-templates` to the `vienna` and `berlin` child zones.
1450 The endpoint configuration would look like:
1452 object Endpoint "nuremberg-master" {
1453 host = "nuremberg.icinga.org"
1456 object Endpoint "berlin-satellite" {
1457 host = "berlin.icinga.org"
1460 object Endpoint "vienna-satellite" {
1461 host = "vienna.icinga.org"
1464 The zones would look like:
1466 object Zone "nuremberg" {
1467 endpoints = [ "nuremberg-master" ]
1470 object Zone "berlin" {
1471 endpoints = [ "berlin-satellite" ]
1472 parent = "nuremberg"
1475 object Zone "vienna" {
1476 endpoints = [ "vienna-satellite" ]
1477 parent = "nuremberg"
1480 object Zone "global-templates" {
1484 The `nuremberg-master` zone will only execute local checks, and receive
1485 check results from the satellite nodes in the zones `berlin` and `vienna`.
1489 > The child zones `berlin` and `vienna` will get their configuration synchronised
1490 > from the configuration master 'nuremberg'. The endpoints in the child
1491 > zones **must not** have their `zones.d` directory populated if this endpoint
1492 > [accepts synced configuration](8-monitoring-remote-systems.md#zone-config-sync-permissions).
1494 #### <a id="cluster-scenarios-load-distribution"></a> Load Distribution
1496 If you are planning to off-load the checks to a defined set of remote workers
1497 you can achieve that by:
1499 * Deploying the configuration on all nodes.
1500 * Let Icinga 2 distribute the load amongst all available nodes.
1502 That way all remote check instances will receive the same configuration
1503 but only execute their part. The master instance located in the `master` zone
1504 can also execute checks, but you may also disable the `Checker` feature.
1506 Configuration on the master node:
1513 If you are planning to have some checks executed by a specific set of checker nodes
1514 you have to define additional zones and define these check objects there.
1518 object Endpoint "master-node" {
1519 host = "master.icinga.org"
1522 object Endpoint "checker1-node" {
1523 host = "checker1.icinga.org"
1526 object Endpoint "checker2-node" {
1527 host = "checker2.icinga.org"
1533 object Zone "master" {
1534 endpoints = [ "master-node" ]
1537 object Zone "checker" {
1538 endpoints = [ "checker1-node", "checker2-node" ]
1542 object Zone "global-templates" {
1548 > The child zones `checker` will get its configuration synchronised
1549 > from the configuration master 'master'. The endpoints in the child
1550 > zone **must not** have their `zones.d` directory populated if this endpoint
1551 > [accepts synced configuration](8-monitoring-remote-systems.md#zone-config-sync-permissions).
1553 #### <a id="cluster-scenarios-high-availability"></a> Cluster High Availability
1555 High availability with Icinga 2 is possible by putting multiple nodes into
1556 a dedicated [zone](8-monitoring-remote-systems.md#configure-cluster-zones). All nodes will elect one
1557 active master, and retry an election once the current active master is down.
1559 Selected features provide advanced [HA functionality](8-monitoring-remote-systems.md#high-availability-features).
1560 Checks and notifications are load-balanced between nodes in the high availability
1563 Connections from other zones will be accepted by all active and passive nodes
1564 but all are forwarded to the current active master dealing with the check results,
1567 object Zone "config-ha-master" {
1568 endpoints = [ "icinga2a", "icinga2b", "icinga2c" ]
1571 Two or more nodes in a high availability setup require an [initial cluster sync](8-monitoring-remote-systems.md#initial-cluster-sync).
1575 > Keep in mind that **only one node acts as configuration master** having the
1576 > configuration files in the `zones.d` directory. All other nodes **must not**
1577 > have that directory populated. Instead they are required to
1578 > [accept synced configuration](8-monitoring-remote-systems.md#zone-config-sync-permissions).
1579 > Details in the [Configuration Sync Chapter](8-monitoring-remote-systems.md#cluster-zone-config-sync).
1581 #### <a id="cluster-scenarios-multiple-hierarchies"></a> Multiple Hierarchies
1583 Your master zone collects all check results for reporting and graphing and also
1584 does some sort of additional notifications.
1585 The customers got their own instances in their local DMZ zones. They are limited to read/write
1586 only their services, but replicate all events back to the master instance.
1587 Within each DMZ there are additional check instances also serving interfaces for local
1588 departments. The customers instances will collect all results, but also send them back to
1589 your master instance.
1590 Additionally the customers instance on the second level in the middle prohibits you from
1591 sending commands to the subjacent department nodes. You're only allowed to receive the
1592 results, and a subset of each customers configuration too.
1594 Your master zone will generate global reports, aggregate alert notifications, and check
1595 additional dependencies (for example, the customers internet uplink and bandwidth usage).
1597 The customers zone instances will only check a subset of local services and delegate the rest
1598 to each department. Even though it acts as configuration master with a master dashboard
1599 for all departments managing their configuration tree which is then deployed to all
1600 department instances. Furthermore the master NOC is able to see what's going on.
1602 The instances in the departments will serve a local interface, and allow the administrators
1603 to reschedule checks or acknowledge problems for their services.
1606 ### <a id="high-availability-features"></a> High Availability for Icinga 2 features
1608 All nodes in the same zone require the same features enabled for High Availability (HA)
1611 By default the following features provide advanced HA functionality:
1613 * [Checks](8-monitoring-remote-systems.md#high-availability-checks) (load balanced, automated failover)
1614 * [Notifications](8-monitoring-remote-systems.md#high-availability-notifications) (load balanced, automated failover)
1615 * [DB IDO](8-monitoring-remote-systems.md#high-availability-db-ido) (Run-Once, automated failover)
1617 #### <a id="high-availability-checks"></a> High Availability with Checks
1619 All nodes in the same zone load-balance the check execution. When one instance
1620 fails the other nodes will automatically take over the reamining checks.
1624 > If a node should not check anything, disable the `checker` feature explicitely and
1627 # icinga2 feature disable checker
1628 # service icinga2 reload
1630 #### <a id="high-availability-notifications"></a> High Availability with Notifications
1632 Notifications are load balanced amongst all nodes in a zone. By default this functionality
1634 If your nodes should notify independent from any other nodes (this will cause
1635 duplicated notifications if not properly handled!), you can set `enable_ha = false`
1636 in the [NotificationComponent](5-object-types.md#objecttype-notificationcomponent) feature.
1638 #### <a id="high-availability-db-ido"></a> High Availability with DB IDO
1640 All instances within the same zone (e.g. the `master` zone as HA cluster) must
1641 have the DB IDO feature enabled.
1643 Example DB IDO MySQL:
1645 # icinga2 feature enable ido-mysql
1646 The feature 'ido-mysql' is already enabled.
1648 By default the DB IDO feature only runs on the elected zone master. All other passive
1649 nodes disable the active IDO database connection at runtime.
1653 > The DB IDO HA feature can be disabled by setting the `enable_ha` attribute to `false`
1654 > for the [IdoMysqlConnection](5-object-types.md#objecttype-idomysqlconnection) or
1655 > [IdoPgsqlConnection](5-object-types.md#objecttype-idopgsqlconnection) object on all nodes in the
1658 > All endpoints will enable the DB IDO feature then, connect to the configured
1659 > database and dump configuration, status and historical data on their own.
1661 If the instance with the active DB IDO connection dies, the HA functionality will
1662 re-enable the DB IDO connection on the newly elected zone master.
1664 The DB IDO feature will try to determine which cluster endpoint is currently writing
1665 to the database and bail out if another endpoint is active. You can manually verify that
1666 by running the following query:
1668 icinga=> SELECT status_update_time, endpoint_name FROM icinga_programstatus;
1669 status_update_time | endpoint_name
1670 ------------------------+---------------
1671 2014-08-15 15:52:26+02 | icinga2a
1674 This is useful when the cluster connection between endpoints breaks, and prevents
1675 data duplication in split-brain-scenarios. The failover timeout can be set for the
1676 `failover_timeout` attribute, but not lower than 60 seconds.
1679 ### <a id="cluster-add-node"></a> Add a new cluster endpoint
1681 These steps are required for integrating a new cluster endpoint:
1683 * generate a new [SSL client certificate](#certificate-authority-certificates)
1684 * identify its location in the zones
1685 * update the `zones.conf` file on each involved node ([endpoint](8-monitoring-remote-systems.md#configure-cluster-endpoints), [zones](8-monitoring-remote-systems.md#configure-cluster-zones))
1686 * a new slave zone node requires updates for the master and slave zones
1687 * verify if this endpoints requires [configuration synchronisation](8-monitoring-remote-systems.md#cluster-zone-config-sync) enabled
1688 * if the node requires the existing zone history: [initial cluster sync](8-monitoring-remote-systems.md#initial-cluster-sync)
1689 * add a [cluster health check](8-monitoring-remote-systems.md#cluster-health-check)
1691 #### <a id="initial-cluster-sync"></a> Initial Cluster Sync
1693 In order to make sure that all of your cluster nodes have the same state you will
1694 have to pick one of the nodes as your initial "master" and copy its state file
1695 to all the other nodes.
1697 You can find the state file in `/var/lib/icinga2/icinga2.state`. Before copying
1698 the state file you should make sure that all your cluster nodes are properly shut
1702 ### <a id="host-multiple-cluster-nodes"></a> Host With Multiple Cluster Nodes
1704 Special scenarios might require multiple cluster nodes running on a single host.
1705 By default Icinga 2 and its features will place their runtime data below the prefix
1706 `LocalStateDir`. By default packages will set that path to `/var`.
1707 You can either set that variable as constant configuration
1708 definition in [icinga2.conf](4-configuring-icinga-2.md#icinga2-conf) or pass it as runtime variable to
1709 the Icinga 2 daemon.
1711 # icinga2 -c /etc/icinga2/node1/icinga2.conf -DLocalStateDir=/opt/node1/var