1 # <a id="monitoring-remote-systems"></a> Monitoring Remote Systems
3 There are multiple ways you can monitor remote clients. Be it using [agent-less](#agent-less-checks)
4 or [agent-based](agent-based-checks-addons) using additional addons & tools.
6 Icinga 2 uses its own unique and secure communitication protol amongst instances.
7 Be it an High-Availability cluster setup, distributed load-balanced setup or just a single
8 agent [monitoring a remote client](#icinga2-remote-client-monitoring).
10 All communication is secured by SSL x509, and fully supports IPv4 and IPv6.
12 If you are planning to use the native Icinga 2 cluster feature for distributed
13 monitoring and high-availability, please continue reading in
14 [this chapter](#distributed-monitoring-high-availability).
18 > Don't panic - there are CLI commands available, including setup wizards for easy installation
19 > with SSL certificates.
20 > If you prefer to use your own CA (for example Puppet) you can do that as well.
22 ## <a id="agent-less-checks"></a> Agent-less Checks
24 If the remote service is available using a network protocol and port,
25 and a [check plugin](#setting-up-check-plugins) is available, you don't
26 necessarily need a local client installed. Rather choose a plugin and
27 configure all parameters and thresholds. The [Icinga 2 Template Library](#itl)
28 already ships various examples like
30 * [ping4](#plugin-check-command-ping4), [ping6](#plugin-check-command-ping6),
31 [fping4](#plugin-check-command-fping4), [fping6](#plugin-check-command-fping6), [hostalive](#plugin-check-command-hostalive)
32 * [tcp](#plugin-check-command-tcp), [udp](#plugin-check-command-udp), [ssl](#plugin-check-command-ssl)
33 * [http](#plugin-check-command-http), [ftp](#plugin-check-command-ftp)
34 * [smtp](#plugin-check-command-smtp), [ssmtp](#plugin-check-command-ssmtp),
35 [imap](#plugin-check-command-imap), [simap](#plugin-check-command-simap),
36 [pop](#plugin-check-command-pop), [spop](#plugin-check-command-spop)
37 * [ntp_time](#plugin-check-command-ntp_time)
38 * [ssh](#plugin-check-command-ssh)
39 * [dns](#plugin-check-command-dns), [dig](#plugin-check-command-dig), [dhcp](#plugin-check-command-dhcp)
41 There are numerous check plugins contributed by community members available
42 on the internet. If you found one for your requirements, [integrate them into Icinga 2](#command-plugin-integration).
46 * [Icinga Exchange](https://exchange.icinga.org)
47 * [Icinga Wiki](https://wiki.icinga.org)
49 An example is provided in the sample configuration in the getting started
50 section shipped with Icinga 2 ([hosts.conf](#hosts-conf), [services.conf](#services-conf)).
52 ## <a id="icinga2-remote-client-monitoring"></a> Monitoring Icinga 2 Remote Clients
54 First, you should decide which role the remote client has:
56 * a single host with local checks and configuration
57 * a remote satellite checking other hosts (for example in your DMZ)
58 * a remote command execution client (similar to NRPE, NSClient++, etc)
60 Later on, you will be asked again and told how to proceed with these
61 different [roles](#icinga2-remote-monitoring-client-roles).
65 > If you are planning to build an Icinga 2 distributed setup using the cluster feature, please skip
66 > the following instructions and jump directly to the
67 > [cluster setup instructions](#distributed-monitoring-high-availability).
71 > Remote instances are independent Icinga 2 instances which schedule
72 > their checks and just synchronize them back to the defined master zone.
74 ## <a id="icinga2-remote-monitoring-master"></a> Master Setup for Remote Monitoring
76 If you are planning to use the [remote Icinga 2 clients](#icinga2-remote-monitoring-client)
77 you'll first need to update your master setup.
79 Your master setup requires the following
81 * SSL CA and signed certificate for the master
82 * Enabled API feature, and a local Endpoint and Zone object configuration
83 * Firewall ACLs for the communication port (default 5665)
85 You can use the [cli command](#cli-command-node) `node wizard` for setting up a new node
86 on the master. The command must be run as root, all Icinga 2 specific files
87 will be updated to the icinga user the daemon is running as (certificate files
90 Make sure to answer the first question with `n` (no).
94 Welcome to the Icinga 2 Setup Wizard!
96 We'll guide you through all required configuration details.
98 If you have questions, please consult the documentation at http://docs.icinga.org
99 or join the community support channels at https://support.icinga.org
102 Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: n
103 Starting the Master setup routine...
104 Please specifiy the common name (CN) [icinga2m]:
105 information/base: Writing private key to '/var/lib/icinga2/ca/ca.key'.
106 information/base: Writing X509 certificate to '/var/lib/icinga2/ca/ca.crt'.
107 information/cli: Initializing serial file in '/var/lib/icinga2/ca/serial.txt'.
108 information/cli: Generating new CSR in '/etc/icinga2/pki/icinga2m.csr'.
109 information/base: Writing private key to '/etc/icinga2/pki/icinga2m.key'.
110 information/base: Writing certificate signing request to '/etc/icinga2/pki/icinga2m.csr'.
111 information/cli: Signing CSR with CA and writing certificate to '/etc/icinga2/pki/icinga2m.crt'.
112 information/cli: Copying CA certificate to '/etc/icinga2/pki/ca.crt'.
113 information/cli: Dumping config items to file '/etc/icinga2/zones.conf'.
114 Please specify the API bind host/port (optional):
117 information/cli: Enabling the APIlistener feature.
118 information/cli: Updating constants.conf.
119 information/cli: Updating constants file '/etc/icinga2/constants.conf'.
120 information/cli: Updating constants file '/etc/icinga2/constants.conf'.
121 information/cli: Edit the constants.conf file '/etc/icinga2/constants.conf' and set a secure 'TicketSalt' constant.
124 Now restart your Icinga 2 daemon to finish the installation!
126 If you encounter problems or bugs, please do not hesitate to
127 get in touch with the community at https://support.icinga.org
130 The setup wizard will do the following:
132 * Generate a local CA in `/var/lib/icinga2/ca` or use the existing one
133 * Generate a new CSR, sign it with the local CA and copying it into `/etc/icinga2/pki`
134 * Generate a local zone and endpoint configuration for this master based on FQDN
135 * Enabling the API feature, and setting optional `bind_host` and `bind_port`
136 * Setting the `NodeName` and `TicketSalt` constants in [constants.conf](#constants.conf)
138 The setup wizard does not automatically restart Icinga 2.
143 > This setup wizard will install a standalone master, HA cluster scenarios are currently
148 ## <a id="icinga2-remote-monitoring-client"></a> Client Setup for Remote Monitoring
150 Icinga 2 can be installed on Linux/Unix and Windows. While
151 [Linux/Unix](#icinga2-remote-monitoring-client-linux) will be using the [cli command](#cli-command-node)
152 `node wizard` for a guided setup, you will need to use the
153 graphical installer for Windows based client setup.
155 Your client setup requires the following
157 * A ready configured and installed [master node](#icinga2-remote-monitoring-master)
158 * SSL signed certificate for communication with the master (Use [CSR auto-signing](certifiates-csr-autosigning)).
159 * Enabled API feature, and a local Endpoint and Zone object configuration
160 * Firewall ACLs for the communication port (default 5665)
164 ### <a id="icinga2-remote-monitoring-client-linux"></a> Linux Client Setup for Remote Monitoring
166 #### <a id="csr-autosigning-requirements"></a> Requirements for CSR Auto-Signing
168 If your remote clients are capable of connecting to the central master, Icinga 2
169 supports CSR auto-signing.
171 First you'll need to define a secure ticket salt in the [constants.conf](#constants-conf).
172 The [setup wizard for the master setup](#icinga2-remote-monitoring-master) will create
175 # grep TicketSalt /etc/icinga2/constants.conf
177 The client setup wizard will ask you to generate a valid ticket number using its CN.
178 If you already know your remote client's Common Names (CNs) - usually the FQDN - you
179 can generate all ticket numbers on-demand.
181 This is also reasonable if you are not capable of installing the remote client, but
182 a colleague of yours, or a customer.
184 Example for a client notebook:
186 # icinga2 pki ticket --cn nbmif.int.netways.de
190 > You can omit the `--salt` parameter using the `TicketSalt` constant from
191 > [constants.conf](#constants-conf) if already defined and Icinga 2 was
192 > reloaded after the master setup.
194 #### <a id="certificates-manual-creation"></a> Manual SSL Certificate Generation
196 This is described separately in the [cluster setup chapter](#manual-certificate-generation).
200 > If you're using [CSR Auto-Signing](#csr-autosigning-requirements), skip this step.
203 #### <a id="icinga2-remote-monitoring-client-linux-setup"></a> Linux Client Setup Wizard for Remote Monitoring
205 Install Icinga 2 from your distribution's package repository as described in the
206 general [installation instructions](#setting-up-icinga2).
208 Please make sure that either [CSR Auto-Signing](#csr-autosigning-requirements) requirements
209 are fulfilled, or that you're using [manual SSL certificate generation](#manual-certificate-generation).
213 > You don't need any features (DB IDO, Livestatus) or user interfaces on the remote client.
214 > Install them only if you're planning to use them.
216 Once the package installation succeeded, use the `node wizard` cli command to install
217 a new Icinga 2 node as client setup.
219 You'll need the following configuration details:
221 * The client common name (CN). Defaults to FQDN.
222 * The client's local zone name. Defaults to FQDN.
223 * The master endpoint name. Look into your master setup `zones.conf` file for the proper name.
224 * The master endpoint connection information. Your master's IP address and port (defaults to 5665)
225 * The [request ticket number](#csr-autosigning-requirements) generated on your master
227 * Bind host/port for the Api feature (optional)
229 The command must be run as root, all Icinga 2 specific files will be updated to the icinga
230 user the daemon is running as (certificate files for example).
233 # icinga2 node wizard
235 Welcome to the Icinga 2 Setup Wizard!
237 We'll guide you through all required configuration details.
239 If you have questions, please consult the documentation at http://docs.icinga.org
240 or join the community support channels at https://support.icinga.org
243 Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]:
244 Starting the Node setup routine...
245 Please specifiy the common name (CN) [nbmif.int.netways.de]:
246 Please specifiy the local zone name [nbmif.int.netways.de]:
247 Please specify the master endpoint(s) this node should connect to:
248 Master Common Name (CN from your master setup, defaults to FQDN): icinga2m
249 Please fill out the master connection information:
250 Master endpoint host (required, your master's IP address or FQDN): 192.168.33.100
251 Master endpoint port (optional) []:
252 Add more master endpoints? [y/N]
253 Please specify the master connection for CSR auto-signing (defaults to master endpoint host):
254 Host [192.168.33.100]:
256 information/base: Writing private key to '/var/lib/icinga2/ca/ca.key'.
257 information/base: Writing X509 certificate to '/var/lib/icinga2/ca/ca.crt'.
258 information/cli: Initializing serial file in '/var/lib/icinga2/ca/serial.txt'.
259 information/base: Writing private key to '/etc/icinga2/pki/nbmif.int.netways.de.key'.
260 information/base: Writing X509 certificate to '/etc/icinga2/pki/nbmif.int.netways.de.crt'.
261 information/cli: Generating self-signed certifiate:
262 information/cli: Fetching public certificate from master (192.168.33.100, 5665):
264 information/cli: Writing trusted certificate to file '/etc/icinga2/pki/trusted-master.crt'.
265 information/cli: Stored trusted master certificate in '/etc/icinga2/pki/trusted-master.crt'.
267 Please specify the request ticket generated on your Icinga 2 master.
268 (Hint: '# icinga2 pki ticket --cn nbmif.int.netways.de'):
269 2e070405fe28f311a455b53a61614afd718596a1
270 information/cli: Processing self-signed certificate request. Ticket '2e070405fe28f311a455b53a61614afd718596a1'.
272 information/cli: Writing signed certificate to file '/etc/icinga2/pki/nbmif.int.netways.de.crt'.
273 information/cli: Writing CA certificate to file '/var/lib/icinga2/ca/ca.crt'.
274 Please specify the API bind host/port (optional):
277 information/cli: Disabling the Notification feature.
278 Disabling feature notification. Make sure to restart Icinga 2 for these changes to take effect.
279 information/cli: Enabling the Apilistener feature.
280 information/cli: Generating local zones.conf.
281 information/cli: Dumping config items to file '/etc/icinga2/zones.conf'.
282 information/cli: Updating constants.conf.
283 information/cli: Updating constants file '/etc/icinga2/constants.conf'.
286 Now restart your Icinga 2 daemon to finish the installation!
288 If you encounter problems or bugs, please do not hesitate to
289 get in touch with the community at https://support.icinga.org
292 The setup wizard will do the following:
294 * Generate a new self-signed certificate and copy it into `/etc/icinga2/pki`
295 * Store the master's certificate as trusted certificate for requesting a new signed certificate
296 (manual step when using `node setup`).
297 * Request a new signed certificate from the master and store updated certificate and master CA in `/etc/icinga2/pki`
298 * Generate a local zone and endpoint configuration for this client and the provided master information
300 * Disabling the notification feature for this client
301 * Enabling the API feature, and setting optional `bind_host` and `bind_port`
302 * Setting the `NodeName` constant in [constants.conf](#constants.conf)
304 The setup wizard does not automatically restart Icinga 2.
306 If you are getting an error when requesting the ticket number, please check the following:
308 * Is the CN the same (from pki ticket on the master and setup node on the client)
309 * Is the ticket expired
312 ### <a id="icinga2-remote-monitoring-client-windows"></a> Windows Client Setup for Remote Monitoring
314 Download the MSI-Installer package from [http://packages.icinga.org/windows/](http://packages.icinga.org/windows/).
317 * [Microsoft .NET Framework 2.0](http://www.microsoft.com/de-de/download/details.aspx?id=1639) if not already installed.
319 The setup wizard will install Icinga 2 and then continue with SSL certificate generation,
320 CSR-Autosigning and configuration setup.
322 You'll need the following configuration details:
324 * The client common name (CN). Defaults to FQDN.
325 * The client's local zone name. Defaults to FQDN.
326 * The master endpoint name. Look into your master setup `zones.conf` file for the proper name.
327 * The master endpoint connection information. Your master's IP address and port (defaults to 5665)
328 * The [request ticket number](#csr-autosigning-requirements) generated on your master
330 * Bind host/port for the Api feature (optional)
332 Once install is done, Icinga 2 is automatically started as a Windows service.
334 ### <a id="icinga2-remote-monitoring-client-roles"></a> Remote Monitoring Client Roles
336 Icinga 2 allows you to use two separate ways of defining a client (or: `agent`) role:
338 * execute commands remotely, but host/service configuration happens on the master.
339 * schedule remote checks on remote satellites with their local configuration.
341 Depending on your scenario, either one or both combined with a cluster setup
342 could be build and put together.
345 ### <a id="icinga2-remote-monitoring-client-command-execution"></a> Remote Client for Command Execution
347 This scenario allows you to configure the checkable objects (hosts, services) on
348 your Icinga 2 master or satellite, and only send commands remotely.
351 * Exact same [CheckCommand](#objecttype-checkcommand) (and
352 [EventCommand](#objecttype-eventcommand)) configuration objects
353 on the master and the remote client(s).
354 * Installed plugin scripts on the remote client (`PluginDir` constant can be locally modified)
355 * `Zone` and `Endpoint` configuration for the client on the master
356 * `command_endpoint` attribute configured for host/service objects pointing to the configured
359 `CheckCommand` objects are already shipped with the Icinga 2 ITL
360 as [plugin check commands](#plugin-check-commands). If you are
361 using your own configuration definitions for example in
362 [commands.conf](#commands-conf) make sure to copy/sync it
363 on your remote client.
365 #### <a id="icinga2-remote-monitoring-client-command-execution-client"></a> Client Configuration Remote Client for Command Execution
369 > Remote clients must explicitely accept commands in a similar
370 > fashion as cluster nodes [accept configuration]#i(cluster-zone-config-sync).
371 > This is due to security reasons.
373 Edit the `api` feature configuration in `/etc/icinga2/features-enabled/api.conf`
374 and set `accept_commands` to `true`.
376 object ApiListener "api" {
377 cert_path = SysconfDir + "/icinga2/pki/" + NodeName + ".crt"
378 key_path = SysconfDir + "/icinga2/pki/" + NodeName + ".key"
379 ca_path = SysconfDir + "/icinga2/pki/ca.crt"
380 accept_commands = true
383 #### <a id="icinga2-remote-monitoring-client-command-execution-master"></a> Master Configuration Remote Client for Command Execution
385 Add an `Endpoint` and `Zone` configuration object for the remote client
386 in [zones.conf](#zones-conf) and define a trusted master zone as `parent`.
388 object Endpoint "remote-client1" {
389 host = "192.168.33.20"
392 object Zone "remote-client1" {
393 endpoints = [ "remote-client1" ]
398 * [configure endpoints](#configure-cluster-endpoints)
399 * [configure zones](#configure-cluster-zones)
402 Configuration example for host and service objects running commands on the remote endpoint `remote-client1`:
404 object Host "host-remote" {
405 import "generic-host"
407 address = "127.0.0.1"
412 vars.remote_client = "remote-client1"
414 /* host specific check arguments */
415 vars.users_wgreater = 10
416 vars.users_wgreater = 20
419 apply Service "users-remote" {
420 import "generic-service"
422 check_command = "users"
423 command_endpoint = host.vars.remote_client
425 /* override (remote) command arguments with host settings */
426 vars.users_wgreater = host.vars.users_wgreater
427 vars.users_cgreater = host.vars.users_cgreater
429 /* assign where a remote client is set */
430 assign where host.vars.remote_client
434 That way you can also execute the `icinga` check remotely
435 thus verifying the health of your remote client(s). As a bonus
436 you'll also get the running Icinga 2 version and may
437 schedule client updates in your management tool (e.g. Puppet).
441 > [Event commands](#event-commands) are executed on the
442 > remote command endpoint as well. You do not need
443 > an additional transport layer such as SSH or similar.
446 > You cannot add any Icinga 2 features like DB IDO on the remote
447 > clients. There are no local configured objects available.
449 > If you require this, please install a full-featured
450 > [local client](#icinga2-remote-monitoring-client-local-config).
452 ### <a id="icinga2-remote-monitoring-client-local-config"></a> Remote Client with Local Configuration
454 This is considered as independant satellite using a local scheduler, configuration
455 and the possibility to add Icinga 2 features on demand.
457 Local configured checks are transferred to the central master and helped
458 with discovery cli commands.
460 Please follow the instructions closely in order to deploy your fully featured
461 client, or `agent` as others might call it.
463 #### <a id="icinga2-remote-monitoring-client-configuration"></a> Client Configuration for Remote Monitoring
465 There is no difference in the configuration syntax on clients to any other Icinga 2 installation.
467 The following convention applies to remote clients:
469 * The hostname in the default host object should be the same as the Common Name (CN) used for SSL setup
470 * Add new services and check commands locally
472 The default setup routine will install a new host based on your FQDN in `repository.d/hosts` with all
473 services in separate configuration files a directory underneath.
475 The repository can be managed using the cli command `repository`.
479 > The cli command `repository` only supports basic configuration manipulation (add, remove). Future
480 > versions will support more options (set, etc.). Please check the Icinga 2 development roadmap
483 You can also use additional features like notifications directly on the remote client, if you are
484 required to. Basically everything a single Icinga 2 instance provides by default.
487 #### <a id="icinga2-remote-monitoring-master-discovery"></a> Discover Client Services on the Master
489 Icinga 2 clients will sync their locally defined objects to the defined master node. That way you can
490 list, add, filter and remove nodes based on their `node`, `zone`, `host` or `service` name.
492 List all discovered nodes (satellites, agents) and their hosts/services:
497 #### <a id="icinga2-remote-monitoring-master-discovery-manual"></a> Manually Discover Clients on the Master
499 Add a to-be-discovered client to the master:
501 # icinga2 node add my-remote-client
503 Set the connection details, and the Icinga 2 master will attempt to connect to this node and sync its
506 # icinga2 node set my-remote-client --host 192.168.33.101 --port 5665
508 You can control that by calling the `node list` command:
511 Node 'my-remote-client' (host: 192.168.33.101, port: 5665, log duration: 1 day, last seen: Sun Nov 2 17:46:29 2014)
513 #### <a id="icinga2-remote-monitoring-master-discovery-remove"></a> Remove Discovered Clients
515 If you don't require a connected agent, you can manually remove it and its discovered hosts and services
516 using the following cli command:
518 # icinga2 node remove my-discovered-agent
522 > Better use [blacklists and/or whitelists](#icinga2-remote-monitoring-master-discovery-blacklist-whitelist)
523 > to control which clients and hosts/services are integrated into your master configuration repository.
525 ### <a id="icinga2-remote-monitoring-master-discovery-generate-config"></a> Generate Icinga 2 Configuration for Client Services on the Master
527 There is a dedicated Icinga 2 CLI command for updating the client services on the master,
528 generating all required configuration.
530 # icinga2 node update-config
532 The generated configuration of all nodes is stored in the `repository.d/` directory.
534 By default, the following additional configuration is generated:
535 * add `Endpoint` and `Zone` objects for the newly added node
536 * add `cluster-zone` health check for the master host detecting if the remote node died
537 * use the default templates `satellite-host` and `satellite-service` defined in `/etc/icinga2/conf.d/satellite.conf`
538 * apply a dependency for all other hosts on the remote satellite prevening failure checks/notifications
542 > If there are existing hosts/services defined or modified, the cli command will not overwrite these (modified)
543 > configuration files.
545 > If hosts or services disappeared from the client discovery, it will remove the existing configuration objects
546 > from the config repository.
548 The `update-config` cli command will fail, if there are uncommitted changes for the
549 configuration repository.
550 Please review these changes manually, or clear the commit and try again. This is a
551 safety hook to prevent unwanted manual changes to be committed by a updating the
552 client discovered objects only.
554 # icinga2 repository commit --simulate
556 # icinga2 repository clear-changes
558 # icinga2 repository commit
560 After updating the configuration repository, make sure to reload Icinga 2.
562 # service icinga2 reload
565 # systemctl reload icinga2.service
569 #### <a id="icinga2-remote-monitoring-master-discovery-blacklist-whitelist"></a> Blacklist/Whitelist for Clients on the Master
571 It's sometimes necessary to `blacklist` an entire remote client, or specific hosts or services
572 provided by this client. While it's reasonable for the local admin to configure for example an
573 additional ping check, you're not interested in that on the master sending out notifications
574 and presenting the dashboard to your support team.
576 Blacklisting an entire set might not be sufficient for excluding several objects, be it a
577 specific remote client with one ping servie you're interested in. Therefore you can `whitelist`
578 clients, hosts, services in a similar manner
580 Example for blacklisting all `ping*` services, but allowing only `probe` host with `ping4`:
582 # icinga2 node blacklist add --zone "*" --host "*" --service "ping*"
583 # icinga2 node whitelist add --zone "*" --host "probe" --service "ping*"
585 You can `list` and `remove` existing blacklists:
587 # icinga2 node blacklist list
588 Listing all blacklist entries:
589 blacklist filter for Node: '*' Host: '*' Service: 'ping*'.
591 # icinga2 node whitelist list
592 Listing all whitelist entries:
593 whitelist filter for Node: '*' Host: 'probe' Service: 'ping*'.
598 > The `--zone` and `--host` arguments are required. A zone is always where the remote client is in.
599 > If you are unsure about it, set a wildcard (`*`) for them and filter only by host/services.
602 ### <a id="icinga2-remote-monitoring-master-manual-add-endpoint-zone"></a> Manually add Client Endpoint and Zone Objects on the Master
604 Define a [Zone](#objecttype-zone) with a new [Endpoint](#objecttype-endpoint) similar to the cluster setup.
606 * [configure the node name](#configure-nodename)
607 * [configure the ApiListener object](#configure-apilistener-object)
608 * [configure cluster endpoints](#configure-cluster-endpoints)
609 * [configure cluster zones](#configure-cluster-zones)
611 on a per remote client basis. If you prefer to synchronize the configuration to remote
612 clients, you can also use the cluster provided [configuration sync](#cluster-zone-config-sync)
616 ### <a id="agent-based-checks-addon"></a> Agent-based Checks using additional Software
618 If the remote services are not directly accessible through the network, a
619 local agent installation exposing the results to check queries can
622 ### <a id="agent-based-checks-snmp"></a> SNMP
624 The SNMP daemon runs on the remote system and answers SNMP queries by plugin
625 binaries. The [Monitoring Plugins package](#setting-up-check-plugins) ships
626 the `check_snmp` plugin binary, but there are plenty of [existing plugins](#integrate-additional-plugins)
627 for specific use cases already around, for example monitoring Cisco routers.
629 The following example uses the [SNMP ITL](#plugin-check-command-snmp) `CheckCommand` and just
630 overrides the `snmp_oid` custom attribute. A service is created for all hosts which
631 have the `snmp-community` custom attribute.
633 apply Service "uptime" {
634 import "generic-service"
636 check_command = "snmp"
637 vars.snmp_oid = "1.3.6.1.2.1.1.3.0"
639 assign where host.vars.snmp_community != ""
642 Additional SNMP plugins are available using the [Manubulon SNMP Plugins](#snmp-manubulon-plugin-check-commands).
644 ### <a id="agent-based-checks-ssh"></a> SSH
646 Calling a plugin using the SSH protocol to execute a plugin on the remote server fetching
647 its return code and output. The `by_ssh` command object is part of the built-in templates and
648 requires the `check_by_ssh` check plugin which is available in the [Monitoring Plugins package](#setting-up-check-plugins).
650 object CheckCommand "by_ssh_swap" {
653 vars.by_ssh_command = "/usr/lib/nagios/plugins/check_swap -w $by_ssh_swap_warn$ -c $by_ssh_swap_crit$"
654 vars.by_ssh_swap_warn = "75%"
655 vars.by_ssh_swap_crit = "50%"
658 object Service "swap" {
659 import "generic-service"
661 host_name = "remote-ssh-host"
663 check_command = "by_ssh_swap"
665 vars.by_ssh_logname = "icinga"
668 ### <a id="agent-based-checks-nrpe"></a> NRPE
670 [NRPE](http://docs.icinga.org/latest/en/nrpe.html) runs as daemon on the remote client including
671 the required plugins and command definitions.
672 Icinga 2 calls the `check_nrpe` plugin binary in order to query the configured command on the
677 > The NRPE protocol is considered insecure and has multiple flaws in its
678 > design. Upstream is not willing to fix these issues.
680 > In order to stay safe, please use the native [Icinga 2 client](#icinga2-remote-monitoring-master)
683 The NRPE daemon uses its own configuration format in nrpe.cfg while `check_nrpe`
684 can be embedded into the Icinga 2 `CheckCommand` configuration syntax.
686 You can use the `check_nrpe` plugin from the NRPE project to query the NRPE daemon.
687 Icinga 2 provides the [nrpe check command](#plugin-check-command-nrpe) for this:
691 object Service "users" {
692 import "generic-service"
694 host_name = "remote-nrpe-host"
696 check_command = "nrpe"
697 vars.nrpe_command = "check_users"
702 command[check_users]=/usr/local/icinga/libexec/check_users -w 5 -c 10
704 If you are planning to pass arguments to NRPE using the `-a`
705 command line parameter, make sure that your NRPE daemon has them
706 supported and enabled.
710 > Enabling command arguments in NRPE is considered harmful
711 > and exposes a security risk allowing attackers to execute
712 > commands remotely. Details at [seclists.org](http://seclists.org/fulldisclosure/2014/Apr/240).
714 The plugin check command `nrpe` provides the `nrpe_arguments` custom
715 attribute which expects either a single value or an array of values.
719 object Service "nrpe-disk-/" {
720 import "generic-service"
722 host_name = "remote-nrpe-host"
724 check_command = "nrpe"
725 vars.nrpe_command = "check_disk"
726 vars.nrpe_arguments = [ "20%", "10%", "/" ]
729 Icinga 2 will execute the nrpe plugin like this:
731 /usr/lib/nagios/plugins/check_nrpe -H <remote-nrpe-host> -c 'check_disk' -a '20%' '10%' '/'
733 NRPE expects all additional arguments in an ordered fashion
734 and interprets the first value as `$ARG1$` macro, the second
735 value as `$ARG2$`, and so on.
739 command[check_disk]=/usr/local/icinga/libexec/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
741 Using the above example with `nrpe_arguments` the command
742 executed by the NRPE daemon looks similar to that:
744 /usr/local/icinga/libexec/check_disk -w 20% -c 10% -p /
746 You can pass arguments in a similar manner to [NSClient++](#agent-based-checks-nsclient)
747 when using its NRPE supported check method.
749 ### <a id="agent-based-checks-nsclient"></a> NSClient++
751 [NSClient++](http://nsclient.org) works on both Windows and Linux platforms and is well
752 known for its magnificent Windows support. There are alternatives like the WMI interface,
753 but using `NSClient++` will allow you to run local scripts similar to check plugins fetching
754 the required output and performance counters.
756 You can use the `check_nt` plugin from the Monitoring Plugins project to query NSClient++.
757 Icinga 2 provides the [nscp check command](#plugin-check-command-nscp) for this:
761 object Service "disk" {
762 import "generic-service"
764 host_name = "remote-windows-host"
766 check_command = "nscp"
768 vars.nscp_variable = "USEDDISKSPACE"
769 vars.nscp_params = "c"
774 For details on the `NSClient++` configuration please refer to the [official documentation](http://www.nsclient.org/nscp/wiki/doc/configuration/0.4.x).
776 ### <a id="agent-based-checks-nsca-ng"></a> NSCA-NG
778 [NSCA-ng](http://www.nsca-ng.org) provides a client-server pair that allows the
779 remote sender to push check results into the Icinga 2 `ExternalCommandListener`
784 > This addon works in a similar fashion like the Icinga 1.x distributed model. If you
785 > are looking for a real distributed architecture with Icinga 2, scroll down.
787 ### <a id="agent-based-checks-snmp-traps"></a> Passive Check Results and SNMP Traps
789 SNMP Traps can be received and filtered by using [SNMPTT](http://snmptt.sourceforge.net/) and specific trap handlers
790 passing the check results to Icinga 2.
794 > The host and service object configuration must be available on the Icinga 2
795 > server in order to process passive check results.
800 ## <a id="distributed-monitoring-high-availability"></a> Distributed Monitoring and High Availability
802 Building distributed environments with high availability included is fairly easy with Icinga 2.
803 The cluster feature is built-in and allows you to build many scenarios based on your requirements:
805 * [High Availability](#cluster-scenarios-high-availability). All instances in the `Zone` elect one active master and run as Active/Active cluster.
806 * [Distributed Zones](#cluster-scenarios-distributed-zones). A master zone and one or more satellites in their zones.
807 * [Load Distribution](#cluster-scenarios-load-distribution). A configuration master and multiple checker satellites.
809 You can combine these scenarios into a global setup fitting your requirements.
811 Each instance got their own event scheduler, and does not depend on a centralized master
812 coordinating and distributing the events. In case of a cluster failure, all nodes
813 continue to run independently. Be alarmed when your cluster fails and a Split-Brain-scenario
814 is in effect - all alive instances continue to do their job, and history will begin to differ.
818 > Before you start, make sure to read the [requirements](#distributed-monitoring-requirements).
821 ### <a id="cluster-requirements"></a> Cluster Requirements
823 Before you start deploying, keep the following things in mind:
825 * Your [SSL CA and certificates](#certificate-authority-certificates) are mandatory for secure communication
826 * Get pen and paper or a drawing board and design your nodes and zones!
827 * all nodes in a cluster zone are providing high availability functionality and trust each other
828 * cluster zones can be built in a Top-Down-design where the child trusts the parent
829 * communication between zones happens bi-directional which means that a DMZ-located node can still reach the master node, or vice versa
830 * Update firewall rules and ACLs
831 * Decide whether to use the built-in [configuration syncronization](#cluster-zone-config-sync) or use an external tool (Puppet, Ansible, Chef, Salt, etc) to manage the configuration deployment
836 > If you're looking for troubleshooting cluster problems, check the general
837 > [troubleshooting](#troubleshooting-cluster) section.
840 ### <a id="manual-certificate-generation"></a> Manual SSL Certificate Generation
842 Icinga 2 ships [cli commands](#cli-command-pki) assisting with CA and node certificate creation
843 for your Icinga 2 distributed setup.
847 > You're free to use your own method to generated a valid ca and signed client
850 The first step is the creation of the certificate authority (CA) by running the
855 Now create a certificate and key file for each node running the following command
856 (replace `icinga2a` with the required hostname):
858 # icinga2 pki new-cert --cn icinga2a --key icinga2a.key --csr icinga2a.csr
859 # icinga2 pki sign-csr --csr icinga2a.csr --cert icinga2a.crt
861 Repeat the step for all nodes in your cluster scenario.
863 Save the CA key in a secure location in case you want to set up certificates for
864 additional nodes at a later time.
866 Navigate to the location of your newly generated certificate files, and manually
867 copy/transfer them to `/etc/icinga2/pki` in your Icinga 2 configuration folder.
871 > The certificate files must be readable by the user Icinga 2 is running as. Also,
872 > the private key file must not be world-readable.
874 Each node requires the following files in `/etc/icinga2/pki` (replace `fqdn-nodename` with
878 * <fqdn-nodename>.crt
879 * <fqdn-nodename>.key
883 #### <a id="cluster-naming-convention"></a> Cluster Naming Convention
885 The SSL certificate common name (CN) will be used by the [ApiListener](#objecttype-apilistener)
886 object to determine the local authority. This name must match the local [Endpoint](#objecttype-endpoint)
891 # icinga2 pki new-cert --cn icinga2a --key icinga2a.key --csr icinga2a.csr
892 # icinga2 pki sign-csr --csr icinga2a.csr --cert icinga2a.crt
896 object Endpoint "icinga2a" {
897 host = "icinga2a.icinga.org"
900 The [Endpoint](#objecttype-endpoint) name is further referenced as `endpoints` attribute on the
901 [Zone](#objecttype-zone) object.
903 object Endpoint "icinga2b" {
904 host = "icinga2b.icinga.org"
907 object Zone "config-ha-master" {
908 endpoints = [ "icinga2a", "icinga2b" ]
911 Specifying the local node name using the [NodeName](#configure-nodename) variable requires
912 the same name as used for the endpoint name and common name above. If not set, the FQDN is used.
914 const NodeName = "icinga2a"
917 ### <a id="cluster-configuration"></a> Cluster Configuration
919 The following section describe which configuration must be updated/created
920 in order to get your cluster running with basic functionality.
922 * [configure the node name](#configure-nodename)
923 * [configure the ApiListener object](#configure-apilistener-object)
924 * [configure cluster endpoints](#configure-cluster-endpoints)
925 * [configure cluster zones](#configure-cluster-zones)
927 Once you're finished with the basic setup the following section will
928 describe how to use [zone configuration synchronisation](#cluster-zone-config-sync)
929 and configure [cluster scenarios](#cluster-scenarios).
931 #### <a id="configure-nodename"></a> Configure the Icinga Node Name
933 Instead of using the default FQDN as node name you can optionally set
934 that value using the [NodeName](#global-constants) constant.
938 > Skip this step if your FQDN already matches the default `NodeName` set
939 > in `/etc/icinga2/constants.conf`.
941 This setting must be unique for each node, and must also match
942 the name of the local [Endpoint](#objecttype-endpoint) object and the
943 SSL certificate common name as described in the
944 [cluster naming convention](#cluster-naming-convention).
946 vim /etc/icinga2/constants.conf
948 /* Our local instance name. By default this is the server's hostname as returned by `hostname --fqdn`.
949 * This should be the common name from the API certificate.
951 const NodeName = "icinga2a"
954 Read further about additional [naming conventions](#cluster-naming-convention).
956 Not specifying the node name will make Icinga 2 using the FQDN. Make sure that all
957 configured endpoint names and common names are in sync.
959 #### <a id="configure-apilistener-object"></a> Configure the ApiListener Object
961 The [ApiListener](#objecttype-apilistener) object needs to be configured on
962 every node in the cluster with the following settings:
964 A sample config looks like:
966 object ApiListener "api" {
967 cert_path = SysconfDir + "/icinga2/pki/" + NodeName + ".crt"
968 key_path = SysconfDir + "/icinga2/pki/" + NodeName + ".key"
969 ca_path = SysconfDir + "/icinga2/pki/ca.crt"
973 You can simply enable the `api` feature using
975 # icinga2 feature enable api
977 Edit `/etc/icinga2/features-enabled/api.conf` if you require the configuration
978 synchronisation enabled for this node. Set the `accept_config` attribute to `true`.
982 > The certificate files must be readable by the user Icinga 2 is running as. Also,
983 > the private key file must not be world-readable.
985 #### <a id="configure-cluster-endpoints"></a> Configure Cluster Endpoints
987 `Endpoint` objects specify the `host` and `port` settings for the cluster nodes.
988 This configuration can be the same on all nodes in the cluster only containing
989 connection information.
991 A sample configuration looks like:
994 * Configure config master endpoint
997 object Endpoint "icinga2a" {
998 host = "icinga2a.icinga.org"
1001 If this endpoint object is reachable on a different port, you must configure the
1002 `ApiListener` on the local `Endpoint` object accordingly too.
1004 #### <a id="configure-cluster-zones"></a> Configure Cluster Zones
1006 `Zone` objects specify the endpoints located in a zone. That way your distributed setup can be
1007 seen as zones connected together instead of multiple instances in that specific zone.
1009 Zones can be used for [high availability](#cluster-scenarios-high-availability),
1010 [distributed setups](#cluster-scenarios-distributed-zones) and
1011 [load distribution](#cluster-scenarios-load-distribution).
1013 Each Icinga 2 `Endpoint` must be put into its respective `Zone`. In this example, you will
1014 define the zone `config-ha-master` where the `icinga2a` and `icinga2b` endpoints
1015 are located. The `check-satellite` zone consists of `icinga2c` only, but more nodes could
1018 The `config-ha-master` zone acts as High-Availability setup - the Icinga 2 instances elect
1019 one active master where all features are running on (for example `icinga2a`). In case of
1020 failure of the `icinga2a` instance, `icinga2b` will take over automatically.
1022 object Zone "config-ha-master" {
1023 endpoints = [ "icinga2a", "icinga2b" ]
1026 The `check-satellite` zone is a separated location and only sends back their checkresults to
1027 the defined parent zone `config-ha-master`.
1029 object Zone "check-satellite" {
1030 endpoints = [ "icinga2c" ]
1031 parent = "config-ha-master"
1035 ### <a id="cluster-zone-config-sync"></a> Zone Configuration Synchronisation
1037 By default all objects for specific zones should be organized in
1039 /etc/icinga2/zones.d/<zonename>
1041 on the configuration master.
1043 Your child zones and endpoint members **must not** have their config copied to `zones.d`.
1044 The built-in configuration synchronisation takes care of that if your nodes accept
1045 configuration from the parent zone. You can define that in the
1046 [ApiListener](#configure-apilistener-object) object by configuring the `accept_config`
1047 attribute accordingly.
1049 You should remove the sample config included in `conf.d` by commenting the `recursive_include`
1050 statement in [icinga2.conf](#icinga2-conf):
1052 //include_recursive "conf.d"
1054 Better use a dedicated directory name like `cluster` or similar, and include that
1055 one if your nodes require local configuration not being synced to other nodes. That's
1056 useful for local [health checks](#cluster-health-check) for example.
1060 > In a [high availability](#cluster-scenarios-high-availability)
1061 > setup only one assigned node can act as configuration master. All other zone
1062 > member nodes **must not** have the `/etc/icinga2/zones.d` directory populated.
1064 These zone packages are then distributed to all nodes in the same zone, and
1065 to their respective target zone instances.
1067 Each configured zone must exist with the same directory name. The parent zone
1068 syncs the configuration to the child zones, if allowed using the `accept_config`
1069 attribute of the [ApiListener](#configure-apilistener-object) object.
1071 Config on node `icinga2a`:
1073 object Zone "master" {
1074 endpoints = [ "icinga2a" ]
1077 object Zone "checker" {
1078 endpoints = [ "icinga2b" ]
1082 /etc/icinga2/zones.d
1089 Config on node `icinga2b`:
1091 object Zone "master" {
1092 endpoints = [ "icinga2a" ]
1095 object Zone "checker" {
1096 endpoints = [ "icinga2b" ]
1100 /etc/icinga2/zones.d
1101 EMPTY_IF_CONFIG_SYNC_ENABLED
1103 If the local configuration is newer than the received update Icinga 2 will skip the synchronisation
1108 > `zones.d` must not be included in [icinga2.conf](#icinga2-conf). Icinga 2 automatically
1109 > determines the required include directory. This can be overridden using the
1110 > [global constant](#global-constants) `ZonesDir`.
1112 #### <a id="zone-global-config-templates"></a> Global Configuration Zone for Templates
1114 If your zone configuration setup shares the same templates, groups, commands, timeperiods, etc.
1115 you would have to duplicate quite a lot of configuration objects making the merged configuration
1116 on your configuration master unique.
1120 > Only put templates, groups, etc into this zone. DO NOT add checkable objects such as
1121 > hosts or services here. If they are checked by all instances globally, this will lead
1122 > into duplicated check results and unclear state history. Not easy to troubleshoot too -
1123 > you've been warned.
1125 That is not necessary by defining a global zone shipping all those templates. By setting
1126 `global = true` you ensure that this zone serving common configuration templates will be
1127 synchronized to all involved nodes (only if they accept configuration though).
1129 Config on configuration master:
1131 /etc/icinga2/zones.d
1141 In this example, the global zone is called `global-templates` and must be defined in
1142 your zone configuration visible to all nodes.
1144 object Zone "global-templates" {
1150 > If the remote node does not have this zone configured, it will ignore the configuration
1151 > update, if it accepts synchronized configuration.
1153 If you don't require any global configuration, skip this setting.
1155 #### <a id="zone-config-sync-permissions"></a> Zone Configuration Synchronisation Permissions
1157 Each [ApiListener](#objecttype-apilistener) object must have the `accept_config` attribute
1158 set to `true` to receive configuration from the parent `Zone` members. Default value is `false`.
1160 object ApiListener "api" {
1161 cert_path = SysconfDir + "/icinga2/pki/" + NodeName + ".crt"
1162 key_path = SysconfDir + "/icinga2/pki/" + NodeName + ".key"
1163 ca_path = SysconfDir + "/icinga2/pki/ca.crt"
1164 accept_config = true
1167 If `accept_config` is set to `false`, this instance won't accept configuration from remote
1168 master instances anymore.
1172 > Look into the [troubleshooting guides](#troubleshooting-cluster-config-sync) for debugging
1173 > problems with the configuration synchronisation.
1176 ### <a id="cluster-health-check"></a> Cluster Health Check
1178 The Icinga 2 [ITL](#itl) ships an internal check command checking all configured
1179 `EndPoints` in the cluster setup. The check result will become critical if
1180 one or more configured nodes are not connected.
1184 object Service "cluster" {
1185 check_command = "cluster"
1189 host_name = "icinga2a"
1192 Each cluster node should execute its own local cluster health check to
1193 get an idea about network related connection problems from different
1196 Additionally you can monitor the connection from the local zone to the remote
1199 Example for the `checker` zone checking the connection to the `master` zone:
1201 object Service "cluster-zone-master" {
1202 check_command = "cluster-zone"
1205 vars.cluster_zone = "master"
1207 host_name = "icinga2b"
1211 ### <a id="cluster-scenarios"></a> Cluster Scenarios
1213 All cluster nodes are full-featured Icinga 2 instances. You only need to enabled
1214 the features for their role (for example, a `Checker` node only requires the `checker`
1215 feature enabled, but not `notification` or `ido-mysql` features).
1217 #### <a id="cluster-scenarios-security"></a> Security in Cluster Scenarios
1219 While there are certain capabilities to ensure the safe communication between all
1220 nodes (firewalls, policies, software hardening, etc) the Icinga 2 cluster also provides
1221 additional security itself:
1223 * [SSL certificates](#certificate-authority-certificates) are mandatory for cluster communication.
1224 * Child zones only receive event updates (check results, commands, etc) for their configured updates.
1225 * Zones cannot influence/interfere other zones. Each checked object is assigned to only one zone.
1226 * All nodes in a zone trust each other.
1227 * [Configuration sync](#zone-config-sync-permissions) is disabled by default.
1229 #### <a id="cluster-scenarios-features"></a> Features in Cluster Zones
1231 Each cluster zone may use all available features. If you have multiple locations
1232 or departments, they may write to their local database, or populate graphite.
1233 Even further all commands are distributed amongst connected nodes. For example, you could
1234 re-schedule a check or acknowledge a problem on the master, and it gets replicated to the
1235 actual slave checker node.
1237 DB IDO on the left, graphite on the right side - works (if you disable
1238 [DB IDO HA](#high-availability-db-ido)).
1239 Icinga Web 2 on the left, checker and notifications on the right side - works too.
1240 Everything on the left and on the right side - make sure to deal with
1241 [load-balanced notifications and checks](#high-availability-features) in a
1242 [HA zone](#cluster-scenarios-high-availability).
1243 configure-cluster-zones
1244 #### <a id="cluster-scenarios-distributed-zones"></a> Distributed Zones
1246 That scenario fits if your instances are spread over the globe and they all report
1247 to a master instance. Their network connection only works towards the master master
1248 (or the master is able to connect, depending on firewall policies) which means
1249 remote instances won't see each/connect to each other.
1251 All events (check results, downtimes, comments, etc) are synced to the master node,
1252 but the remote nodes can still run local features such as a web interface, reporting,
1253 graphing, etc. in their own specified zone.
1255 Imagine the following example with a master node in Nuremberg, and two remote DMZ
1256 based instances in Berlin and Vienna. Additonally you'll specify
1257 [global templates](#zone-global-config-templates) available in all zones.
1259 The configuration tree on the master instance `nuremberg` could look like this:
1272 The configuration deployment will take care of automatically synchronising
1273 the child zone configuration:
1275 * The master node sends `zones.d/berlin` to the `berlin` child zone.
1276 * The master node sends `zones.d/vienna` to the `vienna` child zone.
1277 * The master node sends `zones.d/global-templates` to the `vienna` and `berlin` child zones.
1279 The endpoint configuration would look like:
1281 object Endpoint "nuremberg-master" {
1282 host = "nuremberg.icinga.org"
1285 object Endpoint "berlin-satellite" {
1286 host = "berlin.icinga.org"
1289 object Endpoint "vienna-satellite" {
1290 host = "vienna.icinga.org"
1293 The zones would look like:
1295 object Zone "nuremberg" {
1296 endpoints = [ "nuremberg-master" ]
1299 object Zone "berlin" {
1300 endpoints = [ "berlin-satellite" ]
1301 parent = "nuremberg"
1304 object Zone "vienna" {
1305 endpoints = [ "vienna-satellite" ]
1306 parent = "nuremberg"
1309 object Zone "global-templates" {
1313 The `nuremberg-master` zone will only execute local checks, and receive
1314 check results from the satellite nodes in the zones `berlin` and `vienna`.
1318 > The child zones `berlin` and `vienna` will get their configuration synchronised
1319 > from the configuration master 'nuremberg'. The endpoints in the child
1320 > zones **must not** have their `zones.d` directory populated if this endpoint
1321 > [accepts synced configuration](#zone-config-sync-permissions).
1323 #### <a id="cluster-scenarios-load-distribution"></a> Load Distribution
1325 If you are planning to off-load the checks to a defined set of remote workers
1326 you can achieve that by:
1328 * Deploying the configuration on all nodes.
1329 * Let Icinga 2 distribute the load amongst all available nodes.
1331 That way all remote check instances will receive the same configuration
1332 but only execute their part. The master instance located in the `master` zone
1333 can also execute checks, but you may also disable the `Checker` feature.
1335 Configuration on the master node:
1342 If you are planning to have some checks executed by a specific set of checker nodes
1343 you have to define additional zones and define these check objects there.
1347 object Endpoint "master-node" {
1348 host = "master.icinga.org"
1351 object Endpoint "checker1-node" {
1352 host = "checker1.icinga.org"
1355 object Endpoint "checker2-node" {
1356 host = "checker2.icinga.org"
1362 object Zone "master" {
1363 endpoints = [ "master-node" ]
1366 object Zone "checker" {
1367 endpoints = [ "checker1-node", "checker2-node" ]
1371 object Zone "global-templates" {
1377 > The child zones `checker` will get its configuration synchronised
1378 > from the configuration master 'master'. The endpoints in the child
1379 > zone **must not** have their `zones.d` directory populated if this endpoint
1380 > [accepts synced configuration](#zone-config-sync-permissions).
1382 #### <a id="cluster-scenarios-high-availability"></a> Cluster High Availability
1384 High availability with Icinga 2 is possible by putting multiple nodes into
1385 a dedicated [zone](#configure-cluster-zones). All nodes will elect one
1386 active master, and retry an election once the current active master is down.
1388 Selected features provide advanced [HA functionality](#high-availability-features).
1389 Checks and notifications are load-balanced between nodes in the high availability
1392 Connections from other zones will be accepted by all active and passive nodes
1393 but all are forwarded to the current active master dealing with the check results,
1396 object Zone "config-ha-master" {
1397 endpoints = [ "icinga2a", "icinga2b", "icinga2c" ]
1400 Two or more nodes in a high availability setup require an [initial cluster sync](#initial-cluster-sync).
1404 > Keep in mind that **only one node acts as configuration master** having the
1405 > configuration files in the `zones.d` directory. All other nodes **must not**
1406 > have that directory populated. Instead they are required to
1407 > [accept synced configuration](#zone-config-sync-permissions).
1408 > Details in the [Configuration Sync Chapter](#cluster-zone-config-sync).
1410 #### <a id="cluster-scenarios-multiple-hierachies"></a> Multiple Hierachies
1412 Your master zone collects all check results for reporting and graphing and also
1413 does some sort of additional notifications.
1414 The customers got their own instances in their local DMZ zones. They are limited to read/write
1415 only their services, but replicate all events back to the master instance.
1416 Within each DMZ there are additional check instances also serving interfaces for local
1417 departments. The customers instances will collect all results, but also send them back to
1418 your master instance.
1419 Additionally the customers instance on the second level in the middle prohibits you from
1420 sending commands to the subjacent department nodes. You're only allowed to receive the
1421 results, and a subset of each customers configuration too.
1423 Your master zone will generate global reports, aggregate alert notifications, and check
1424 additional dependencies (for example, the customers internet uplink and bandwidth usage).
1426 The customers zone instances will only check a subset of local services and delegate the rest
1427 to each department. Even though it acts as configuration master with a master dashboard
1428 for all departments managing their configuration tree which is then deployed to all
1429 department instances. Furthermore the master NOC is able to see what's going on.
1431 The instances in the departments will serve a local interface, and allow the administrators
1432 to reschedule checks or acknowledge problems for their services.
1435 ### <a id="high-availability-features"></a> High Availability for Icinga 2 features
1437 All nodes in the same zone require the same features enabled for High Availability (HA)
1440 By default the following features provide advanced HA functionality:
1442 * [Checks](#high-availability-checks) (load balanced, automated failover)
1443 * [Notifications](#high-availability-notifications) (load balanced, automated failover)
1444 * [DB IDO](#high-availability-db-ido) (Run-Once, automated failover)
1446 #### <a id="high-availability-checks"></a> High Availability with Checks
1448 All nodes in the same zone load-balance the check execution. When one instance
1449 fails the other nodes will automatically take over the reamining checks.
1453 > If a node should not check anything, disable the `checker` feature explicitely and
1456 # icinga2 feature disable checker
1457 # service icinga2 reload
1459 #### <a id="high-availability-notifications"></a> High Availability with Notifications
1461 Notifications are load balanced amongst all nodes in a zone. By default this functionality
1463 If your nodes should notify independent from any other nodes (this will cause
1464 duplicated notifications if not properly handled!), you can set `enable_ha = false`
1465 in the [NotificationComponent](#objecttype-notificationcomponent) feature.
1467 #### <a id="high-availability-db-ido"></a> High Availability with DB IDO
1469 All instances within the same zone (e.g. the `master` zone as HA cluster) must
1470 have the DB IDO feature enabled.
1472 Example DB IDO MySQL:
1474 # icinga2 feature enable ido-mysql
1475 The feature 'ido-mysql' is already enabled.
1477 By default the DB IDO feature only runs on the elected zone master. All other passive
1478 nodes disable the active IDO database connection at runtime.
1482 > The DB IDO HA feature can be disabled by setting the `enable_ha` attribute to `false`
1483 > for the [IdoMysqlConnection](#objecttype-idomysqlconnection) or
1484 > [IdoPgsqlConnection](#objecttype-idopgsqlconnection) object on all nodes in the
1487 > All endpoints will enable the DB IDO feature then, connect to the configured
1488 > database and dump configuration, status and historical data on their own.
1490 If the instance with the active DB IDO connection dies, the HA functionality will
1491 re-enable the DB IDO connection on the newly elected zone master.
1493 The DB IDO feature will try to determine which cluster endpoint is currently writing
1494 to the database and bail out if another endpoint is active. You can manually verify that
1495 by running the following query:
1497 icinga=> SELECT status_update_time, endpoint_name FROM icinga_programstatus;
1498 status_update_time | endpoint_name
1499 ------------------------+---------------
1500 2014-08-15 15:52:26+02 | icinga2a
1503 This is useful when the cluster connection between endpoints breaks, and prevents
1504 data duplication in split-brain-scenarios. The failover timeout can be set for the
1505 `failover_timeout` attribute, but not lower than 60 seconds.
1508 ### <a id="cluster-add-node"></a> Add a new cluster endpoint
1510 These steps are required for integrating a new cluster endpoint:
1512 * generate a new [SSL client certificate](#certificate-authority-certificates)
1513 * identify its location in the zones
1514 * update the `zones.conf` file on each involved node ([endpoint](#configure-cluster-endpoints), [zones](#configure-cluster-zones))
1515 * a new slave zone node requires updates for the master and slave zones
1516 * verify if this endpoints requires [configuration synchronisation](#cluster-zone-config-sync) enabled
1517 * if the node requires the existing zone history: [initial cluster sync](#initial-cluster-sync)
1518 * add a [cluster health check](#cluster-health-check)
1520 #### <a id="initial-cluster-sync"></a> Initial Cluster Sync
1522 In order to make sure that all of your cluster nodes have the same state you will
1523 have to pick one of the nodes as your initial "master" and copy its state file
1524 to all the other nodes.
1526 You can find the state file in `/var/lib/icinga2/icinga2.state`. Before copying
1527 the state file you should make sure that all your cluster nodes are properly shut
1531 ### <a id="host-multiple-cluster-nodes"></a> Host With Multiple Cluster Nodes
1533 Special scenarios might require multiple cluster nodes running on a single host.
1534 By default Icinga 2 and its features will place their runtime data below the prefix
1535 `LocalStateDir`. By default packages will set that path to `/var`.
1536 You can either set that variable as constant configuration
1537 definition in [icinga2.conf](#icinga2-conf) or pass it as runtime variable to
1538 the Icinga 2 daemon.
1540 # icinga2 -c /etc/icinga2/node1/icinga2.conf -DLocalStateDir=/opt/node1/var