1 # Distributed Monitoring with Master, Satellites, and Clients <a id="distributed-monitoring"></a>
3 This chapter will guide you through the setup of a distributed monitoring
4 environment, including high-availability clustering and setup details
5 for the Icinga 2 client.
7 ## Roles: Master, Satellites, and Clients <a id="distributed-monitoring-roles"></a>
9 Icinga 2 nodes can be given names for easier understanding:
11 * A `master` node which is on top of the hierarchy.
12 * A `satellite` node which is a child of a `satellite` or `master` node.
13 * A `client` node which works as an `agent` connected to `master` and/or `satellite` nodes.
15 ![Icinga 2 Distributed Roles](images/distributed-monitoring/icinga2_distributed_roles.png)
17 Rephrasing this picture into more details:
19 * A `master` node has no parent node.
20 * A `master`node is where you usually install Icinga Web 2.
21 * A `master` node can combine executed checks from child nodes into backends and notifications.
22 * A `satellite` node has a parent and a child node.
23 * A `satellite` node may execute checks on its own or delegate check execution to child nodes.
24 * A `satellite` node can receive configuration for hosts/services, etc. from the parent node.
25 * A `satellite` node continues to run even if the master node is temporarily unavailable.
26 * A `client` node only has a parent node.
27 * A `client` node will either run its own configured checks or receive command execution events from the parent node.
29 The following sections will refer to these roles and explain the
30 differences and the possibilities this kind of setup offers.
32 **Tip**: If you just want to install a single master node that monitors several hosts
33 (i.e. Icinga 2 clients), continue reading -- we'll start with
35 In case you are planning a huge cluster setup with multiple levels and
36 lots of clients, read on -- we'll deal with these cases later on.
38 The installation on each system is the same: You need to install the
39 [Icinga 2 package](02-getting-started.md#setting-up-icinga2) and the required [plugins](02-getting-started.md#setting-up-check-plugins).
41 The required configuration steps are mostly happening
42 on the command line. You can also [automate the setup](06-distributed-monitoring.md#distributed-monitoring-automation).
44 The first thing you need learn about a distributed setup is the hierarchy of the single components.
46 ## Zones <a id="distributed-monitoring-zones"></a>
48 The Icinga 2 hierarchy consists of so-called [zone](09-object-types.md#objecttype-zone) objects.
49 Zones depend on a parent-child relationship in order to trust each other.
51 ![Icinga 2 Distributed Zones](images/distributed-monitoring/icinga2_distributed_zones.png)
53 Have a look at this example for the `satellite` zones which have the `master` zone as a parent zone:
55 object Zone "master" {
59 object Zone "satellite region 1" {
64 object Zone "satellite region 2" {
69 There are certain limitations for child zones, e.g. their members are not allowed
70 to send configuration commands to the parent zone members. Vice versa, the
71 trust hierarchy allows for example the `master` zone to send
72 configuration files to the `satellite` zone. Read more about this
73 in the [security section](06-distributed-monitoring.md#distributed-monitoring-security).
75 `client` nodes also have their own unique zone. By convention you
76 can use the FQDN for the zone name.
78 ## Endpoints <a id="distributed-monitoring-endpoints"></a>
80 Nodes which are a member of a zone are so-called [Endpoint](09-object-types.md#objecttype-endpoint) objects.
82 ![Icinga 2 Distributed Endpoints](images/distributed-monitoring/icinga2_distributed_endpoints.png)
84 Here is an example configuration for two endpoints in different zones:
86 object Endpoint "icinga2-master1.localdomain" {
87 host = "192.168.56.101"
90 object Endpoint "icinga2-satellite1.localdomain" {
91 host = "192.168.56.105"
94 object Zone "master" {
95 endpoints = [ "icinga2-master1.localdomain" ]
98 object Zone "satellite" {
99 endpoints = [ "icinga2-satellite1.localdomain" ]
103 All endpoints in the same zone work as high-availability setup. For
104 example, if you have two nodes in the `master` zone, they will load-balance the check execution.
106 Endpoint objects are important for specifying the connection
107 information, e.g. if the master should actively try to connect to a client.
109 The zone membership is defined inside the `Zone` object definition using
110 the `endpoints` attribute with an array of `Endpoint` names.
114 > There is a known [problem](https://github.com/Icinga/icinga2/issues/3533)
115 > with >2 endpoints in a zone and a message routing loop.
116 > The config validation will log a warning to let you know about this too.
118 If you want to check the availability (e.g. ping checks) of the node
119 you still need a [Host](09-object-types.md#objecttype-host) object.
121 ## ApiListener <a id="distributed-monitoring-apilistener"></a>
123 In case you are using the CLI commands later, you don't have to write
124 this configuration from scratch in a text editor.
125 The [ApiListener](09-object-types.md#objecttype-apilistener) object is
126 used to load the TLS certificates and specify restrictions, e.g.
127 for accepting configuration commands.
129 It is also used for the [Icinga 2 REST API](12-icinga2-api.md#icinga2-api) which shares
130 the same host and port with the Icinga 2 Cluster protocol.
132 The object configuration is stored in the `/etc/icinga2/features-enabled/api.conf`
133 file. Depending on the configuration mode the attributes `accept_commands`
134 and `accept_config` can be configured here.
136 In order to use the `api` feature you need to enable it and restart Icinga 2.
138 icinga2 feature enable api
140 ## Conventions <a id="distributed-monitoring-conventions"></a>
142 By convention all nodes should be configured using their FQDN.
144 Furthermore, you must ensure that the following names
145 are exactly the same in all configuration files:
147 * Host certificate common name (CN).
148 * Endpoint configuration object for the host.
149 * NodeName constant for the local host.
151 Setting this up on the command line will help you to minimize the effort.
152 Just keep in mind that you need to use the FQDN for endpoints and for
153 common names when asked.
155 ## Security <a id="distributed-monitoring-security"></a>
157 While there are certain mechanisms to ensure a secure communication between all
158 nodes (firewalls, policies, software hardening, etc.), Icinga 2 also provides
161 * SSL certificates are mandatory for communication between nodes. The CLI commands
162 help you create those certificates.
163 * Child zones only receive updates (check results, commands, etc.) for their configured objects.
164 * Child zones are not allowed to push configuration updates to parent zones.
165 * Zones cannot interfere with other zones and influence each other. Each checkable host or service object is assigned to **one zone** only.
166 * All nodes in a zone trust each other.
167 * [Config sync](06-distributed-monitoring.md#distributed-monitoring-top-down-config-sync) and [remote command endpoint execution](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint) is disabled by default.
169 The underlying protocol uses JSON-RPC event notifications exchanged by nodes.
170 The connection is secured by TLS. The message protocol uses an internal API,
171 and as such message types and names may change internally and are not documented.
173 Zones build the trust relationship in a distributed environment. If you do not specify
174 a zone for a client and specify the parent zone, its zone members e.g. the master instance
175 won't trust the client.
177 Building this trust is key in your distributed environment. That way the parent node
178 knows that it is able to send messages to the child zone, e.g. configuration objects,
179 configuration in global zones, commands to be executed in this zone/for this endpoint.
180 It also receives check results from the child zone for checkable objects (host/service).
182 Vice versa, the client trusts the master and accepts configuration and commands if enabled
183 in the api feature. If the client would send configuration to the parent zone, the parent nodes
184 will deny it. The parent zone is the configuration entity, and does not trust clients in this matter.
185 A client could attempt to modify a different client for example, or inject a check command
188 While it may sound complicated for client setups, it removes the problem with different roles
189 and configurations for a master and a client. Both of them work the same way, are configured
190 in the same way (Zone, Endpoint, ApiListener), and you can troubleshoot and debug them in just one go.
192 ## Versions and Upgrade <a id="distributed-monitoring-versions-upgrade"></a>
194 It generally is advised to use the newest releases with the same version on all instances.
195 Prior to upgrading, make sure to plan a maintenance window.
197 The Icinga project aims to allow the following compatibility:
200 master (2.9) >= satellite (2.8) >= clients (2.7)
203 Older client versions may work, but there's no guarantee. Always keep in mind that
204 older versions are out of support and can contain bugs.
206 In terms of an upgrade, ensure that the master is upgraded first, then
207 involved satellites, and last the Icinga 2 clients. If you are on v2.8
208 currently, first upgrade the master instance(s) to 2.9, and then proceed
209 with the satellites. Things are getting easier with any sort of automation
210 tool (Puppet, Ansible, etc.).
212 Releases and new features may require you to upgrade master/satellite instances at once,
213 this is highlighted in the [upgrading docs](16-upgrading-icinga-2.md#upgrading-icinga-2) if needed.
214 One example is the CA Proxy and on-demand signing feature
215 available since v2.8 where all involved instances need this version
216 to function properly.
218 ## Master Setup <a id="distributed-monitoring-setup-master"></a>
220 This section explains how to install a central single master node using
221 the `node wizard` command. If you prefer to do an automated installation, please
222 refer to the [automated setup](06-distributed-monitoring.md#distributed-monitoring-automation) section.
224 Install the [Icinga 2 package](02-getting-started.md#setting-up-icinga2) and setup
225 the required [plugins](02-getting-started.md#setting-up-check-plugins) if you haven't done
228 **Note**: Windows is not supported for a master node setup.
230 The next step is to run the `node wizard` CLI command. Prior to that
231 ensure to collect the required information:
233 Parameter | Description
234 --------------------|--------------------
235 Common name (CN) | **Required.** By convention this should be the host's FQDN. Defaults to the FQDN.
236 Master zone name | **Optional.** Allows to specify the master zone name. Defaults to `master`.
237 Global zones | **Optional.** Allows to specify more global zones in addition to `global-templates` and `director-global`. Defaults to `n`.
238 API bind host | **Optional.** Allows to specify the address the ApiListener is bound to. For advanced usage only.
239 API bind port | **Optional.** Allows to specify the port the ApiListener is bound to. For advanced usage only (requires changing the default port 5665 everywhere).
240 Disable conf.d | **Optional.** Allows to disable the `include_recursive "conf.d"` directive except for the `api-users.conf` file in the `icinga2.conf` file. Defaults to `y`. Configuration on the master is discussed below.
242 The setup wizard will ensure that the following steps are taken:
244 * Enable the `api` feature.
245 * Generate a new certificate authority (CA) in `/var/lib/icinga2/ca` if it doesn't exist.
246 * Create a certificate for this node signed by the CA key.
247 * Update the [zones.conf](04-configuring-icinga-2.md#zones-conf) file with the new zone hierarchy.
248 * Update the [ApiListener](06-distributed-monitoring.md#distributed-monitoring-apilistener) and [constants](04-configuring-icinga-2.md#constants-conf) configuration.
249 * Update the [icinga2.conf](04-configuring-icinga-2.md#icinga2-conf) to disable the `conf.d` inclusion, and add the `api-users.conf` file inclusion.
251 Here is an example of a master setup for the `icinga2-master1.localdomain` node on CentOS 7:
254 [root@icinga2-master1.localdomain /]# icinga2 node wizard
256 Welcome to the Icinga 2 Setup Wizard!
258 We will guide you through all required configuration details.
260 Please specify if this is a satellite/client setup ('n' installs a master setup) [Y/n]: n
262 Starting the Master setup routine...
264 Please specify the common name (CN) [icinga2-master1.localdomain]: icinga2-master1.localdomain
265 Reconfiguring Icinga...
266 Checking for existing certificates for common name 'icinga2-master1.localdomain'...
267 Certificates not yet generated. Running 'api setup' now.
268 Generating master configuration for Icinga 2.
269 Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
271 Master zone name [master]:
273 Default global zones: global-templates director-global
274 Do you want to specify additional global zones? [y/N]: N
276 Please specify the API bind host/port (optional):
280 Do you want to disable the inclusion of the conf.d directory [Y/n]:
281 Disabling the inclusion of the conf.d directory...
282 Checking if the api-users.conf file exists...
286 Now restart your Icinga 2 daemon to finish the installation!
289 You can verify that the CA public and private keys are stored in the `/var/lib/icinga2/ca` directory.
290 Keep this path secure and include it in your [backups](02-getting-started.md#install-backup).
292 In case you lose the CA private key you have to generate a new CA for signing new client
293 certificate requests. You then have to also re-create new signed certificates for all
296 Once the master setup is complete, you can also use this node as primary [CSR auto-signing](06-distributed-monitoring.md#distributed-monitoring-setup-csr-auto-signing)
297 master. The following section will explain how to use the CLI commands in order to fetch their
298 signed certificate from this master node.
300 ## Signing Certificates on the Master <a id="distributed-monitoring-setup-sign-certificates-master"></a>
302 All certificates must be signed by the same certificate authority (CA). This ensures
303 that all nodes trust each other in a distributed monitoring environment.
305 This CA is generated during the [master setup](06-distributed-monitoring.md#distributed-monitoring-setup-master)
306 and should be the same on all master instances.
308 You can avoid signing and deploying certificates [manually](06-distributed-monitoring.md#distributed-monitoring-advanced-hints-certificates-manual)
309 by using built-in methods for auto-signing certificate signing requests (CSR):
311 * [CSR Auto-Signing](06-distributed-monitoring.md#distributed-monitoring-setup-csr-auto-signing) which uses a client ticket generated on the master as trust identifier.
312 * [On-Demand CSR Signing](06-distributed-monitoring.md#distributed-monitoring-setup-on-demand-csr-signing) which allows to sign pending certificate requests on the master.
314 Both methods are described in detail below.
318 > [On-Demand CSR Signing](06-distributed-monitoring.md#distributed-monitoring-setup-on-demand-csr-signing) is available in Icinga 2 v2.8+.
320 ### CSR Auto-Signing <a id="distributed-monitoring-setup-csr-auto-signing"></a>
322 A client which sends a certificate signing request (CSR) must authenticate itself
323 in a trusted way. The master generates a client ticket which is included in this request.
324 That way the master can verify that the request matches the previously trusted ticket
325 and sign the request.
329 > Icinga 2 v2.8 adds the possibility to forward signing requests on a satellite
330 > to the master node. This is called `CA Proxy` in blog posts and design drafts.
331 > This functionality helps with the setup of [three level clusters](#06-distributed-monitoring.md#distributed-monitoring-scenarios-master-satellite-client)
336 * Nodes can be installed by different users who have received the client ticket.
337 * No manual interaction necessary on the master node.
338 * Automation tools like Puppet, Ansible, etc. can retrieve the pre-generated ticket in their client catalog
339 and run the node setup directly.
343 * Tickets need to be generated on the master and copied to client setup wizards.
344 * No central signing management.
347 Setup wizards for satellite/client nodes will ask you for this specific client ticket.
349 There are two possible ways to retrieve the ticket:
351 * [CLI command](11-cli-commands.md#cli-command-pki) executed on the master node.
352 * [REST API](12-icinga2-api.md#icinga2-api) request against the master node.
354 Required information:
356 Parameter | Description
357 --------------------|--------------------
358 Common name (CN) | **Required.** The common name for the satellite/client. By convention this should be the FQDN.
360 The following example shows how to generate a ticket on the master node `icinga2-master1.localdomain` for the client `icinga2-client1.localdomain`:
362 [root@icinga2-master1.localdomain /]# icinga2 pki ticket --cn icinga2-client1.localdomain
364 Querying the [Icinga 2 API](12-icinga2-api.md#icinga2-api) on the master requires an [ApiUser](12-icinga2-api.md#icinga2-api-authentication)
365 object with at least the `actions/generate-ticket` permission.
367 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/conf.d/api-users.conf
369 object ApiUser "client-pki-ticket" {
370 password = "bea11beb7b810ea9ce6ea" //change this
371 permissions = [ "actions/generate-ticket" ]
374 [root@icinga2-master1.localdomain /]# systemctl restart icinga2
376 Retrieve the ticket on the master node `icinga2-master1.localdomain` with `curl`, for example:
378 [root@icinga2-master1.localdomain /]# curl -k -s -u client-pki-ticket:bea11beb7b810ea9ce6ea -H 'Accept: application/json' \
379 -X POST 'https://localhost:5665/v1/actions/generate-ticket' -d '{ "cn": "icinga2-client1.localdomain" }'
381 Store that ticket number for the satellite/client setup below.
385 > Never expose the ticket salt and/or ApiUser credentials to your client nodes.
386 > Example: Retrieve the ticket on the Puppet master node and send the compiled catalog
387 > to the authorized Puppet agent node which will invoke the
388 > [automated setup steps](06-distributed-monitoring.md#distributed-monitoring-automation-cli-node-setup).
390 ### On-Demand CSR Signing <a id="distributed-monitoring-setup-on-demand-csr-signing"></a>
392 The client sends a certificate signing request to specified parent node without any
393 ticket. The admin on the master is responsible for reviewing and signing the requests
394 with the private CA key.
396 This could either be directly the master, or a satellite which forwards the request
397 to the signing master.
401 * Central certificate request signing management.
402 * No pre-generated ticket is required for client setups.
406 * Asynchronous step for automated deployments.
407 * Needs client verification on the master.
410 You can list certificate requests by using the `ca list` CLI command. This also shows
411 which requests already have been signed.
414 [root@icinga2-master1.localdomain /]# icinga2 ca list
415 Fingerprint | Timestamp | Signed | Subject
416 -----------------------------------------------------------------|---------------------|--------|--------
417 403da5b228df384f07f980f45ba50202529cded7c8182abf96740660caa09727 | 2017/09/06 17:02:40 | * | CN = icinga2-client1.localdomain
418 71700c28445109416dd7102038962ac3fd421fbb349a6e7303b6033ec1772850 | 2017/09/06 17:20:02 | | CN = icinga2-client2.localdomain
421 **Tip**: Add `--json` to the CLI command to retrieve the details in JSON format.
423 If you want to sign a specific request, you need to use the `ca sign` CLI command
424 and pass its fingerprint as argument.
427 [root@icinga2-master1.localdomain /]# icinga2 ca sign 71700c28445109416dd7102038962ac3fd421fbb349a6e7303b6033ec1772850
428 information/cli: Signed certificate for 'CN = icinga2-client2.localdomain'.
431 ## Client/Satellite Setup <a id="distributed-monitoring-setup-satellite-client"></a>
433 This section describes the setup of a satellite and/or client connected to an
434 existing master node setup. If you haven't done so already, please [run the master setup](06-distributed-monitoring.md#distributed-monitoring-setup-master).
436 Icinga 2 on the master node must be running and accepting connections on port `5665`.
439 ### Client/Satellite Setup on Linux <a id="distributed-monitoring-setup-client-linux"></a>
441 Please ensure that you've run all the steps mentioned in the [client/satellite section](06-distributed-monitoring.md#distributed-monitoring-setup-satellite-client).
443 Install the [Icinga 2 package](02-getting-started.md#setting-up-icinga2) and setup
444 the required [plugins](02-getting-started.md#setting-up-check-plugins) if you haven't done
447 The next step is to run the `node wizard` CLI command.
449 In this example we're generating a ticket on the master node `icinga2-master1.localdomain` for the client `icinga2-client1.localdomain`:
451 [root@icinga2-master1.localdomain /]# icinga2 pki ticket --cn icinga2-client1.localdomain
452 4f75d2ecd253575fe9180938ebff7cbca262f96e
454 Note: You don't need this step if you have chosen to use [On-Demand CSR Signing](06-distributed-monitoring.md#distributed-monitoring-setup-on-demand-csr-signing).
456 Start the wizard on the client `icinga2-client1.localdomain`:
459 [root@icinga2-client1.localdomain /]# icinga2 node wizard
461 Welcome to the Icinga 2 Setup Wizard!
463 We will guide you through all required configuration details.
466 Press `Enter` or add `y` to start a satellite or client setup.
469 Please specify if this is a satellite/client setup ('n' installs a master setup) [Y/n]:
472 Press `Enter` to use the proposed name in brackets, or add a specific common name (CN). By convention
473 this should be the FQDN.
476 Starting the Client/Satellite setup routine...
478 Please specify the common name (CN) [icinga2-client1.localdomain]: icinga2-client1.localdomain
481 Specify the direct parent for this node. This could be your primary master `icinga2-master1.localdomain`
482 or a satellite node in a multi level cluster scenario.
485 Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
486 Master/Satellite Common Name (CN from your master/satellite node): icinga2-master1.localdomain
489 Press `Enter` or choose `y` to establish a connection to the parent node.
492 Do you want to establish a connection to the parent node from this node? [Y/n]:
497 > If this node cannot connect to the parent node, choose `n`. The setup
498 > wizard will provide instructions for this scenario -- signing questions are disabled then.
500 Add the connection details for `icinga2-master1.localdomain`.
503 Please specify the master/satellite connection information:
504 Master/Satellite endpoint host (IP address or FQDN): 192.168.56.101
505 Master/Satellite endpoint port [5665]: 5665
508 You can add more parent nodes if necessary. Press `Enter` or choose `n`
509 if you don't want to add any. This comes in handy if you have more than one
510 parent node, e.g. two masters or two satellites.
513 Add more master/satellite endpoints? [y/N]:
516 Verify the parent node's certificate:
519 Parent certificate information:
521 Subject: CN = icinga2-master1.localdomain
522 Issuer: CN = Icinga CA
523 Valid From: Sep 7 13:41:24 2017 GMT
524 Valid Until: Sep 3 13:41:24 2032 GMT
525 Fingerprint: AC 99 8B 2B 3D B0 01 00 E5 21 FA 05 2E EC D5 A9 EF 9E AA E3
527 Is this information correct? [y/N]: y
530 The setup wizard fetches the parent node's certificate and ask
531 you to verify this information. This is to prevent MITM attacks or
532 any kind of untrusted parent relationship.
534 Note: The certificate is not fetched if you have chosen not to connect
537 Proceed with adding the optional client ticket for [CSR auto-signing](06-distributed-monitoring.md#distributed-monitoring-setup-csr-auto-signing):
540 Please specify the request ticket generated on your Icinga 2 master (optional).
541 (Hint: # icinga2 pki ticket --cn 'icinga2-client1.localdomain'):
542 4f75d2ecd253575fe9180938ebff7cbca262f96e
545 In case you've chosen to use [On-Demand CSR Signing](06-distributed-monitoring.md#distributed-monitoring-setup-on-demand-csr-signing)
546 you can leave the ticket question blank.
548 Instead, Icinga 2 tells you to approve the request later on the master node.
551 No ticket was specified. Please approve the certificate signing request manually
552 on the master (see 'icinga2 ca list' and 'icinga2 ca sign --help' for details).
555 You can optionally specify a different bind host and/or port.
558 Please specify the API bind host/port (optional):
563 The next step asks you to accept configuration (required for [config sync mode](06-distributed-monitoring.md#distributed-monitoring-top-down-config-sync))
564 and commands (required for [command endpoint mode](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint)).
567 Accept config from parent node? [y/N]: y
568 Accept commands from parent node? [y/N]: y
571 Next you can optionally specify the local and parent zone names. This will be reflected
572 in the generated zone configuration file.
574 Set the local zone name to something else, if you are installing a satellite or secondary master instance.
577 Local zone name [icinga2-client1.localdomain]:
580 Set the parent zone name to something else than `master` if this client connects to a satellite instance instead of the master.
583 Parent zone name [master]:
586 You can add more global zones in addition to `global-templates` and `director-global` if necessary.
587 Press `Enter` or choose `n`, if you don't want to add any additional.
590 Reconfiguring Icinga...
592 Default global zones: global-templates director-global
593 Do you want to specify additional global zones? [y/N]: N
596 Last but not least the wizard asks you whether you want to disable the inclusion of the local configuration
597 directory in `conf.d`, or not. Defaults to disabled, as clients either are checked via command endpoint, or
598 they receive configuration synced from the parent zone.
601 Do you want to disable the inclusion of the conf.d directory [Y/n]: Y
602 Disabling the inclusion of the conf.d directory...
606 The wizard proceeds and you are good to go.
611 Now restart your Icinga 2 daemon to finish the installation!
616 > If you have chosen not to connect to the parent node, you cannot start
617 > Icinga 2 yet. The wizard asked you to manually copy the master's public
618 > CA certificate file into `/var/lib/icinga2/certs/ca.crt`.
620 > You need to manually sign the CSR on the master node.
622 Restart Icinga 2 as requested.
625 [root@icinga2-client1.localdomain /]# systemctl restart icinga2
628 Here is an overview of all parameters in detail:
630 Parameter | Description
631 --------------------|--------------------
632 Common name (CN) | **Required.** By convention this should be the host's FQDN. Defaults to the FQDN.
633 Master common name | **Required.** Use the common name you've specified for your master node before.
634 Establish connection to the parent node | **Optional.** Whether the node should attempt to connect to the parent node or not. Defaults to `y`.
635 Master/Satellite endpoint host | **Required if the the client needs to connect to the master/satellite.** The parent endpoint's IP address or FQDN. This information is included in the `Endpoint` object configuration in the `zones.conf` file.
636 Master/Satellite endpoint port | **Optional if the the client needs to connect to the master/satellite.** The parent endpoints's listening port. This information is included in the `Endpoint` object configuration.
637 Add more master/satellite endpoints | **Optional.** If you have multiple master/satellite nodes configured, add them here.
638 Parent Certificate information | **Required.** Verify that the connecting host really is the requested master node.
639 Request ticket | **Optional.** Add the [ticket](06-distributed-monitoring.md#distributed-monitoring-setup-csr-auto-signing) generated on the master.
640 API bind host | **Optional.** Allows to specify the address the ApiListener is bound to. For advanced usage only.
641 API bind port | **Optional.** Allows to specify the port the ApiListener is bound to. For advanced usage only (requires changing the default port 5665 everywhere).
642 Accept config | **Optional.** Whether this node accepts configuration sync from the master node (required for [config sync mode](06-distributed-monitoring.md#distributed-monitoring-top-down-config-sync)). For [security reasons](06-distributed-monitoring.md#distributed-monitoring-security) this defaults to `n`.
643 Accept commands | **Optional.** Whether this node accepts command execution messages from the master node (required for [command endpoint mode](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint)). For [security reasons](06-distributed-monitoring.md#distributed-monitoring-security) this defaults to `n`.
644 Local zone name | **Optional.** Allows to specify the name for the local zone. This comes in handy when this instance is a satellite, not a client. Defaults to the FQDN.
645 Parent zone name | **Optional.** Allows to specify the name for the parent zone. This is important if the client has a satellite instance as parent, not the master. Defaults to `master`.
646 Global zones | **Optional.** Allows to specify more global zones in addition to `global-templates` and `director-global`. Defaults to `n`.
647 Disable conf.d | **Optional.** Allows to disable the inclusion of the `conf.d` directory which holds local example configuration. Clients should retrieve their configuration from the parent node, or act as command endpoint execution bridge. Defaults to `y`.
649 The setup wizard will ensure that the following steps are taken:
651 * Enable the `api` feature.
652 * Create a certificate signing request (CSR) for the local node.
653 * Request a signed certificate i(optional with the provided ticket number) on the master node.
654 * Allow to verify the parent node's certificate.
655 * Store the signed client certificate and ca.crt in `/var/lib/icinga2/certs`.
656 * Update the `zones.conf` file with the new zone hierarchy.
657 * Update `/etc/icinga2/features-enabled/api.conf` (`accept_config`, `accept_commands`) and `constants.conf`.
658 * Update `/etc/icinga2/icinga2.conf` and comment out `include_recursive "conf.d"`.
660 You can verify that the certificate files are stored in the `/var/lib/icinga2/certs` directory.
664 > If the client is not directly connected to the certificate signing master,
665 > signing requests and responses might need some minutes to fully update the client certificates.
667 > If you have chosen to use [On-Demand CSR Signing](06-distributed-monitoring.md#distributed-monitoring-setup-on-demand-csr-signing)
668 > certificates need to be signed on the master first. Ticket-less setups require at least Icinga 2 v2.8+ on all involved instances.
670 Now that you've successfully installed a Linux/Unix satellite/client instance, please proceed to
671 the [configuration modes](06-distributed-monitoring.md#distributed-monitoring-configuration-modes).
675 ### Client Setup on Windows <a id="distributed-monitoring-setup-client-windows"></a>
677 Download the MSI-Installer package from [https://packages.icinga.com/windows/](https://packages.icinga.com/windows/).
681 * Windows Vista/Server 2008 or higher
682 * Versions older than Windows 10/Server 2016 require the [Universal C Runtime for Windows](https://support.microsoft.com/en-us/help/2999226/update-for-universal-c-runtime-in-windows)
683 * [Microsoft .NET Framework 2.0](https://www.microsoft.com/de-de/download/details.aspx?id=1639) for the setup wizard
685 The installer package includes the [NSClient++](https://www.nsclient.org/) package
686 so that Icinga 2 can use its built-in plugins. You can find more details in
687 [this chapter](06-distributed-monitoring.md#distributed-monitoring-windows-nscp).
688 The Windows package also installs native [monitoring plugin binaries](06-distributed-monitoring.md#distributed-monitoring-windows-plugins)
689 to get you started more easily.
693 > Please note that Icinga 2 was designed to run as light-weight client on Windows.
694 > There is no support for satellite instances.
696 #### Windows Client Setup Start <a id="distributed-monitoring-setup-client-windows-start"></a>
698 Run the MSI-Installer package and follow the instructions shown in the screenshots.
700 ![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_installer_01.png)
701 ![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_installer_02.png)
702 ![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_installer_03.png)
703 ![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_installer_04.png)
704 ![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_installer_05.png)
706 The graphical installer offers to run the Icinga 2 setup wizard after the installation. Select
707 the check box to proceed.
711 > You can also run the Icinga 2 setup wizard from the Start menu later.
713 On a fresh installation the setup wizard guides you through the initial configuration.
714 It also provides a mechanism to send a certificate request to the [CSR signing master](distributed-monitoring-setup-sign-certificates-master).
716 The following configuration details are required:
718 Parameter | Description
719 --------------------|--------------------
720 Instance name | **Required.** By convention this should be the host's FQDN. Defaults to the FQDN.
721 Setup ticket | **Optional.** Paste the previously generated [ticket number](06-distributed-monitoring.md#distributed-monitoring-setup-csr-auto-signing). If left blank, the certificate request must be [signed on the master node](06-distributed-monitoring.md#distributed-monitoring-setup-on-demand-csr-signing).
723 Fill in the required information and click `Add` to add a new master connection.
725 ![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_wizard_01.png)
727 Add the following details:
729 Parameter | Description
730 -------------------------------|-------------------------------
731 Instance name | **Required.** The master/satellite endpoint name where this client is a direct child of.
732 Master/Satellite endpoint host | **Required.** The master or satellite's IP address or FQDN. This information is included in the `Endpoint` object configuration in the `zones.conf` file.
733 Master/Satellite endpoint port | **Optional.** The master or satellite's listening port. This information is included in the `Endpoint` object configuration.
735 ![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_wizard_02.png)
737 When needed you can add an additional global zone (the zones `global-templates` and `director-global` are added by default):
739 ![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_wizard_02_global_zone.png)
741 Optionally enable the following settings:
743 Parameter | Description
744 ----------------------------------|----------------------------------
745 Accept config | **Optional.** Whether this node accepts configuration sync from the master node (required for [config sync mode](06-distributed-monitoring.md#distributed-monitoring-top-down-config-sync)). For [security reasons](06-distributed-monitoring.md#distributed-monitoring-security) this is disabled by default.
746 Accept commands | **Optional.** Whether this node accepts command execution messages from the master node (required for [command endpoint mode](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint)). For [security reasons](06-distributed-monitoring.md#distributed-monitoring-security) this is disabled by default.
747 Run Icinga 2 service as this user | **Optional.** Specify a different Windows user. This defaults to `NT AUTHORITY\Network Service` and is required for more privileged service checks.
748 Install NSClient++ | **Optional.** The Windows installer bundles the NSClient++ installer for additional [plugin checks](06-distributed-monitoring.md#distributed-monitoring-windows-nscp).
749 Disable conf.d | **Optional.** Allows to disable the `include_recursive "conf.d"` directive except for the `api-users.conf` file in the `icinga2.conf` file. Defaults to `true`.
751 ![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_wizard_03.png)
753 Verify the certificate from the master/satellite instance where this node should connect to.
755 ![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_wizard_04.png)
758 #### Bundled NSClient++ Setup <a id="distributed-monitoring-setup-client-windows-nsclient"></a>
760 If you have chosen to install/update the NSClient++ package, the Icinga 2 setup wizard asks
763 ![Icinga 2 Windows Setup NSClient++](images/distributed-monitoring/icinga2_windows_setup_wizard_05_nsclient_01.png)
765 Choose the `Generic` setup.
767 ![Icinga 2 Windows Setup NSClient++](images/distributed-monitoring/icinga2_windows_setup_wizard_05_nsclient_02.png)
769 Choose the `Custom` setup type.
771 ![Icinga 2 Windows Setup NSClient++](images/distributed-monitoring/icinga2_windows_setup_wizard_05_nsclient_03.png)
773 NSClient++ does not install a sample configuration by default. Change this as shown in the screenshot.
775 ![Icinga 2 Windows Setup NSClient++](images/distributed-monitoring/icinga2_windows_setup_wizard_05_nsclient_04.png)
777 Generate a secure password and enable the web server module. **Note**: The webserver module is
778 available starting with NSClient++ 0.5.0. Icinga 2 v2.6+ is required which includes this version.
780 ![Icinga 2 Windows Setup NSClient++](images/distributed-monitoring/icinga2_windows_setup_wizard_05_nsclient_05.png)
782 Finish the installation.
784 ![Icinga 2 Windows Setup NSClient++](images/distributed-monitoring/icinga2_windows_setup_wizard_05_nsclient_06.png)
786 Open a web browser and navigate to `https://localhost:8443`. Enter the password you've configured
787 during the setup. In case you lost it, look into the `C:\Program Files\NSClient++\nsclient.ini`
790 ![Icinga 2 Windows Setup NSClient++](images/distributed-monitoring/icinga2_windows_setup_wizard_05_nsclient_07.png)
792 The NSClient++ REST API can be used to query metrics. [check_nscp_api](06-distributed-monitoring.md#distributed-monitoring-windows-nscp-check-api)
793 uses this transport method.
796 #### Finish Windows Client Setup <a id="distributed-monitoring-setup-client-windows-finish"></a>
798 Finish the Windows setup wizard.
800 ![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_wizard_06_finish_with_ticket.png)
802 If you did not provide a setup ticket, you need to sign the certificate request on the master.
803 The setup wizards tells you to do so. The Icinga 2 service is running at this point already
804 and will automatically receive and update a signed client certificate.
806 ![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_wizard_06_finish_no_ticket.png)
808 Icinga 2 is automatically started as a Windows service.
810 ![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_running_service.png)
812 The Icinga 2 configuration is stored inside the `C:\ProgramData\icinga2` directory.
813 Click `Examine Config` in the setup wizard to open a new Explorer window.
815 ![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_wizard_examine_config.png)
817 The configuration files can be modified with your favorite editor e.g. Notepad.
819 In order to use the [top down](06-distributed-monitoring.md#distributed-monitoring-top-down) client
820 configuration prepare the following steps.
822 You don't need any local configuration on the client except for
823 CheckCommand definitions which can be synced using the global zone
824 above. Therefore disable the inclusion of the `conf.d` directory
825 in the `icinga2.conf` file.
826 Navigate to `C:\ProgramData\icinga2\etc\icinga2` and open
827 the `icinga2.conf` file in your preferred editor. Remove or comment (`//`)
831 // Commented out, not required on a client with top down mode
832 //include_recursive "conf.d"
837 > Packages >= 2.9 provide an option in the setup wizard to disable this.
838 > Defaults to disabled.
840 Validate the configuration on Windows open an administrator terminal
841 and run the following command:
844 C:\WINDOWS\system32>cd "C:\Program Files\ICINGA2\sbin"
845 C:\Program Files\ICINGA2\sbin>icinga2.exe daemon -C
848 **Note**: You have to run this command in a shell with `administrator` privileges.
850 Now you need to restart the Icinga 2 service. Run `services.msc` from the start menu
851 and restart the `icinga2` service. Alternatively, you can use the `net {start,stop}` CLI commands.
853 ![Icinga 2 Windows Service Start/Stop](images/distributed-monitoring/icinga2_windows_cmd_admin_net_start_stop.png)
855 Now that you've successfully installed a Windows client, please proceed to
856 the [detailed configuration modes](06-distributed-monitoring.md#distributed-monitoring-configuration-modes).
858 ## Configuration Modes <a id="distributed-monitoring-configuration-modes"></a>
860 There are different ways to ensure that the Icinga 2 cluster nodes execute
861 checks, send notifications, etc.
863 The preferred method is to configure monitoring objects on the master
864 and distribute the configuration to satellites and clients.
866 The following chapters will explain this in detail with hands-on manual configuration
867 examples. You should test and implement this once to fully understand how it works.
869 Once you are familiar with Icinga 2 and distributed monitoring, you
870 can start with additional integrations to manage and deploy your
873 * [Icinga Director](https://github.com/icinga/icingaweb2-module-director) provides a web interface to manage configuration and also allows to sync imported resources (CMDB, PuppetDB, etc.)
874 * [Ansible Roles](https://github.com/Icinga/icinga2-ansible)
875 * [Puppet Module](https://github.com/Icinga/puppet-icinga2)
876 * [Chef Cookbook](https://github.com/Icinga/chef-icinga2)
878 More details can be found [here](13-addons.md#configuration-tools).
880 ### Top Down <a id="distributed-monitoring-top-down"></a>
882 There are two different behaviors with check execution:
884 * Send a command execution event remotely: The scheduler still runs on the parent node.
885 * Sync the host/service objects directly to the child node: Checks are executed locally.
887 Again, technically it does not matter whether this is a `client` or a `satellite`
888 which is receiving configuration or command execution events.
890 ### Top Down Command Endpoint <a id="distributed-monitoring-top-down-command-endpoint"></a>
892 This mode will force the Icinga 2 node to execute commands remotely on a specified endpoint.
893 The host/service object configuration is located on the master/satellite and the client only
894 needs the CheckCommand object definitions being used there.
896 Every endpoint has its own remote check queue. The amount of checks executed simultaneously
897 can be limited on the endpoint with the `MaxConcurrentChecks` constant defined in [constants.conf](04-configuring-icinga-2.md#constants-conf). Icinga 2 may discard check requests,
898 if the remote check queue is full.
900 ![Icinga 2 Distributed Top Down Command Endpoint](images/distributed-monitoring/icinga2_distributed_top_down_command_endpoint.png)
904 * No local checks need to be defined on the child node (client).
905 * Light-weight remote check execution (asynchronous events).
906 * No [replay log](06-distributed-monitoring.md#distributed-monitoring-advanced-hints-command-endpoint-log-duration) is necessary for the child node.
907 * Pin checks to specific endpoints (if the child zone consists of 2 endpoints).
911 * If the child node is not connected, no more checks are executed.
912 * Requires additional configuration attribute specified in host/service objects.
913 * Requires local `CheckCommand` object configuration. Best practice is to use a [global config zone](06-distributed-monitoring.md#distributed-monitoring-global-zone-config-sync).
915 To make sure that all nodes involved will accept configuration and/or
916 commands, you need to configure the `Zone` and `Endpoint` hierarchy
919 * `icinga2-master1.localdomain` is the configuration master in this scenario.
920 * `icinga2-client1.localdomain` acts as client which receives command execution messages via command endpoint from the master. In addition, it receives the global check command configuration from the master.
922 Include the endpoint and zone configuration on **both** nodes in the file `/etc/icinga2/zones.conf`.
924 The endpoint configuration could look like this, for example:
926 [root@icinga2-client1.localdomain /]# vim /etc/icinga2/zones.conf
928 object Endpoint "icinga2-master1.localdomain" {
929 host = "192.168.56.101"
932 object Endpoint "icinga2-client1.localdomain" {
933 host = "192.168.56.111"
936 Next, you need to define two zones. There is no naming convention, best practice is to either use `master`, `satellite`/`client-fqdn` or to choose region names for example `Europe`, `USA` and `Asia`, though.
938 **Note**: Each client requires its own zone and endpoint configuration. Best practice
939 is to use the client's FQDN for all object names.
941 The `master` zone is a parent of the `icinga2-client1.localdomain` zone:
943 [root@icinga2-client1.localdomain /]# vim /etc/icinga2/zones.conf
945 object Zone "master" {
946 endpoints = [ "icinga2-master1.localdomain" ] //array with endpoint names
949 object Zone "icinga2-client1.localdomain" {
950 endpoints = [ "icinga2-client1.localdomain" ]
952 parent = "master" //establish zone hierarchy
955 You don't need any local configuration on the client except for
956 CheckCommand definitions which can be synced using the global zone
957 above. Therefore disable the inclusion of the `conf.d` directory
958 in `/etc/icinga2/icinga2.conf`.
961 [root@icinga2-client1.localdomain /]# vim /etc/icinga2/icinga2.conf
963 // Commented out, not required on a client as command endpoint
964 //include_recursive "conf.d"
969 > Packages >= 2.9 provide an option in the setup wizard to disable this.
970 > Defaults to disabled.
972 Now it is time to validate the configuration and to restart the Icinga 2 daemon
977 [root@icinga2-client1.localdomain /]# icinga2 daemon -C
978 [root@icinga2-client1.localdomain /]# systemctl restart icinga2
980 [root@icinga2-master1.localdomain /]# icinga2 daemon -C
981 [root@icinga2-master1.localdomain /]# systemctl restart icinga2
983 Once the clients have successfully connected, you are ready for the next step: **execute
984 a remote check on the client using the command endpoint**.
986 Include the host and service object configuration in the `master` zone
987 -- this will help adding a secondary master for high-availability later.
989 [root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/master
991 Add the host and service objects you want to monitor. There is
992 no limitation for files and directories -- best practice is to
995 By convention a master/satellite/client host object should use the same name as the endpoint object.
996 You can also add multiple hosts which execute checks against remote services/clients.
998 [root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/master
999 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim hosts.conf
1001 object Host "icinga2-client1.localdomain" {
1002 check_command = "hostalive" //check is executed on the master
1003 address = "192.168.56.111"
1005 vars.client_endpoint = name //follows the convention that host name == endpoint name
1008 Given that you are monitoring a Linux client, we'll add a remote [disk](10-icinga-template-library.md#plugin-check-command-disk)
1011 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim services.conf
1013 apply Service "disk" {
1014 check_command = "disk"
1016 //specify where the check is executed
1017 command_endpoint = host.vars.client_endpoint
1019 assign where host.vars.client_endpoint
1022 If you have your own custom `CheckCommand` definition, add it to the global zone:
1024 [root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/global-templates
1025 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/global-templates/commands.conf
1027 object CheckCommand "my-cmd" {
1031 Save the changes and validate the configuration on the master node:
1033 [root@icinga2-master1.localdomain /]# icinga2 daemon -C
1035 Restart the Icinga 2 daemon (example for CentOS 7):
1037 [root@icinga2-master1.localdomain /]# systemctl restart icinga2
1039 The following steps will happen:
1041 * Icinga 2 validates the configuration on `icinga2-master1.localdomain` and restarts.
1042 * The `icinga2-master1.localdomain` node schedules and executes the checks.
1043 * The `icinga2-client1.localdomain` node receives the execute command event with additional command parameters.
1044 * The `icinga2-client1.localdomain` node maps the command parameters to the local check command, executes the check locally, and sends back the check result message.
1046 As you can see, no interaction from your side is required on the client itself, and it's not necessary to reload the Icinga 2 service on the client.
1048 You have learned the basics about command endpoint checks. Proceed with
1049 the [scenarios](06-distributed-monitoring.md#distributed-monitoring-scenarios)
1050 section where you can find detailed information on extending the setup.
1053 ### Top Down Config Sync <a id="distributed-monitoring-top-down-config-sync"></a>
1055 This mode syncs the object configuration files within specified zones.
1056 It comes in handy if you want to configure everything on the master node
1057 and sync the satellite checks (disk, memory, etc.). The satellites run their
1058 own local scheduler and will send the check result messages back to the master.
1060 ![Icinga 2 Distributed Top Down Config Sync](images/distributed-monitoring/icinga2_distributed_top_down_config_sync.png)
1064 * Sync the configuration files from the parent zone to the child zones.
1065 * No manual restart is required on the child nodes, as syncing, validation, and restarts happen automatically.
1066 * Execute checks directly on the child node's scheduler.
1067 * Replay log if the connection drops (important for keeping the check history in sync, e.g. for SLA reports).
1068 * Use a global zone for syncing templates, groups, etc.
1072 * Requires a config directory on the master node with the zone name underneath `/etc/icinga2/zones.d`.
1073 * Additional zone and endpoint configuration needed.
1074 * Replay log is replicated on reconnect after connection loss. This might increase the data transfer and create an overload on the connection.
1076 To make sure that all involved nodes accept configuration and/or
1077 commands, you need to configure the `Zone` and `Endpoint` hierarchy
1080 * `icinga2-master1.localdomain` is the configuration master in this scenario.
1081 * `icinga2-client2.localdomain` acts as client which receives configuration from the master. Checks are scheduled locally.
1083 Include the endpoint and zone configuration on **both** nodes in the file `/etc/icinga2/zones.conf`.
1085 The endpoint configuration could look like this:
1087 [root@icinga2-client2.localdomain /]# vim /etc/icinga2/zones.conf
1089 object Endpoint "icinga2-master1.localdomain" {
1090 host = "192.168.56.101"
1093 object Endpoint "icinga2-client2.localdomain" {
1094 host = "192.168.56.112"
1097 Next, you need to define two zones. There is no naming convention, best practice is to either use `master`, `satellite`/`client-fqdn` or to choose region names for example `Europe`, `USA` and `Asia`, though.
1099 **Note**: Each client requires its own zone and endpoint configuration. Best practice
1100 is to use the client's FQDN for all object names.
1102 The `master` zone is a parent of the `icinga2-client2.localdomain` zone:
1104 [root@icinga2-client2.localdomain /]# vim /etc/icinga2/zones.conf
1106 object Zone "master" {
1107 endpoints = [ "icinga2-master1.localdomain" ] //array with endpoint names
1110 object Zone "icinga2-client2.localdomain" {
1111 endpoints = [ "icinga2-client2.localdomain" ]
1113 parent = "master" //establish zone hierarchy
1116 Edit the `api` feature on the client `icinga2-client2.localdomain` in
1117 the `/etc/icinga2/features-enabled/api.conf` file and set
1118 `accept_config` to `true`.
1120 [root@icinga2-client2.localdomain /]# vim /etc/icinga2/features-enabled/api.conf
1122 object ApiListener "api" {
1124 accept_config = true
1127 Now it is time to validate the configuration and to restart the Icinga 2 daemon
1130 Example on CentOS 7:
1132 [root@icinga2-client2.localdomain /]# icinga2 daemon -C
1133 [root@icinga2-client2.localdomain /]# systemctl restart icinga2
1135 [root@icinga2-master1.localdomain /]# icinga2 daemon -C
1136 [root@icinga2-master1.localdomain /]# systemctl restart icinga2
1139 **Tip**: Best practice is to use a [global zone](06-distributed-monitoring.md#distributed-monitoring-global-zone-config-sync)
1140 for common configuration items (check commands, templates, groups, etc.).
1142 Once the clients have connected successfully, it's time for the next step: **execute
1143 a local check on the client using the configuration sync**.
1145 Navigate to `/etc/icinga2/zones.d` on your master node
1146 `icinga2-master1.localdomain` and create a new directory with the same
1147 name as your satellite/client zone name:
1149 [root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/icinga2-client2.localdomain
1151 Add the host and service objects you want to monitor. There is
1152 no limitation for files and directories -- best practice is to
1153 sort things by type.
1155 By convention a master/satellite/client host object should use the same name as the endpoint object.
1156 You can also add multiple hosts which execute checks against remote services/clients.
1158 [root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/icinga2-client2.localdomain
1159 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/icinga2-client2.localdomain]# vim hosts.conf
1161 object Host "icinga2-client2.localdomain" {
1162 check_command = "hostalive"
1163 address = "192.168.56.112"
1164 zone = "master" //optional trick: sync the required host object to the client, but enforce the "master" zone to execute the check
1167 Given that you are monitoring a Linux client we'll just add a local [disk](10-icinga-template-library.md#plugin-check-command-disk)
1170 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/icinga2-client2.localdomain]# vim services.conf
1172 object Service "disk" {
1173 host_name = "icinga2-client2.localdomain"
1175 check_command = "disk"
1178 Save the changes and validate the configuration on the master node:
1180 [root@icinga2-master1.localdomain /]# icinga2 daemon -C
1182 Restart the Icinga 2 daemon (example for CentOS 7):
1184 [root@icinga2-master1.localdomain /]# systemctl restart icinga2
1186 The following steps will happen:
1188 * Icinga 2 validates the configuration on `icinga2-master1.localdomain`.
1189 * Icinga 2 copies the configuration into its zone config store in `/var/lib/icinga2/api/zones`.
1190 * The `icinga2-master1.localdomain` node sends a config update event to all endpoints in the same or direct child zones.
1191 * The `icinga2-client2.localdomain` node accepts config and populates the local zone config store with the received config files.
1192 * The `icinga2-client2.localdomain` node validates the configuration and automatically restarts.
1194 Again, there is no interaction required on the client
1197 You can also use the config sync inside a high-availability zone to
1198 ensure that all config objects are synced among zone members.
1200 **Note**: You can only have one so-called "config master" in a zone which stores
1201 the configuration in the `zones.d` directory.
1202 Multiple nodes with configuration files in the `zones.d` directory are
1205 Now that you've learned the basics about the configuration sync, proceed with
1206 the [scenarios](06-distributed-monitoring.md#distributed-monitoring-scenarios)
1207 section where you can find detailed information on extending the setup.
1211 If you are eager to start fresh instead you might take a look into the
1212 [Icinga Director](https://icinga.com/docs/director/latest/).
1214 ## Scenarios <a id="distributed-monitoring-scenarios"></a>
1216 The following examples should give you an idea on how to build your own
1217 distributed monitoring environment. We've seen them all in production
1218 environments and received feedback from our [community](https://icinga.com/community/)
1219 and [partner support](https://icinga.com/support/) channels:
1221 * Single master with clients.
1222 * HA master with clients as command endpoint.
1223 * Three level cluster with config HA masters, satellites receiving config sync, and clients checked using command endpoint.
1225 ### Master with Clients <a id="distributed-monitoring-master-clients"></a>
1227 ![Icinga 2 Distributed Master with Clients](images/distributed-monitoring/icinga2_distributed_scenarios_master_clients.png)
1229 * `icinga2-master1.localdomain` is the primary master node.
1230 * `icinga2-client1.localdomain` and `icinga2-client2.localdomain` are two child nodes as clients.
1234 * Set up `icinga2-master1.localdomain` as [master](06-distributed-monitoring.md#distributed-monitoring-setup-master).
1235 * Set up `icinga2-client1.localdomain` and `icinga2-client2.localdomain` as [client](06-distributed-monitoring.md#distributed-monitoring-setup-satellite-client).
1237 Edit the `zones.conf` configuration file on the master:
1239 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.conf
1241 object Endpoint "icinga2-master1.localdomain" {
1244 object Endpoint "icinga2-client1.localdomain" {
1245 host = "192.168.56.111" //the master actively tries to connect to the client
1248 object Endpoint "icinga2-client2.localdomain" {
1249 host = "192.168.56.112" //the master actively tries to connect to the client
1252 object Zone "master" {
1253 endpoints = [ "icinga2-master1.localdomain" ]
1256 object Zone "icinga2-client1.localdomain" {
1257 endpoints = [ "icinga2-client1.localdomain" ]
1262 object Zone "icinga2-client2.localdomain" {
1263 endpoints = [ "icinga2-client2.localdomain" ]
1268 /* sync global commands */
1269 object Zone "global-templates" {
1273 The two client nodes do not necessarily need to know about each other. The only important thing
1274 is that they know about the parent zone and their endpoint members (and optionally the global zone).
1276 If you specify the `host` attribute in the `icinga2-master1.localdomain` endpoint object,
1277 the client will actively try to connect to the master node. Since we've specified the client
1278 endpoint's attribute on the master node already, we don't want the clients to connect to the
1279 master. **Choose one [connection direction](06-distributed-monitoring.md#distributed-monitoring-advanced-hints-connection-direction).**
1281 [root@icinga2-client1.localdomain /]# vim /etc/icinga2/zones.conf
1283 object Endpoint "icinga2-master1.localdomain" {
1284 //do not actively connect to the master by leaving out the 'host' attribute
1287 object Endpoint "icinga2-client1.localdomain" {
1290 object Zone "master" {
1291 endpoints = [ "icinga2-master1.localdomain" ]
1294 object Zone "icinga2-client1.localdomain" {
1295 endpoints = [ "icinga2-client1.localdomain" ]
1300 /* sync global commands */
1301 object Zone "global-templates" {
1305 [root@icinga2-client2.localdomain /]# vim /etc/icinga2/zones.conf
1307 object Endpoint "icinga2-master1.localdomain" {
1308 //do not actively connect to the master by leaving out the 'host' attribute
1311 object Endpoint "icinga2-client2.localdomain" {
1314 object Zone "master" {
1315 endpoints = [ "icinga2-master1.localdomain" ]
1318 object Zone "icinga2-client2.localdomain" {
1319 endpoints = [ "icinga2-client2.localdomain" ]
1324 /* sync global commands */
1325 object Zone "global-templates" {
1329 Now it is time to define the two client hosts and apply service checks using
1330 the command endpoint execution method on them. Note: You can also use the
1331 config sync mode here.
1333 Create a new configuration directory on the master node:
1335 [root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/master
1337 Add the two client nodes as host objects:
1339 [root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/master
1340 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim hosts.conf
1342 object Host "icinga2-client1.localdomain" {
1343 check_command = "hostalive"
1344 address = "192.168.56.111"
1345 vars.client_endpoint = name //follows the convention that host name == endpoint name
1348 object Host "icinga2-client2.localdomain" {
1349 check_command = "hostalive"
1350 address = "192.168.56.112"
1351 vars.client_endpoint = name //follows the convention that host name == endpoint name
1354 Add services using command endpoint checks:
1356 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim services.conf
1358 apply Service "ping4" {
1359 check_command = "ping4"
1360 //check is executed on the master node
1361 assign where host.address
1364 apply Service "disk" {
1365 check_command = "disk"
1367 //specify where the check is executed
1368 command_endpoint = host.vars.client_endpoint
1370 assign where host.vars.client_endpoint
1373 Validate the configuration and restart Icinga 2 on the master node `icinga2-master1.localdomain`.
1375 [root@icinga2-master1.localdomain /]# icinga2 daemon -C
1376 [root@icinga2-master1.localdomain /]# systemctl restart icinga2
1378 Open Icinga Web 2 and check the two newly created client hosts with two new services
1379 -- one executed locally (`ping4`) and one using command endpoint (`disk`).
1381 ### High-Availability Master with Clients <a id="distributed-monitoring-scenarios-ha-master-clients"></a>
1383 ![Icinga 2 Distributed High Availability Master with Clients](images/distributed-monitoring/icinga2_distributed_scenarios_ha_master_clients.png)
1385 This scenario is similar to the one in the [previous section](06-distributed-monitoring.md#distributed-monitoring-master-clients). The only difference is that we will now set up two master nodes in a high-availability setup.
1386 These nodes must be configured as zone and endpoints objects.
1388 The setup uses the capabilities of the Icinga 2 cluster. All zone members
1389 replicate cluster events amongst each other. In addition to that, several Icinga 2
1390 features can enable HA functionality.
1392 **Note**: All nodes in the same zone require that you enable the same features for high-availability (HA).
1396 * `icinga2-master1.localdomain` is the config master master node.
1397 * `icinga2-master2.localdomain` is the secondary master master node without config in `zones.d`.
1398 * `icinga2-client1.localdomain` and `icinga2-client2.localdomain` are two child nodes as clients.
1402 * Set up `icinga2-master1.localdomain` as [master](06-distributed-monitoring.md#distributed-monitoring-setup-master).
1403 * Set up `icinga2-master2.localdomain` as [client](06-distributed-monitoring.md#distributed-monitoring-setup-satellite-client) (we will modify the generated configuration).
1404 * Set up `icinga2-client1.localdomain` and `icinga2-client2.localdomain` as [clients](06-distributed-monitoring.md#distributed-monitoring-setup-satellite-client) (when asked for adding multiple masters, set to `y` and add the secondary master `icinga2-master2.localdomain`).
1406 In case you don't want to use the CLI commands, you can also manually create and sync the
1407 required SSL certificates. We will modify and discuss all the details of the automatically generated configuration here.
1409 Since there are now two nodes in the same zone, we must consider the
1410 [high-availability features](06-distributed-monitoring.md#distributed-monitoring-high-availability-features).
1412 * Checks and notifications are balanced between the two master nodes. That's fine, but it requires check plugins and notification scripts to exist on both nodes.
1413 * The IDO feature will only be active on one node by default. Since all events are replicated between both nodes, it is easier to just have one central database.
1415 One possibility is to use a dedicated MySQL cluster VIP (external application cluster)
1416 and leave the IDO feature with enabled HA capabilities. Alternatively,
1417 you can disable the HA feature and write to a local database on each node.
1418 Both methods require that you configure Icinga Web 2 accordingly (monitoring
1419 backend, IDO database, used transports, etc.).
1421 The zone hierarchy could look like this. It involves putting the two master nodes
1422 `icinga2-master1.localdomain` and `icinga2-master2.localdomain` into the `master` zone.
1424 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.conf
1426 object Endpoint "icinga2-master1.localdomain" {
1427 host = "192.168.56.101"
1430 object Endpoint "icinga2-master2.localdomain" {
1431 host = "192.168.56.102"
1434 object Endpoint "icinga2-client1.localdomain" {
1435 host = "192.168.56.111" //the master actively tries to connect to the client
1438 object Endpoint "icinga2-client2.localdomain" {
1439 host = "192.168.56.112" //the master actively tries to connect to the client
1442 object Zone "master" {
1443 endpoints = [ "icinga2-master1.localdomain", "icinga2-master2.localdomain" ]
1446 object Zone "icinga2-client1.localdomain" {
1447 endpoints = [ "icinga2-client1.localdomain" ]
1452 object Zone "icinga2-client2.localdomain" {
1453 endpoints = [ "icinga2-client2.localdomain" ]
1458 /* sync global commands */
1459 object Zone "global-templates" {
1463 The two client nodes do not necessarily need to know about each other. The only important thing
1464 is that they know about the parent zone and their endpoint members (and optionally about the global zone).
1466 If you specify the `host` attribute in the `icinga2-master1.localdomain` and `icinga2-master2.localdomain`
1467 endpoint objects, the client will actively try to connect to the master node. Since we've specified the client
1468 endpoint's attribute on the master node already, we don't want the clients to connect to the
1469 master nodes. **Choose one [connection direction](06-distributed-monitoring.md#distributed-monitoring-advanced-hints-connection-direction).**
1471 [root@icinga2-client1.localdomain /]# vim /etc/icinga2/zones.conf
1473 object Endpoint "icinga2-master1.localdomain" {
1474 //do not actively connect to the master by leaving out the 'host' attribute
1477 object Endpoint "icinga2-master2.localdomain" {
1478 //do not actively connect to the master by leaving out the 'host' attribute
1481 object Endpoint "icinga2-client1.localdomain" {
1484 object Zone "master" {
1485 endpoints = [ "icinga2-master1.localdomain", "icinga2-master2.localdomain" ]
1488 object Zone "icinga2-client1.localdomain" {
1489 endpoints = [ "icinga2-client1.localdomain" ]
1494 /* sync global commands */
1495 object Zone "global-templates" {
1499 [root@icinga2-client2.localdomain /]# vim /etc/icinga2/zones.conf
1501 object Endpoint "icinga2-master1.localdomain" {
1502 //do not actively connect to the master by leaving out the 'host' attribute
1505 object Endpoint "icinga2-master2.localdomain" {
1506 //do not actively connect to the master by leaving out the 'host' attribute
1509 object Endpoint "icinga2-client2.localdomain" {
1512 object Zone "master" {
1513 endpoints = [ "icinga2-master1.localdomain", "icinga2-master2.localdomain" ]
1516 object Zone "icinga2-client2.localdomain" {
1517 endpoints = [ "icinga2-client2.localdomain" ]
1522 /* sync global commands */
1523 object Zone "global-templates" {
1527 Now it is time to define the two client hosts and apply service checks using
1528 the command endpoint execution method. Note: You can also use the
1529 config sync mode here.
1531 Create a new configuration directory on the master node `icinga2-master1.localdomain`.
1532 **Note**: The secondary master node `icinga2-master2.localdomain` receives the
1533 configuration using the [config sync mode](06-distributed-monitoring.md#distributed-monitoring-top-down-config-sync).
1535 [root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/master
1537 Add the two client nodes as host objects:
1539 [root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/master
1540 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim hosts.conf
1542 object Host "icinga2-client1.localdomain" {
1543 check_command = "hostalive"
1544 address = "192.168.56.111"
1545 vars.client_endpoint = name //follows the convention that host name == endpoint name
1548 object Host "icinga2-client2.localdomain" {
1549 check_command = "hostalive"
1550 address = "192.168.56.112"
1551 vars.client_endpoint = name //follows the convention that host name == endpoint name
1554 Add services using command endpoint checks:
1556 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim services.conf
1558 apply Service "ping4" {
1559 check_command = "ping4"
1560 //check is executed on the master node
1561 assign where host.address
1564 apply Service "disk" {
1565 check_command = "disk"
1567 //specify where the check is executed
1568 command_endpoint = host.vars.client_endpoint
1570 assign where host.vars.client_endpoint
1573 Validate the configuration and restart Icinga 2 on the master node `icinga2-master1.localdomain`.
1575 [root@icinga2-master1.localdomain /]# icinga2 daemon -C
1576 [root@icinga2-master1.localdomain /]# systemctl restart icinga2
1578 Open Icinga Web 2 and check the two newly created client hosts with two new services
1579 -- one executed locally (`ping4`) and one using command endpoint (`disk`).
1581 **Tip**: It's a good idea to add [health checks](06-distributed-monitoring.md#distributed-monitoring-health-checks)
1582 to make sure that your cluster notifies you in case of failure.
1585 ### Three Levels with Master, Satellites, and Clients <a id="distributed-monitoring-scenarios-master-satellite-client"></a>
1587 ![Icinga 2 Distributed Master and Satellites with Clients](images/distributed-monitoring/icinga2_distributed_scenarios_master_satellite_client.png)
1589 This scenario combines everything you've learned so far: High-availability masters,
1590 satellites receiving their configuration from the master zone, and clients checked via command
1591 endpoint from the satellite zones.
1595 > It can get complicated, so grab a pen and paper and bring your thoughts to life.
1596 > Play around with a test setup before using it in a production environment!
1600 * `icinga2-master1.localdomain` is the configuration master master node.
1601 * `icinga2-master2.localdomain` is the secondary master master node without configuration in `zones.d`.
1602 * `icinga2-satellite1.localdomain` and `icinga2-satellite2.localdomain` are satellite nodes in a `master` child zone. They forward CSR signing requests to the master zone.
1603 * `icinga2-client1.localdomain` and `icinga2-client2.localdomain` are two child nodes as clients.
1607 * Set up `icinga2-master1.localdomain` as [master](06-distributed-monitoring.md#distributed-monitoring-setup-master).
1608 * Set up `icinga2-master2.localdomain`, `icinga2-satellite1.localdomain` and `icinga2-satellite2.localdomain` as [clients](06-distributed-monitoring.md#distributed-monitoring-setup-satellite-client) (we will modify the generated configuration).
1609 * Set up `icinga2-client1.localdomain` and `icinga2-client2.localdomain` as [clients](06-distributed-monitoring.md#distributed-monitoring-setup-satellite-client).
1611 When being asked for the parent endpoint providing CSR auto-signing capabilities,
1612 please add one of the satellite nodes. **Note**: This requires Icinga 2 v2.8+
1613 and the `CA Proxy` on all master, satellite and client nodes.
1615 Example for `icinga2-client1.localdomain`:
1618 Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
1621 Parent endpoint is the first satellite `icinga2-satellite1.localdomain`:
1624 Master/Satellite Common Name (CN from your master/satellite node): icinga2-satellite1.localdomain
1625 Do you want to establish a connection to the parent node from this node? [Y/n]: y
1627 Please specify the master/satellite connection information:
1628 Master/Satellite endpoint host (IP address or FQDN): 192.168.56.105
1629 Master/Satellite endpoint port [5665]: 5665
1632 Add the second satellite `icinga2-satellite2.localdomain` as parent:
1635 Add more master/satellite endpoints? [y/N]: y
1637 Master/Satellite Common Name (CN from your master/satellite node): icinga2-satellite2.localdomain
1638 Do you want to establish a connection to the parent node from this node? [Y/n]: y
1640 Please specify the master/satellite connection information:
1641 Master/Satellite endpoint host (IP address or FQDN): 192.168.56.106
1642 Master/Satellite endpoint port [5665]: 5665
1644 Add more master/satellite endpoints? [y/N]: n
1647 The specified parent nodes will forward the CSR signing request to the master instances.
1649 Proceed with adding the optional client ticket for [CSR auto-signing](06-distributed-monitoring.md#distributed-monitoring-setup-csr-auto-signing):
1652 Please specify the request ticket generated on your Icinga 2 master (optional).
1653 (Hint: # icinga2 pki ticket --cn 'icinga2-client1.localdomain'):
1654 4f75d2ecd253575fe9180938ebff7cbca262f96e
1657 In case you've chosen to use [On-Demand CSR Signing](06-distributed-monitoring.md#distributed-monitoring-setup-on-demand-csr-signing)
1658 you can leave the ticket question blank.
1660 Instead, Icinga 2 tells you to approve the request later on the master node.
1663 No ticket was specified. Please approve the certificate signing request manually
1664 on the master (see 'icinga2 ca list' and 'icinga2 ca sign --help' for details).
1667 You can optionally specify a different bind host and/or port.
1670 Please specify the API bind host/port (optional):
1675 The next step asks you to accept configuration (required for [config sync mode](06-distributed-monitoring.md#distributed-monitoring-top-down-config-sync))
1676 and commands (required for [command endpoint mode](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint)).
1679 Accept config from parent node? [y/N]: y
1680 Accept commands from parent node? [y/N]: y
1683 Next you can optionally specify the local and parent zone names. This will be reflected
1684 in the generated zone configuration file.
1687 Local zone name [icinga2-client1.localdomain]: icinga2-client1.localdomain
1690 Set the parent zone name to `satellite` for this client.
1693 Parent zone name [master]: satellite
1696 You can add more global zones in addition to `global-templates` and `director-global` if necessary.
1697 Press `Enter` or choose `n`, if you don't want to add any additional.
1700 Reconfiguring Icinga...
1702 Default global zones: global-templates director-global
1703 Do you want to specify additional global zones? [y/N]: N
1706 Last but not least the wizard asks you whether you want to disable the inclusion of the local configuration
1707 directory in `conf.d`, or not. Defaults to disabled, as clients either are checked via command endpoint, or
1708 they receive configuration synced from the parent zone.
1711 Do you want to disable the inclusion of the conf.d directory [Y/n]: Y
1712 Disabling the inclusion of the conf.d directory...
1716 **We'll discuss the details of the required configuration below. Most of this
1717 configuration can be rendered by the setup wizards.**
1719 The zone hierarchy can look like this. We'll define only the directly connected zones here.
1721 The master instances should actively connect to the satellite instances, therefore
1722 the configuration on `icinga2-master1.localdomain` and `icinga2-master2.localdomain`
1723 must include the `host` attribute for the satellite endpoints:
1726 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.conf
1728 object Endpoint "icinga2-master1.localdomain" {
1732 object Endpoint "icinga2-master2.localdomain" {
1733 host = "192.168.56.102"
1736 object Endpoint "icinga2-satellite1.localdomain" {
1737 host = "192.168.56.105"
1740 object Endpoint "icinga2-satellite2.localdomain" {
1741 host = "192.168.56.106"
1744 object Zone "master" {
1745 endpoints = [ "icinga2-master1.localdomain", "icinga2-master2.localdomain" ]
1748 object Zone "satellite" {
1749 endpoints = [ "icinga2-satellite1.localdomain", "icinga2-satellite2.localdomain" ]
1754 /* sync global commands */
1755 object Zone "global-templates" {
1759 object Zone "director-global" {
1765 In contrast to that, the satellite instances `icinga2-satellite1.localdomain`
1766 and `icinga2-satellite2.localdomain` should not actively connect to the master
1770 [root@icinga2-satellite1.localdomain /]# vim /etc/icinga2/zones.conf
1772 object Endpoint "icinga2-master1.localdomain" {
1773 //this endpoint will connect to us
1776 object Endpoint "icinga2-master2.localdomain" {
1777 //this endpoint will connect to us
1780 object Endpoint "icinga2-satellite1.localdomain" {
1784 object Endpoint "icinga2-satellite2.localdomain" {
1785 host = "192.168.56.106"
1788 object Zone "master" {
1789 endpoints = [ "icinga2-master1.localdomain", "icinga2-master2.localdomain" ]
1792 object Zone "satellite" {
1793 endpoints = [ "icinga2-satellite1.localdomain", "icinga2-satellite2.localdomain" ]
1798 /* sync global commands */
1799 object Zone "global-templates" {
1803 object Zone "director-global" {
1807 Keep in mind to control the endpoint [connection direction](06-distributed-monitoring.md#distributed-monitoring-advanced-hints-connection-direction)
1808 using the `host` attribute, also for other endpoints in the same zone.
1810 Adopt the configuration for `icinga2-master2.localdomain` and `icinga2-satellite2.localdomain`.
1812 Since we want to use [top down command endpoint](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint) checks,
1813 we must configure the client endpoint and zone objects.
1814 In order to minimize the effort, we'll sync the client zone and endpoint configuration to the
1815 satellites where the connection information is needed as well.
1817 [root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/{master,satellite,global-templates}
1818 [root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/satellite
1820 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim icinga2-client1.localdomain.conf
1822 object Endpoint "icinga2-client1.localdomain" {
1823 host = "192.168.56.111" //the satellite actively tries to connect to the client
1826 object Zone "icinga2-client1.localdomain" {
1827 endpoints = [ "icinga2-client1.localdomain" ]
1829 parent = "satellite"
1832 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim icinga2-client2.localdomain.conf
1834 object Endpoint "icinga2-client2.localdomain" {
1835 host = "192.168.56.112" //the satellite actively tries to connect to the client
1838 object Zone "icinga2-client2.localdomain" {
1839 endpoints = [ "icinga2-client2.localdomain" ]
1841 parent = "satellite"
1844 The two client nodes do not necessarily need to know about each other, either. The only important thing
1845 is that they know about the parent zone (the satellite) and their endpoint members (and optionally the global zone).
1847 If you specify the `host` attribute in the `icinga2-satellite1.localdomain` and `icinga2-satellite2.localdomain`
1848 endpoint objects, the client node will actively try to connect to the satellite node. Since we've specified the client
1849 endpoint's attribute on the satellite node already, we don't want the client node to connect to the
1850 satellite nodes. **Choose one [connection direction](06-distributed-monitoring.md#distributed-monitoring-advanced-hints-connection-direction).**
1852 Example for `icinga2-client1.localdomain`:
1854 [root@icinga2-client1.localdomain /]# vim /etc/icinga2/zones.conf
1856 object Endpoint "icinga2-satellite1.localdomain" {
1857 //do not actively connect to the satellite by leaving out the 'host' attribute
1860 object Endpoint "icinga2-satellite2.localdomain" {
1861 //do not actively connect to the satellite by leaving out the 'host' attribute
1864 object Endpoint "icinga2-client1.localdomain" {
1868 object Zone "satellite" {
1869 endpoints = [ "icinga2-satellite1.localdomain", "icinga2-satellite2.localdomain" ]
1872 object Zone "icinga2-client1.localdomain" {
1873 endpoints = [ "icinga2-client1.localdomain" ]
1875 parent = "satellite"
1878 /* sync global commands */
1879 object Zone "global-templates" {
1883 object Zone "director-global" {
1887 Example for `icinga2-client2.localdomain`:
1889 [root@icinga2-client2.localdomain /]# vim /etc/icinga2/zones.conf
1891 object Endpoint "icinga2-satellite1.localdomain" {
1892 //do not actively connect to the satellite by leaving out the 'host' attribute
1895 object Endpoint "icinga2-satellite2.localdomain" {
1896 //do not actively connect to the satellite by leaving out the 'host' attribute
1899 object Endpoint "icinga2-client2.localdomain" {
1903 object Zone "satellite" {
1904 endpoints = [ "icinga2-satellite1.localdomain", "icinga2-satellite2.localdomain" ]
1907 object Zone "icinga2-client2.localdomain" {
1908 endpoints = [ "icinga2-client2.localdomain" ]
1910 parent = "satellite"
1913 /* sync global commands */
1914 object Zone "global-templates" {
1918 object Zone "director-global" {
1922 Now it is time to define the two client hosts on the master, sync them to the satellites
1923 and apply service checks using the command endpoint execution method to them.
1924 Add the two client nodes as host objects to the `satellite` zone.
1926 We've already created the directories in `/etc/icinga2/zones.d` including the files for the
1927 zone and endpoint configuration for the clients.
1929 [root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/satellite
1931 Add the host object configuration for the `icinga2-client1.localdomain` client. You should
1932 have created the configuration file in the previous steps and it should contain the endpoint
1933 and zone object configuration already.
1935 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim icinga2-client1.localdomain.conf
1937 object Host "icinga2-client1.localdomain" {
1938 check_command = "hostalive"
1939 address = "192.168.56.111"
1940 vars.client_endpoint = name //follows the convention that host name == endpoint name
1943 Add the host object configuration for the `icinga2-client2.localdomain` client configuration file:
1945 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim icinga2-client2.localdomain.conf
1947 object Host "icinga2-client2.localdomain" {
1948 check_command = "hostalive"
1949 address = "192.168.56.112"
1950 vars.client_endpoint = name //follows the convention that host name == endpoint name
1953 Add a service object which is executed on the satellite nodes (e.g. `ping4`). Pin the apply rule to the `satellite` zone only.
1955 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim services.conf
1957 apply Service "ping4" {
1958 check_command = "ping4"
1959 //check is executed on the satellite node
1960 assign where host.zone == "satellite" && host.address
1963 Add services using command endpoint checks. Pin the apply rules to the `satellite` zone only.
1965 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim services.conf
1967 apply Service "disk" {
1968 check_command = "disk"
1970 //specify where the check is executed
1971 command_endpoint = host.vars.client_endpoint
1973 assign where host.zone == "satellite" && host.vars.client_endpoint
1976 Validate the configuration and restart Icinga 2 on the master node `icinga2-master1.localdomain`.
1978 [root@icinga2-master1.localdomain /]# icinga2 daemon -C
1979 [root@icinga2-master1.localdomain /]# systemctl restart icinga2
1981 Open Icinga Web 2 and check the two newly created client hosts with two new services
1982 -- one executed locally (`ping4`) and one using command endpoint (`disk`).
1986 > It's a good idea to add [health checks](06-distributed-monitoring.md#distributed-monitoring-health-checks)
1987 to make sure that your cluster notifies you in case of failure.
1989 ## Best Practice <a id="distributed-monitoring-best-practice"></a>
1991 We've put together a collection of configuration examples from community feedback.
1992 If you like to share your tips and tricks with us, please join the [community channels](https://icinga.com/community/)!
1994 ### Global Zone for Config Sync <a id="distributed-monitoring-global-zone-config-sync"></a>
1996 Global zones can be used to sync generic configuration objects
1997 to all nodes depending on them. Common examples are:
1999 * Templates which are imported into zone specific objects.
2000 * Command objects referenced by Host, Service, Notification objects.
2001 * Apply rules for services, notifications and dependencies.
2002 * User objects referenced in notifications.
2004 * TimePeriod objects.
2006 Plugin scripts and binaries cannot be synced, this is for Icinga 2
2007 configuration files only. Use your preferred package repository
2008 and/or configuration management tool (Puppet, Ansible, Chef, etc.)
2011 **Note**: Checkable objects (hosts and services) cannot be put into a global
2012 zone. The configuration validation will terminate with an error.
2014 The zone object configuration must be deployed on all nodes which should receive
2015 the global configuration files:
2017 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.conf
2019 object Zone "global-commands" {
2023 The default global zones generated by the setup wizards are called `global-templates` and `director-global`.
2025 Similar to the zone configuration sync you'll need to create a new directory in
2026 `/etc/icinga2/zones.d`:
2028 [root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/global-commands
2030 Next, add a new check command, for example:
2032 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/global-commands/web.conf
2034 object CheckCommand "webinject" {
2038 Restart the client(s) which should receive the global zone before
2039 before restarting the parent master/satellite nodes.
2041 Then validate the configuration on the master node and restart Icinga 2.
2043 **Tip**: You can copy the example configuration files located in `/etc/icinga2/conf.d`
2044 into the default global zone `global-templates`.
2048 [root@icinga2-master1.localdomain /]# cd /etc/icinga2/conf.d
2049 [root@icinga2-master1.localdomain /etc/icinga2/conf.d]# cp {commands,groups,notifications,services,templates,timeperiods,users}.conf /etc/icinga2/zones.d/global-templates
2051 ### Health Checks <a id="distributed-monitoring-health-checks"></a>
2053 In case of network failures or other problems, your monitoring might
2054 either have late check results or just send out mass alarms for unknown
2057 In order to minimize the problems caused by this, you should configure
2058 additional health checks.
2060 The `cluster` check, for example, will check if all endpoints in the current zone and the directly
2061 connected zones are working properly:
2063 [root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/master
2064 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/icinga2-master1.localdomain.conf
2066 object Host "icinga2-master1.localdomain" {
2067 check_command = "hostalive"
2068 address = "192.168.56.101"
2071 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/cluster.conf
2073 object Service "cluster" {
2074 check_command = "cluster"
2078 host_name = "icinga2-master1.localdomain"
2081 The `cluster-zone` check will test whether the configured target zone is currently
2082 connected or not. This example adds a health check for the [ha master with clients scenario](06-distributed-monitoring.md#distributed-monitoring-scenarios-ha-master-clients).
2084 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/services.conf
2086 apply Service "cluster-health" {
2087 check_command = "cluster-zone"
2089 display_name = "cluster-health-" + host.name
2091 /* This follows the convention that the client zone name is the FQDN which is the same as the host object name. */
2092 vars.cluster_zone = host.name
2094 assign where host.vars.client_endpoint
2097 In case you cannot assign the `cluster_zone` attribute, add specific
2098 checks to your cluster:
2100 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/cluster.conf
2102 object Service "cluster-zone-satellite" {
2103 check_command = "cluster-zone"
2106 vars.cluster_zone = "satellite"
2108 host_name = "icinga2-master1.localdomain"
2112 If you are using top down checks with command endpoint configuration, you can
2113 add a dependency which prevents notifications for all other failing services:
2115 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/dependencies.conf
2117 apply Dependency "health-check" to Service {
2118 parent_service_name = "child-health"
2121 disable_notifications = true
2123 assign where host.vars.client_endpoint
2124 ignore where service.name == "child-health"
2127 ### Pin Checks in a Zone <a id="distributed-monitoring-pin-checks-zone"></a>
2129 In case you want to pin specific checks to their endpoints in a given zone you'll need to use
2130 the `command_endpoint` attribute. This is reasonable if you want to
2131 execute a local disk check in the `master` Zone on a specific endpoint then.
2133 [root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/master
2134 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/icinga2-master1.localdomain.conf
2136 object Host "icinga2-master1.localdomain" {
2137 check_command = "hostalive"
2138 address = "192.168.56.101"
2141 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/services.conf
2143 apply Service "disk" {
2144 check_command = "disk"
2146 command_endpoint = host.name //requires a host object matching the endpoint object name e.g. icinga2-master1.localdomain
2148 assign where host.zone == "master" && match("icinga2-master*", host.name)
2151 The `host.zone` attribute check inside the expression ensures that
2152 the service object is only created for host objects inside the `master`
2153 zone. In addition to that the [match](18-library-reference.md#global-functions-match)
2154 function ensures to only create services for the master nodes.
2156 ### Windows Firewall <a id="distributed-monitoring-windows-firewall"></a>
2158 #### ICMP Requests <a id="distributed-monitoring-windows-firewall-icmp"></a>
2160 By default ICMP requests are disabled in the Windows firewall. You can
2161 change that by [adding a new rule](https://support.microsoft.com/en-us/kb/947709).
2163 C:\WINDOWS\system32>netsh advfirewall firewall add rule name="ICMP Allow incoming V4 echo request" protocol=icmpv4:8,any dir=in action=allow
2165 #### Icinga 2 <a id="distributed-monitoring-windows-firewall-icinga2"></a>
2167 If your master/satellite nodes should actively connect to the Windows client
2168 you'll also need to ensure that port `5665` is enabled.
2170 C:\WINDOWS\system32>netsh advfirewall firewall add rule name="Open port 5665 (Icinga 2)" dir=in action=allow protocol=TCP localport=5665
2172 #### NSClient++ API <a id="distributed-monitoring-windows-firewall-nsclient-api"></a>
2174 If the [check_nscp_api](06-distributed-monitoring.md#distributed-monitoring-windows-nscp-check-api)
2175 plugin is used to query NSClient++, you need to ensure that its port is enabled.
2177 C:\WINDOWS\system32>netsh advfirewall firewall add rule name="Open port 8443 (NSClient++ API)" dir=in action=allow protocol=TCP localport=8443
2179 For security reasons, it is advised to enable the NSClient++ HTTP API for local
2180 connection from the Icinga 2 client only. Remote connections to the HTTP API
2181 are not recommended with using the legacy HTTP API.
2183 ### Windows Client and Plugins <a id="distributed-monitoring-windows-plugins"></a>
2185 The Icinga 2 package on Windows already provides several plugins.
2186 Detailed [documentation](10-icinga-template-library.md#windows-plugins) is available for all check command definitions.
2188 Add the following `include` statement on all your nodes (master, satellite, client):
2190 vim /etc/icinga2/icinga2.conf
2192 include <windows-plugins>
2194 Based on the [master with clients](06-distributed-monitoring.md#distributed-monitoring-master-clients)
2195 scenario we'll now add a local disk check.
2197 First, add the client node as host object:
2199 [root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/master
2200 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim hosts.conf
2202 object Host "icinga2-client2.localdomain" {
2203 check_command = "hostalive"
2204 address = "192.168.56.112"
2205 vars.client_endpoint = name //follows the convention that host name == endpoint name
2206 vars.os_type = "windows"
2209 Next, add the disk check using command endpoint checks (details in the
2210 [disk-windows](10-icinga-template-library.md#windows-plugins-disk-windows) documentation):
2212 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim services.conf
2214 apply Service "disk C:" {
2215 check_command = "disk-windows"
2217 vars.disk_win_path = "C:"
2219 //specify where the check is executed
2220 command_endpoint = host.vars.client_endpoint
2222 assign where host.vars.os_type == "windows" && host.vars.client_endpoint
2225 Validate the configuration and restart Icinga 2.
2227 [root@icinga2-master1.localdomain /]# icinga2 daemon -C
2228 [root@icinga2-master1.localdomain /]# systemctl restart icinga2
2230 Open Icinga Web 2 and check your newly added Windows disk check :)
2232 ![Icinga 2 Client Windows](images/distributed-monitoring/icinga2_distributed_windows_client_disk_icingaweb2.png)
2234 If you want to add your own plugins please check [this chapter](05-service-monitoring.md#service-monitoring-requirements)
2235 for the requirements.
2237 ### Windows Client and NSClient++ <a id="distributed-monitoring-windows-nscp"></a>
2239 There are two methods available for querying NSClient++:
2241 * Query the [HTTP API](06-distributed-monitoring.md#distributed-monitoring-windows-nscp-check-api) locally from an Icinga 2 client (requires a running NSClient++ service)
2242 * Run a [local CLI check](06-distributed-monitoring.md#distributed-monitoring-windows-nscp-check-local) (does not require NSClient++ as a service)
2244 Both methods have their advantages and disadvantages. One thing to
2245 note: If you rely on performance counter delta calculations such as
2246 CPU utilization, please use the HTTP API instead of the CLI sample call.
2248 #### NSCLient++ with check_nscp_api <a id="distributed-monitoring-windows-nscp-check-api"></a>
2250 The [Windows setup](06-distributed-monitoring.md#distributed-monitoring-setup-client-windows) already allows
2251 you to install the NSClient++ package. In addition to the Windows plugins you can
2252 use the [nscp_api command](10-icinga-template-library.md#nscp-check-api) provided by the Icinga Template Library (ITL).
2254 The initial setup for the NSClient++ API and the required arguments
2255 is the described in the ITL chapter for the [nscp_api](10-icinga-template-library.md#nscp-check-api) CheckCommand.
2257 Based on the [master with clients](06-distributed-monitoring.md#distributed-monitoring-master-clients)
2258 scenario we'll now add a local nscp check which queries the NSClient++ API to check the free disk space.
2260 Define a host object called `icinga2-client2.localdomain` on the master. Add the `nscp_api_password`
2261 custom attribute and specify the drives to check.
2263 [root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/master
2264 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim hosts.conf
2266 object Host "icinga2-client1.localdomain" {
2267 check_command = "hostalive"
2268 address = "192.168.56.111"
2269 vars.client_endpoint = name //follows the convention that host name == endpoint name
2270 vars.os_type = "Windows"
2271 vars.nscp_api_password = "icinga"
2272 vars.drives = [ "C:", "D:" ]
2275 The service checks are generated using an [apply for](03-monitoring-basics.md#using-apply-for)
2276 rule based on `host.vars.drives`:
2278 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim services.conf
2280 apply Service "nscp-api-" for (drive in host.vars.drives) {
2281 import "generic-service"
2283 check_command = "nscp_api"
2284 command_endpoint = host.vars.client_endpoint
2286 //display_name = "nscp-drive-" + drive
2288 vars.nscp_api_host = "localhost"
2289 vars.nscp_api_query = "check_drivesize"
2290 vars.nscp_api_password = host.vars.nscp_api_password
2291 vars.nscp_api_arguments = [ "drive=" + drive ]
2293 ignore where host.vars.os_type != "Windows"
2296 Validate the configuration and restart Icinga 2.
2298 [root@icinga2-master1.localdomain /]# icinga2 daemon -C
2299 [root@icinga2-master1.localdomain /]# systemctl restart icinga2
2301 Two new services ("nscp-drive-D:" and "nscp-drive-C:") will be visible in Icinga Web 2.
2303 ![Icinga 2 Distributed Monitoring Windows Client with NSClient++ nscp-api](images/distributed-monitoring/icinga2_distributed_windows_nscp_api_drivesize_icingaweb2.png)
2305 Note: You can also omit the `command_endpoint` configuration to execute
2306 the command on the master. This also requires a different value for `nscp_api_host`
2307 which defaults to `host.address`.
2309 //command_endpoint = host.vars.client_endpoint
2311 //vars.nscp_api_host = "localhost"
2313 You can verify the check execution by looking at the `Check Source` attribute
2314 in Icinga Web 2 or the REST API.
2316 If you want to monitor specific Windows services, you could use the following example:
2318 [root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/master
2319 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim hosts.conf
2321 object Host "icinga2-client1.localdomain" {
2322 check_command = "hostalive"
2323 address = "192.168.56.111"
2324 vars.client_endpoint = name //follows the convention that host name == endpoint name
2325 vars.os_type = "Windows"
2326 vars.nscp_api_password = "icinga"
2327 vars.services = [ "Windows Update", "wscsvc" ]
2330 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim services.conf
2332 apply Service "nscp-api-" for (svc in host.vars.services) {
2333 import "generic-service"
2335 check_command = "nscp_api"
2336 command_endpoint = host.vars.client_endpoint
2338 //display_name = "nscp-service-" + svc
2340 vars.nscp_api_host = "localhost"
2341 vars.nscp_api_query = "check_service"
2342 vars.nscp_api_password = host.vars.nscp_api_password
2343 vars.nscp_api_arguments = [ "service=" + svc ]
2345 ignore where host.vars.os_type != "Windows"
2348 #### NSCLient++ with nscp-local <a id="distributed-monitoring-windows-nscp-check-local"></a>
2350 The [Windows setup](06-distributed-monitoring.md#distributed-monitoring-setup-client-windows) already allows
2351 you to install the NSClient++ package. In addition to the Windows plugins you can
2352 use the [nscp-local commands](10-icinga-template-library.md#nscp-plugin-check-commands)
2353 provided by the Icinga Template Library (ITL).
2355 ![Icinga 2 Distributed Monitoring Windows Client with NSClient++](images/distributed-monitoring/icinga2_distributed_windows_nscp.png)
2357 Add the following `include` statement on all your nodes (master, satellite, client):
2359 vim /etc/icinga2/icinga2.conf
2363 The CheckCommand definitions will automatically determine the installed path
2364 to the `nscp.exe` binary.
2366 Based on the [master with clients](06-distributed-monitoring.md#distributed-monitoring-master-clients)
2367 scenario we'll now add a local nscp check querying a given performance counter.
2369 First, add the client node as host object:
2371 [root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/master
2372 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim hosts.conf
2374 object Host "icinga2-client1.localdomain" {
2375 check_command = "hostalive"
2376 address = "192.168.56.111"
2377 vars.client_endpoint = name //follows the convention that host name == endpoint name
2378 vars.os_type = "windows"
2381 Next, add a performance counter check using command endpoint checks (details in the
2382 [nscp-local-counter](10-icinga-template-library.md#nscp-check-local-counter) documentation):
2384 [root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim services.conf
2386 apply Service "nscp-local-counter-cpu" {
2387 check_command = "nscp-local-counter"
2388 command_endpoint = host.vars.client_endpoint
2390 vars.nscp_counter_name = "\\Processor(_total)\\% Processor Time"
2391 vars.nscp_counter_perfsyntax = "Total Processor Time"
2392 vars.nscp_counter_warning = 1
2393 vars.nscp_counter_critical = 5
2395 vars.nscp_counter_showall = true
2397 assign where host.vars.os_type == "windows" && host.vars.client_endpoint
2400 Validate the configuration and restart Icinga 2.
2402 [root@icinga2-master1.localdomain /]# icinga2 daemon -C
2403 [root@icinga2-master1.localdomain /]# systemctl restart icinga2
2405 Open Icinga Web 2 and check your newly added Windows NSClient++ check :)
2407 ![Icinga 2 Distributed Monitoring Windows Client with NSClient++ nscp-local](images/distributed-monitoring/icinga2_distributed_windows_nscp_counter_icingaweb2.png)
2411 > In order to measure CPU load, you'll need a running NSClient++ service.
2412 > Therefore it is advised to use a local [nscp-api](06-distributed-monitoring.md#distributed-monitoring-windows-nscp-check-api)
2413 > check against its REST API.
2415 ## Advanced Hints <a id="distributed-monitoring-advanced-hints"></a>
2417 You can find additional hints in this section if you prefer to go your own route
2418 with automating setups (setup, certificates, configuration).
2420 ### Certificate Auto-Renewal <a id="distributed-monitoring-certificate-auto-renewal"></a>
2422 Icinga 2 v2.8+ adds the possibility that nodes request certificate updates
2423 on their own. If their expiration date is soon enough, they automatically
2424 renew their already signed certificate by sending a signing request to the
2425 parent node. You'll also see a message in the logs if certificate renewal
2428 ### High-Availability for Icinga 2 Features <a id="distributed-monitoring-high-availability-features"></a>
2430 All nodes in the same zone require that you enable the same features for high-availability (HA).
2432 By default, the following features provide advanced HA functionality:
2434 * [Checks](06-distributed-monitoring.md#distributed-monitoring-high-availability-checks) (load balanced, automated failover).
2435 * [Notifications](06-distributed-monitoring.md#distributed-monitoring-high-availability-notifications) (load balanced, automated failover).
2436 * [DB IDO](06-distributed-monitoring.md#distributed-monitoring-high-availability-db-ido) (Run-Once, automated failover).
2437 * [Elasticsearch](09-object-types.md#objecttype-elasticsearchwriter)
2438 * [Gelf](09-object-types.md#objecttype-gelfwriter)
2439 * [Graphite](09-object-types.md#objecttype-graphitewriter)
2440 * [InfluxDB](09-object-types.md#objecttype-influxdbwriter)
2441 * [OpenTsdb](09-object-types.md#objecttype-opentsdbwriter)
2442 * [Perfdata](09-object-types.md#objecttype-perfdatawriter) (for PNP)
2444 #### High-Availability with Checks <a id="distributed-monitoring-high-availability-checks"></a>
2446 All instances within the same zone (e.g. the `master` zone as HA cluster) must
2447 have the `checker` feature enabled.
2451 # icinga2 feature enable checker
2453 All nodes in the same zone load-balance the check execution. If one instance shuts down,
2454 the other nodes will automatically take over the remaining checks.
2456 #### High-Availability with Notifications <a id="distributed-monitoring-high-availability-notifications"></a>
2458 All instances within the same zone (e.g. the `master` zone as HA cluster) must
2459 have the `notification` feature enabled.
2463 # icinga2 feature enable notification
2465 Notifications are load-balanced amongst all nodes in a zone. By default this functionality
2467 If your nodes should send out notifications independently from any other nodes (this will cause
2468 duplicated notifications if not properly handled!), you can set `enable_ha = false`
2469 in the [NotificationComponent](09-object-types.md#objecttype-notificationcomponent) feature.
2471 #### High-Availability with DB IDO <a id="distributed-monitoring-high-availability-db-ido"></a>
2473 All instances within the same zone (e.g. the `master` zone as HA cluster) must
2474 have the DB IDO feature enabled.
2476 Example DB IDO MySQL:
2478 # icinga2 feature enable ido-mysql
2480 By default the DB IDO feature only runs on one node. All other nodes in the same zone disable
2481 the active IDO database connection at runtime. The node with the active DB IDO connection is
2482 not necessarily the zone master.
2484 **Note**: The DB IDO HA feature can be disabled by setting the `enable_ha` attribute to `false`
2485 for the [IdoMysqlConnection](09-object-types.md#objecttype-idomysqlconnection) or
2486 [IdoPgsqlConnection](09-object-types.md#objecttype-idopgsqlconnection) object on **all** nodes in the
2489 All endpoints will enable the DB IDO feature and connect to the configured
2490 database and dump configuration, status and historical data on their own.
2492 If the instance with the active DB IDO connection dies, the HA functionality will
2493 automatically elect a new DB IDO master.
2495 The DB IDO feature will try to determine which cluster endpoint is currently writing
2496 to the database and bail out if another endpoint is active. You can manually verify that
2497 by running the following query command:
2499 icinga=> SELECT status_update_time, endpoint_name FROM icinga_programstatus;
2500 status_update_time | endpoint_name
2501 ------------------------+---------------
2502 2016-08-15 15:52:26+02 | icinga2-master1.localdomain
2505 This is useful when the cluster connection between endpoints breaks, and prevents
2506 data duplication in split-brain-scenarios. The failover timeout can be set for the
2507 `failover_timeout` attribute, but not lower than 60 seconds.
2509 ### Endpoint Connection Direction <a id="distributed-monitoring-advanced-hints-connection-direction"></a>
2511 Nodes will attempt to connect to another node when its local [Endpoint](09-object-types.md#objecttype-endpoint) object
2512 configuration specifies a valid `host` attribute (FQDN or IP address).
2514 Example for the master node `icinga2-master1.localdomain` actively connecting
2515 to the client node `icinga2-client1.localdomain`:
2517 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.conf
2521 object Endpoint "icinga2-client1.localdomain" {
2522 host = "192.168.56.111" //the master actively tries to connect to the client
2526 Example for the client node `icinga2-client1.localdomain` not actively
2527 connecting to the master node `icinga2-master1.localdomain`:
2529 [root@icinga2-client1.localdomain /]# vim /etc/icinga2/zones.conf
2533 object Endpoint "icinga2-master1.localdomain" {
2534 //do not actively connect to the master by leaving out the 'host' attribute
2538 It is not necessary that both the master and the client node establish
2539 two connections to each other. Icinga 2 will only use one connection
2540 and close the second connection if established.
2542 **Tip**: Choose either to let master/satellite nodes connect to client nodes
2546 ### Disable Log Duration for Command Endpoints <a id="distributed-monitoring-advanced-hints-command-endpoint-log-duration"></a>
2548 The replay log is a built-in mechanism to ensure that nodes in a distributed setup
2549 keep the same history (check results, notifications, etc.) when nodes are temporarily
2550 disconnected and then reconnect.
2552 This functionality is not needed when a master/satellite node is sending check
2553 execution events to a client which is purely configured for [command endpoint](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint)
2556 The [Endpoint](09-object-types.md#objecttype-endpoint) object attribute `log_duration` can
2557 be lower or set to 0 to fully disable any log replay updates when the
2558 client is not connected.
2560 Configuration on the master node `icinga2-master1.localdomain`:
2562 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.conf
2566 object Endpoint "icinga2-client1.localdomain" {
2567 host = "192.168.56.111" //the master actively tries to connect to the client
2571 object Endpoint "icinga2-client2.localdomain" {
2572 host = "192.168.56.112" //the master actively tries to connect to the client
2576 Configuration on the client `icinga2-client1.localdomain`:
2578 [root@icinga2-client1.localdomain /]# vim /etc/icinga2/zones.conf
2582 object Endpoint "icinga2-master1.localdomain" {
2583 //do not actively connect to the master by leaving out the 'host' attribute
2587 object Endpoint "icinga2-master2.localdomain" {
2588 //do not actively connect to the master by leaving out the 'host' attribute
2592 ### Manual Certificate Creation <a id="distributed-monitoring-advanced-hints-certificates-manual"></a>
2594 #### Create CA on the Master <a id="distributed-monitoring-advanced-hints-certificates-manual-ca"></a>
2596 Choose the host which should store the certificate authority (one of the master nodes).
2598 The first step is the creation of the certificate authority (CA) by running the following command
2601 [root@icinga2-master1.localdomain /root]# icinga2 pki new-ca
2603 #### Create CSR and Certificate <a id="distributed-monitoring-advanced-hints-certificates-manual-create"></a>
2605 Create a certificate signing request (CSR) for the local instance:
2608 [root@icinga2-master1.localdomain /root]# icinga2 pki new-cert --cn icinga2-master1.localdomain \
2609 --key icinga2-master1.localdomain.key \
2610 --csr icinga2-master1.localdomain.csr
2613 Sign the CSR with the previously created CA:
2616 [root@icinga2-master1.localdomain /root]# icinga2 pki sign-csr --csr icinga2-master1.localdomain.csr --cert icinga2-master1.localdomain
2619 Repeat the steps for all instances in your setup.
2621 #### Copy Certificates <a id="distributed-monitoring-advanced-hints-certificates-manual-copy"></a>
2623 Copy the host's certificate files and the public CA certificate to `/var/lib/icinga2/certs`:
2626 [root@icinga2-master1.localdomain /root]# mkdir -p /var/lib/icinga2/certs
2627 [root@icinga2-master1.localdomain /root]# cp icinga2-master1.localdomain.{crt,key} /var/lib/icinga2/certs
2628 [root@icinga2-master1.localdomain /root]# cp /var/lib/icinga2/ca/ca.crt /var/lib/icinga2/certs
2631 Ensure that proper permissions are set (replace `icinga` with the Icinga 2 daemon user):
2634 [root@icinga2-master1.localdomain /root]# chown -R icinga:icinga /var/lib/icinga2/certs
2635 [root@icinga2-master1.localdomain /root]# chmod 600 /var/lib/icinga2/certs/*.key
2636 [root@icinga2-master1.localdomain /root]# chmod 644 /var/lib/icinga2/certs/*.crt
2639 The CA public and private key are stored in the `/var/lib/icinga2/ca` directory. Keep this path secure and include
2642 #### Create Multiple Certificates <a id="distributed-monitoring-advanced-hints-certificates-manual-multiple"></a>
2644 Use your preferred method to automate the certificate generation process.
2647 [root@icinga2-master1.localdomain /var/lib/icinga2/certs]# for node in icinga2-master1.localdomain icinga2-master2.localdomain icinga2-satellite1.localdomain; do icinga2 pki new-cert --cn $node --csr $node.csr --key $node.key; done
2648 information/base: Writing private key to 'icinga2-master1.localdomain.key'.
2649 information/base: Writing certificate signing request to 'icinga2-master1.localdomain.csr'.
2650 information/base: Writing private key to 'icinga2-master2.localdomain.key'.
2651 information/base: Writing certificate signing request to 'icinga2-master2.localdomain.csr'.
2652 information/base: Writing private key to 'icinga2-satellite1.localdomain.key'.
2653 information/base: Writing certificate signing request to 'icinga2-satellite1.localdomain.csr'.
2655 [root@icinga2-master1.localdomain /var/lib/icinga2/certs]# for node in icinga2-master1.localdomain icinga2-master2.localdomain icinga2-satellite1.localdomain; do sudo icinga2 pki sign-csr --csr $node.csr --cert $node.crt; done
2656 information/pki: Writing certificate to file 'icinga2-master1.localdomain.crt'.
2657 information/pki: Writing certificate to file 'icinga2-master2.localdomain.crt'.
2658 information/pki: Writing certificate to file 'icinga2-satellite1.localdomain.crt'.
2661 Copy and move these certificates to the respective instances e.g. with SSH/SCP.
2663 ## Automation <a id="distributed-monitoring-automation"></a>
2665 These hints should get you started with your own automation tools (Puppet, Ansible, Chef, Salt, etc.)
2666 or custom scripts for automated setup.
2668 These are collected best practices from various community channels.
2670 * [Silent Windows setup](06-distributed-monitoring.md#distributed-monitoring-automation-windows-silent)
2671 * [Node Setup CLI command](06-distributed-monitoring.md#distributed-monitoring-automation-cli-node-setup) with parameters
2673 If you prefer an alternate method, we still recommend leaving all the Icinga 2 features intact (e.g. `icinga2 feature enable api`).
2674 You should also use well known and documented default configuration file locations (e.g. `zones.conf`).
2675 This will tremendously help when someone is trying to help in the [community channels](https://icinga.com/community/).
2678 ### Silent Windows Setup <a id="distributed-monitoring-automation-windows-silent"></a>
2680 If you want to install the client silently/unattended, use the `/qn` modifier. The
2681 installation should not trigger a restart, but if you want to be completely sure, you can use the `/norestart` modifier.
2683 C:> msiexec /i C:\Icinga2-v2.5.0-x86.msi /qn /norestart
2685 Once the setup is completed you can use the `node setup` cli command too.
2687 ### Node Setup using CLI Parameters <a id="distributed-monitoring-automation-cli-node-setup"></a>
2689 Instead of using the `node wizard` CLI command, there is an alternative `node setup`
2690 command available which has some prerequisites.
2692 **Note**: The CLI command can be used on Linux/Unix and Windows operating systems.
2693 The graphical Windows setup wizard actively uses these CLI commands.
2695 #### Node Setup on the Master Node <a id="distributed-monitoring-automation-cli-node-setup-master"></a>
2697 In case you want to setup a master node you must add the `--master` parameter
2698 to the `node setup` CLI command. In addition to that the `--cn` can optionally
2699 be passed (defaults to the FQDN).
2701 Parameter | Description
2702 --------------------|--------------------
2703 Common name (CN) | **Optional.** Specified with the `--cn` parameter. By convention this should be the host's FQDN. Defaults to the FQDN.
2704 Zone name | **Optional.** Specified with the `--zone` parameter. Defaults to `master`.
2705 Listen on | **Optional.** Specified with the `--listen` parameter. Syntax is `host,port`.
2706 Disable conf.d | **Optional.** Specified with the `disable-confd` parameter. If provided, this disables the `include_recursive "conf.d"` directive and adds the `api-users.conf` file inclusion to `icinga2.conf`. Available since v2.9+. Not set by default for compatibility reasons with Puppet, Ansible, Chef, etc.
2711 [root@icinga2-master1.localdomain /]# icinga2 node setup --master
2714 In case you want to bind the `ApiListener` object to a specific
2715 host/port you can specify it like this:
2717 --listen 192.68.56.101,5665
2719 In case you don't need anything in `conf.d`, use the following command line:
2722 [root@icinga2-master1.localdomain /]# icinga2 node setup --master --disable-confd
2726 #### Node Setup with Satellites/Clients <a id="distributed-monitoring-automation-cli-node-setup-satellite-client"></a>
2728 Make sure that the `/var/lib/icinga2/certs` directory exists and is owned by the `icinga`
2729 user (or the user Icinga 2 is running as).
2731 [root@icinga2-client1.localdomain /]# mkdir -p /var/lib/icinga2/certs
2732 [root@icinga2-client1.localdomain /]# chown -R icinga:icinga /var/lib/icinga2/certs
2734 First you'll need to generate a new local self-signed certificate.
2735 Pass the following details to the `pki new-cert` CLI command:
2737 Parameter | Description
2738 --------------------|--------------------
2739 Common name (CN) | **Required.** By convention this should be the host's FQDN.
2740 Client certificate files | **Required.** These generated files will be put into the specified location (--key and --file). By convention this should be using `/var/lib/icinga2/certs` as directory.
2744 [root@icinga2-client1.localdomain /]# icinga2 pki new-cert --cn icinga2-client1.localdomain \
2745 --key /var/lib/icinga2/certs/icinga2-client1.localdomain.key \
2746 --cert /var/lib/icinga2/certs/icinga2-client1.localdomain.crt
2748 Request the master certificate from the master host (`icinga2-master1.localdomain`)
2749 and store it as `trusted-master.crt`. Review it and continue.
2751 Pass the following details to the `pki save-cert` CLI command:
2753 Parameter | Description
2754 --------------------|--------------------
2755 Client certificate files | **Required.** Pass the previously generated files using the `--key` and `--cert` parameters.
2756 Trusted parent certificate | **Required.** Store the parent's certificate file. Manually verify that you're trusting it.
2757 Parent host | **Required.** FQDN or IP address of the parent host.
2761 [root@icinga2-client1.localdomain /]# icinga2 pki save-cert --key /var/lib/icinga2/certs/icinga2-client1.localdomain.key \
2762 --cert /var/lib/icinga2/certs/icinga2-client1.localdomain.crt \
2763 --trustedcert /var/lib/icinga2/certs/trusted-parent.crt \
2764 --host icinga2-master1.localdomain
2766 Continue with the additional node setup step. Specify a local endpoint and zone name (`icinga2-client1.localdomain`)
2767 and set the master host (`icinga2-master1.localdomain`) as parent zone configuration. Specify the path to
2768 the previously stored trusted master certificate.
2770 Pass the following details to the `node setup` CLI command:
2772 Parameter | Description
2773 --------------------|--------------------
2774 Common name (CN) | **Optional.** Specified with the `--cn` parameter. By convention this should be the host's FQDN.
2775 Request ticket | **Required.** Add the previously generated [ticket number](06-distributed-monitoring.md#distributed-monitoring-setup-csr-auto-signing).
2776 Trusted master certificate | **Required.** Add the previously fetched trusted master certificate (this step means that you've verified its origin).
2777 Parent host | **Optional.** FQDN or IP address of the parent host. This is where the command connects for CSR signing. If not specified, you need to manually copy the parent's public CA certificate file into `/var/lib/icinga2/certs/ca.crt` in order to start Icinga 2.
2778 Parent endpoint | **Required.** Specify the parent's endpoint name.
2779 Client zone name | **Required.** Specify the client's zone name.
2780 Parent zone name | **Optional.** Specify the parent's zone name.
2781 Accept config | **Optional.** Whether this node accepts configuration sync from the master node (required for [config sync mode](06-distributed-monitoring.md#distributed-monitoring-top-down-config-sync)).
2782 Accept commands | **Optional.** Whether this node accepts command execution messages from the master node (required for [command endpoint mode](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint)).
2783 Global zones | **Optional.** Allows to specify more global zones in addition to `global-templates` and `director-global`.
2784 Disable conf.d | **Optional.** Specified with the `disable-confd` parameter. If provided, this disables the `include_recursive "conf.d"` directive in `icinga2.conf`. Available since v2.9+. Not set by default for compatibility reasons with Puppet, Ansible, Chef, etc.
2788 > The `master_host` parameter is deprecated and will be removed in 2.10.0. Please use `--parent_host` instead.
2790 Example for Icinga 2 v2.9:
2793 [root@icinga2-client1.localdomain /]# icinga2 node setup --ticket ead2d570e18c78abf285d6b85524970a0f69c22d \
2794 --cn icinga2-client1.localdomain \
2795 --endpoint icinga2-master1.localdomain \
2796 --zone icinga2-client1.localdomain \
2797 --parent_zone master \
2798 --parent_host icinga2-master1.localdomain \
2799 --trustedcert /var/lib/icinga2/certs/trusted-parent.crt \
2800 --accept-commands --accept-config \
2804 In case the client should connect to the master node, you'll
2805 need to modify the `--endpoint` parameter using the format `cn,host,port`:
2807 --endpoint icinga2-master1.localdomain,192.168.56.101,5665
2809 Specify the parent zone using the `--parent_zone` parameter. This is useful
2810 if the client connects to a satellite, not the master instance.
2812 --parent_zone satellite
2814 In case the client should know the additional global zone `linux-templates`, you'll
2815 need to set the `--global-zones` parameter.
2817 --global_zones linux-templates
2819 The `--parent-host` parameter is optional since v2.9 and allows you to perform a connection-less setup.
2820 You cannot restart Icinga 2 yet, the CLI command asked to to manually copy the parent's public CA
2821 certificate file in `/var/lib/icinga2/certs/ca.crt`. Once Icinga 2 is started, it sends
2822 a ticket signing request to the parent node. If you have provided a ticket, the master node
2823 signs the request and sends it back to the client which performs a certificate update in-memory.
2825 In case you did not provide a ticket, you need to manually sign the CSR on the master node
2826 which holds the CA's key pair.
2829 **You can find additional best practices below.**
2831 If this client node is configured as [remote command endpoint execution](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint)
2832 you can safely disable the `checker` feature. The `node setup` CLI command already disabled the `notification` feature.
2834 [root@icinga2-client1.localdomain /]# icinga2 feature disable checker
2836 Disable "conf.d" inclusion if this is a [top down](06-distributed-monitoring.md#distributed-monitoring-top-down)
2839 [root@icinga2-client1.localdomain /]# sed -i 's/include_recursive "conf.d"/\/\/include_recursive "conf.d"/g' /etc/icinga2/icinga2.conf
2841 **Note**: This is the default since v2.9.
2843 **Optional**: Add an ApiUser object configuration for remote troubleshooting.
2845 [root@icinga2-client1.localdomain /]# cat <<EOF >/etc/icinga2/conf.d/api-users.conf
2846 object ApiUser "root" {
2847 password = "clientsupersecretpassword"
2852 In case you've previously disabled the "conf.d" directory only
2853 add the file file `conf.d/api-users.conf`:
2855 [root@icinga2-client1.localdomain /]# echo 'include "conf.d/api-users.conf"' >> /etc/icinga2/icinga2.conf
2857 Finally restart Icinga 2.
2859 [root@icinga2-client1.localdomain /]# systemctl restart icinga2
2861 Your automation tool must then configure master node in the meantime.
2863 # cat <<EOF >>/etc/icinga2/zones.conf
2864 object Endpoint "icinga2-client1.localdomain" {
2865 //client connects itself
2868 object Zone "icinga2-client1.localdomain" {
2869 endpoints = [ "icinga2-client1.localdomain" ]
2875 ## Using Multiple Environments <a id="distributed-monitoring-environments"></a>
2877 In some cases it can be desired to run multiple Icinga instances on the same host.
2878 Two potential scenarios include:
2880 * Different versions of the same monitoring configuration (e.g. production and testing)
2881 * Disparate sets of checks for entirely unrelated monitoring environments (e.g. infrastructure and applications)
2883 The configuration is done with the global constants `ApiBindHost` and `ApiBindPort`
2884 or the `bind_host` and `bind_port` attributes of the
2885 [ApiListener](09-object-types.md#objecttype-apilistener) object.
2887 The environment must be set with the global constant `Environment` or as object attribute
2888 of the [IcingaApplication](09-object-types.md#objecttype-icingaapplication) object.
2890 In any case the constant is default value for the attribute and the direct configuration in the objects
2891 have more precedence. The constants have been added to allow the values being set from the CLI on startup.
2893 When Icinga establishes a TLS connection to another cluster instance it automatically uses the [SNI extension](https://en.wikipedia.org/wiki/Server_Name_Indication)
2894 to signal which endpoint it is attempting to connect to. On its own this can already be used to position multiple
2895 Icinga instances behind a load balancer.
2897 SNI example: `icinga2-client1.localdomain`
2899 However, if the environment is configured to `production`, Icinga appends the environment name to the SNI hostname like this:
2901 SNI example with environment: `icinga2-client1.localdomain:production`
2903 Middleware like loadbalancers or TLS proxies can read the SNI header and route the connection to the appropriate target.
2904 I.e., it uses a single externally-visible TCP port (usually 5665) and forwards connections to one or more Icinga
2905 instances which are bound to a local TCP port. It does so by inspecting the environment name that is sent as part of the