1 # <a id="distributed-monitoring-high-availability"></a> Distributed Monitoring and High Availability
3 Building distributed environments with high availability included is fairly easy with Icinga 2.
4 The cluster feature is built-in and allows you to build many scenarios based on your requirements:
6 * [High Availability](13-distributed-monitoring-ha.md#cluster-scenarios-high-availability). All instances in the `Zone` run as Active/Active cluster.
7 * [Distributed Zones](13-distributed-monitoring-ha.md#cluster-scenarios-distributed-zones). A master zone and one or more satellites in their zones.
8 * [Load Distribution](13-distributed-monitoring-ha.md#cluster-scenarios-load-distribution). A configuration master and multiple checker satellites.
10 You can combine these scenarios into a global setup fitting your requirements.
12 Each instance got their own event scheduler, and does not depend on a centralized master
13 coordinating and distributing the events. In case of a cluster failure, all nodes
14 continue to run independently. Be alarmed when your cluster fails and a Split-Brain-scenario
15 is in effect - all alive instances continue to do their job, and history will begin to differ.
18 ## <a id="cluster-requirements"></a> Cluster Requirements
20 Before you start deploying, keep the following things in mind:
22 Your [SSL CA and certificates](13-distributed-monitoring-ha.md#manual-certificate-generation) are mandatory for secure communication.
24 Communication between zones requires one of these connection directions:
26 * The parent zone nodes are able to connect to the child zone nodes (`parent => child`).
27 * The child zone nodes are able to connect to the parent zone nodes (`parent <= child`).
28 * Both connnection directions work.
30 Update firewall rules and ACLs.
32 * Icinga 2 master, satellite and client instances communicate using the default tcp port `5665`.
34 Get pen and paper or a drawing board and design your nodes and zones!
36 * Keep the [naming convention](13-distributed-monitoring-ha.md#cluster-naming-convention) for nodes in mind.
37 * All nodes (endpoints) in a cluster zone provide high availability functionality and trust each other.
38 * Cluster zones can be built in a Top-Down-design where the child trusts the parent.
40 Decide whether to use the built-in [configuration syncronization](13-distributed-monitoring-ha.md#cluster-zone-config-sync) or use an external tool (Puppet, Ansible, Chef, Salt, etc) to manage the configuration deployment.
45 > If you're looking for troubleshooting cluster problems, check the general
46 > [troubleshooting](16-troubleshooting.md#troubleshooting-cluster) section.
48 ## <a id="manual-certificate-generation"></a> Manual SSL Certificate Generation
50 Icinga 2 provides [CLI commands](8-cli-commands.md#cli-command-pki) assisting with CA
51 and node certificate creation for your Icinga 2 distributed setup.
55 > You can also use the master and client setup wizards to install the cluster nodes
56 > using CSR-Autosigning.
58 > The manual steps are helpful if you want to use your own and/or existing CA (for example
63 > You're free to use your own method to generated a valid ca and signed client
66 The first step is the creation of the certificate authority (CA) by running the
71 Now create a certificate and key file for each node running the following command
72 (replace `icinga2a` with the required hostname):
74 # icinga2 pki new-cert --cn icinga2a --key icinga2a.key --csr icinga2a.csr
75 # icinga2 pki sign-csr --csr icinga2a.csr --cert icinga2a.crt
77 Repeat the step for all nodes in your cluster scenario.
79 Save the CA key in a secure location in case you want to set up certificates for
80 additional nodes at a later time.
82 Navigate to the location of your newly generated certificate files, and manually
83 copy/transfer them to `/etc/icinga2/pki` in your Icinga 2 configuration folder.
87 > The certificate files must be readable by the user Icinga 2 is running as. Also,
88 > the private key file must not be world-readable.
90 Each node requires the following files in `/etc/icinga2/pki` (replace `fqdn-nodename` with
94 * <fqdn-nodename>.crt
95 * <fqdn-nodename>.key
97 If you're planning to use your existing CA and certificates please note that you *must not*
98 use wildcard certificates. The common name (CN) is mandatory for the cluster communication and
99 therefore must be unique for each connecting instance.
101 ## <a id="cluster-naming-convention"></a> Cluster Naming Convention
103 The SSL certificate common name (CN) will be used by the [ApiListener](6-object-types.md#objecttype-apilistener)
104 object to determine the local authority. This name must match the local [Endpoint](6-object-types.md#objecttype-endpoint)
107 Certificate generation for host with the FQDN `icinga2a`:
109 # icinga2 pki new-cert --cn icinga2a --key icinga2a.key --csr icinga2a.csr
110 # icinga2 pki sign-csr --csr icinga2a.csr --cert icinga2a.crt
112 Add a new `Endpoint` object named `icinga2a`:
116 object Endpoint "icinga2a" {
117 host = "icinga2a.icinga.org"
120 The [Endpoint](6-object-types.md#objecttype-endpoint) name is further referenced as `endpoints` attribute on the
121 [Zone](6-object-types.md#objecttype-zone) object.
123 object Endpoint "icinga2b" {
124 host = "icinga2b.icinga.org"
127 object Zone "config-ha-master" {
128 endpoints = [ "icinga2a", "icinga2b" ]
131 Specifying the local node name using the [NodeName](13-distributed-monitoring-ha.md#configure-nodename) variable requires
132 the same name as used for the endpoint name and common name above. If not set, the FQDN is used.
134 const NodeName = "icinga2a"
136 If you're using the host's FQDN everywhere, you're on the safe side. The setup wizards
137 will do the very same.
139 ## <a id="cluster-configuration"></a> Cluster Configuration
141 The following section describe which configuration must be updated/created
142 in order to get your cluster running with basic functionality.
144 * [configure the node name](13-distributed-monitoring-ha.md#configure-nodename)
145 * [configure the ApiListener object](13-distributed-monitoring-ha.md#configure-apilistener-object)
146 * [configure cluster endpoints](13-distributed-monitoring-ha.md#configure-cluster-endpoints)
147 * [configure cluster zones](13-distributed-monitoring-ha.md#configure-cluster-zones)
149 Once you're finished with the basic setup the following section will
150 describe how to use [zone configuration synchronisation](13-distributed-monitoring-ha.md#cluster-zone-config-sync)
151 and configure [cluster scenarios](13-distributed-monitoring-ha.md#cluster-scenarios).
153 ### <a id="configure-nodename"></a> Configure the Icinga Node Name
155 Instead of using the default FQDN as node name you can optionally set
156 that value using the [NodeName](18-language-reference.md#constants) constant.
160 > Skip this step if your FQDN already matches the default `NodeName` set
161 > in `/etc/icinga2/constants.conf`.
163 This setting must be unique for each node, and must also match
164 the name of the local [Endpoint](6-object-types.md#objecttype-endpoint) object and the
165 SSL certificate common name as described in the
166 [cluster naming convention](13-distributed-monitoring-ha.md#cluster-naming-convention).
168 vim /etc/icinga2/constants.conf
170 /* Our local instance name. By default this is the server's hostname as returned by `hostname --fqdn`.
171 * This should be the common name from the API certificate.
173 const NodeName = "icinga2a"
176 Read further about additional [naming conventions](13-distributed-monitoring-ha.md#cluster-naming-convention).
178 Not specifying the node name will make Icinga 2 using the FQDN. Make sure that all
179 configured endpoint names and common names are in sync.
181 ### <a id="configure-apilistener-object"></a> Configure the ApiListener Object
183 The [ApiListener](6-object-types.md#objecttype-apilistener) object needs to be configured on
184 every node in the cluster with the following settings:
186 A sample config looks like:
188 object ApiListener "api" {
189 cert_path = SysconfDir + "/icinga2/pki/" + NodeName + ".crt"
190 key_path = SysconfDir + "/icinga2/pki/" + NodeName + ".key"
191 ca_path = SysconfDir + "/icinga2/pki/ca.crt"
193 accept_commands = true
196 You can simply enable the `api` feature using
198 # icinga2 feature enable api
200 Edit `/etc/icinga2/features-enabled/api.conf` if you require the configuration
201 synchronisation enabled for this node. Set the `accept_config` attribute to `true`.
203 If you want to use this node as [remote client for command execution](11-icinga2-client.md#icinga2-client-configuration-command-bridge)
204 set the `accept_commands` attribute to `true`.
208 > The certificate files must be readable by the user Icinga 2 is running as. Also,
209 > the private key file must not be world-readable.
211 ### <a id="configure-cluster-endpoints"></a> Configure Cluster Endpoints
213 `Endpoint` objects specify the `host` and `port` settings for the cluster node
214 connection information.
215 This configuration can be the same on all nodes in the cluster only containing
216 connection information.
218 A sample configuration looks like:
221 * Configure config master endpoint
224 object Endpoint "icinga2a" {
225 host = "icinga2a.icinga.org"
228 If this endpoint object is reachable on a different port, you must configure the
229 `ApiListener` on the local `Endpoint` object accordingly too.
231 If you don't want the local instance to connect to the remote instance, remove the
232 `host` attribute locally. Keep in mind that the configuration is now different amongst
233 all instances and point-of-view dependant.
235 ### <a id="configure-cluster-zones"></a> Configure Cluster Zones
237 `Zone` objects specify the endpoints located in a zone. That way your distributed setup can be
238 seen as zones connected together instead of multiple instances in that specific zone.
240 Zones can be used for [high availability](13-distributed-monitoring-ha.md#cluster-scenarios-high-availability),
241 [distributed setups](13-distributed-monitoring-ha.md#cluster-scenarios-distributed-zones) and
242 [load distribution](13-distributed-monitoring-ha.md#cluster-scenarios-load-distribution).
243 Furthermore zones are used for the [Icinga 2 remote client](11-icinga2-client.md#icinga2-client).
245 Each Icinga 2 `Endpoint` must be put into its respective `Zone`. In this example, you will
246 define the zone `config-ha-master` where the `icinga2a` and `icinga2b` endpoints
247 are located. The `check-satellite` zone consists of `icinga2c` only, but more nodes could
250 The `config-ha-master` zone acts as High-Availability setup - the Icinga 2 instances elect
251 one instance running a check, notification or feature (DB IDO), for example `icinga2a`. In case of
252 failure of the `icinga2a` instance, `icinga2b` will take over automatically.
254 object Zone "config-ha-master" {
255 endpoints = [ "icinga2a", "icinga2b" ]
258 The `check-satellite` zone is a separated location and only sends back their checkresults to
259 the defined parent zone `config-ha-master`.
261 object Zone "check-satellite" {
262 endpoints = [ "icinga2c" ]
263 parent = "config-ha-master"
267 ## <a id="cluster-zone-config-sync"></a> Zone Configuration Synchronisation
269 In case you are using the Icinga 2 API for creating, modifying and deleting objects
270 at runtime, please continue over [here](9-icinga2-api.md#icinga2-api-config-objects-cluster-sync).
272 By default all objects for specific zones should be organized in
274 /etc/icinga2/zones.d/<zonename>
276 on the configuration master.
278 Your child zones and endpoint members **must not** have their config copied to `zones.d`.
279 The built-in configuration synchronisation takes care of that if your nodes accept
280 configuration from the parent zone. You can define that in the
281 [ApiListener](13-distributed-monitoring-ha.md#configure-apilistener-object) object by configuring the `accept_config`
282 attribute accordingly.
284 You should remove the sample config included in `conf.d` by commenting the `recursive_include`
285 statement in [icinga2.conf](4-configuring-icinga-2.md#icinga2-conf):
287 //include_recursive "conf.d"
289 This applies to any other non-used configuration directories as well (e.g. `repository.d`
292 Better use a dedicated directory name for local configuration like `local` or similar, and
293 include that one if your nodes require local configuration not being synced to other nodes. That's
294 useful for local [health checks](13-distributed-monitoring-ha.md#cluster-health-check) for example.
298 > In a [high availability](13-distributed-monitoring-ha.md#cluster-scenarios-high-availability)
299 > setup only one assigned node can act as configuration master. All other zone
300 > member nodes **must not** have the `/etc/icinga2/zones.d` directory populated.
303 These zone packages are then distributed to all nodes in the same zone, and
304 to their respective target zone instances.
306 Each configured zone must exist with the same directory name. The parent zone
307 syncs the configuration to the child zones, if allowed using the `accept_config`
308 attribute of the [ApiListener](13-distributed-monitoring-ha.md#configure-apilistener-object) object.
310 Config on node `icinga2a`:
312 object Zone "master" {
313 endpoints = [ "icinga2a" ]
316 object Zone "checker" {
317 endpoints = [ "icinga2b" ]
328 Config on node `icinga2b`:
330 object Zone "master" {
331 endpoints = [ "icinga2a" ]
334 object Zone "checker" {
335 endpoints = [ "icinga2b" ]
340 EMPTY_IF_CONFIG_SYNC_ENABLED
342 If the local configuration is newer than the received update Icinga 2 will skip the synchronisation
347 > `zones.d` must not be included in [icinga2.conf](4-configuring-icinga-2.md#icinga2-conf). Icinga 2 automatically
348 > determines the required include directory. This can be overridden using the
349 > [global constant](18-language-reference.md#constants) `ZonesDir`.
351 ### <a id="zone-global-config-templates"></a> Global Configuration Zone for Templates
353 If your zone configuration setup shares the same templates, groups, commands, timeperiods, etc.
354 you would have to duplicate quite a lot of configuration objects making the merged configuration
355 on your configuration master unique.
359 > Only put templates, groups, etc into this zone. DO NOT add checkable objects such as
360 > hosts or services here. If they are checked by all instances globally, this will lead
361 > into duplicated check results and unclear state history. Not easy to troubleshoot too -
362 > you have been warned.
364 That is not necessary by defining a global zone shipping all those templates. By setting
365 `global = true` you ensure that this zone serving common configuration templates will be
366 synchronized to all involved nodes (only if they accept configuration though).
368 Config on configuration master:
380 In this example, the global zone is called `global-templates` and must be defined in
381 your zone configuration visible to all nodes.
383 object Zone "global-templates" {
387 If the remote node does not have this zone configured, it will ignore the configuration
388 update, if it accepts synchronized configuration.
390 If you do not require any global configuration, skip this setting.
392 ### <a id="zone-config-sync-permissions"></a> Zone Configuration Synchronisation Permissions
394 Each [ApiListener](6-object-types.md#objecttype-apilistener) object must have the `accept_config` attribute
395 set to `true` to receive configuration from the parent `Zone` members. Default value is `false`.
397 object ApiListener "api" {
398 cert_path = SysconfDir + "/icinga2/pki/" + NodeName + ".crt"
399 key_path = SysconfDir + "/icinga2/pki/" + NodeName + ".key"
400 ca_path = SysconfDir + "/icinga2/pki/ca.crt"
404 If `accept_config` is set to `false`, this instance won't accept configuration from remote
405 master instances anymore.
409 > Look into the [troubleshooting guides](16-troubleshooting.md#troubleshooting-cluster-config-sync) for debugging
410 > problems with the configuration synchronisation.
413 ### <a id="zone-config-sync-best-practice"></a> Zone Configuration Synchronisation Best Practice
415 The configuration synchronisation works with multiple hierarchies. The following example
416 illustrate a quite common setup where the master is reponsible for configuration deployment:
418 * [High-Availability master zone](13-distributed-monitoring-ha.md#distributed-monitoring-high-availability)
419 * [Distributed satellites](12-distributed-monitoring-ha.md#)
420 * [Remote clients](11-icinga2-client.md#icinga2-client-scenarios) connected to the satellite
422 While you could use the clients with local configuration and service discovery on the satellite/master
423 **bottom up**, the configuration sync could be more reasonable working **top-down** in a cascaded scenario.
425 Take pen and paper and draw your network scenario including the involved zone and endpoint names.
426 Once you've added them to your zones.conf as connection and permission configuration, start over with
427 the actual configuration organization:
429 * Ensure that `command` object definitions are globally available. That way you can use the
430 `command_endpoint` configuration more easily on clients as [command execution bridge](11-icinga2-client.md#icinga2-client-configuration-command-bridge)
431 * Generic `Templates`, `timeperiods`, `downtimes` should be synchronized in a global zone as well.
432 * [Apply rules](3-monitoring-basics.md#using-apply) can be synchronized globally. Keep in mind that they are evaluated on each instance,
433 and might require additional filters (e.g. `match("icinga2*", NodeName) or similar based on the zone information.
434 * [Apply rules](3-monitoring-basics.md#using-apply) specified inside zone directories will only affect endpoints in the same zone or below.
435 * Host configuration must be put into the specific zone directory.
436 * Duplicated host and service objects (also generated by faulty apply rules) will generate a configuration error.
437 * Consider using custom constants in your host/service configuration. Each instance may set their local value, e.g. for `PluginDir`.
439 This example specifies the following hierarchy over three levels:
441 * `ha-master` zone with two child zones `dmz1-checker` and `dmz2-checker`
442 * `dmz1-checker` has two client child zones `dmz1-client1` and `dmz1-client2`
443 * `dmz2-checker` has one client child zone `dmz2-client9`
445 The configuration tree could look like this:
447 # tree /etc/icinga2/zones.d
449 ├── dmz1-checker
450 │  └── health.conf
451 ├── dmz1-client1
452 │  └── hosts.conf
453 ├── dmz1-client2
454 │  └── hosts.conf
455 ├── dmz2-checker
456 │  └── health.conf
457 ├── dmz2-client9
458 │  └── hosts.conf
459 ├── global-templates
460 │  ├── apply_notifications.conf
461 │  ├── apply_services.conf
462 │  ├── commands.conf
463 │  ├── groups.conf
464 │  ├── templates.conf
465 │  └── users.conf
467 │  └── health.conf
470 7 directories, 13 files
472 If you prefer a different naming schema for directories or files names, go for it. If you
473 are unsure about the best method, join the [support channels](1-about.md#support) and discuss
476 If you are planning to synchronize local service health checks inside a zone, look into the
477 [command endpoint](13-distributed-monitoring-ha.md#cluster-health-check-command-endpoint)
482 ## <a id="cluster-health-check"></a> Cluster Health Check
484 The Icinga 2 [ITL](7-icinga-template-library.md#icinga-template-library) provides
485 an internal check command checking all configured `EndPoints` in the cluster setup.
486 The check result will become critical if one or more configured nodes are not connected.
490 object Host "icinga2a" {
491 display_name = "Health Checks on icinga2a"
493 address = "192.168.33.10"
494 check_command = "hostalive"
497 object Service "cluster" {
498 check_command = "cluster"
502 host_name = "icinga2a"
505 Each cluster node should execute its own local cluster health check to
506 get an idea about network related connection problems from different
509 Additionally you can monitor the connection from the local zone to the remote
512 Example for the `checker` zone checking the connection to the `master` zone:
514 object Service "cluster-zone-master" {
515 check_command = "cluster-zone"
518 vars.cluster_zone = "master"
520 host_name = "icinga2b"
523 ## <a id="cluster-health-check-command-endpoint"></a> Cluster Health Check with Command Endpoints
525 If you are planning to sync the zone configuration inside a [High-Availability]()
526 cluster zone, you can also use the `command_endpoint` object attribute to
527 pin host/service checks to a specific endpoint inside the same zone.
529 This requires the `accept_commands` setting inside the [ApiListener](13-distributed-monitoring-ha.md#configure-apilistener-object)
530 object set to `true` similar to the [remote client command execution bridge](11-icinga2-client.md#icinga2-client-configuration-command-bridge)
533 Make sure to set `command_endpoint` to the correct endpoint instance.
534 The example below assumes that the endpoint name is the same as the
535 host name configured for health checks. If it differs, define a host
536 custom attribute providing [this information](11-icinga2-client.md#icinga2-client-configuration-command-bridge-master-config).
538 apply Service "cluster-ha" {
539 check_command = "cluster"
542 /* make sure host.name is the same as endpoint name */
543 command_endpoint = host.name
545 assign where regex("^icinga2[a|b]", host.name)
549 ## <a id="cluster-scenarios"></a> Cluster Scenarios
551 All cluster nodes are full-featured Icinga 2 instances. You only need to enabled
552 the features for their role (for example, a `Checker` node only requires the `checker`
553 feature enabled, but not `notification` or `ido-mysql` features).
557 > There's a [Vagrant demo setup](https://github.com/Icinga/icinga-vagrant/tree/master/icinga2x-cluster)
558 > available featuring a two node cluster showcasing several aspects (config sync,
559 > remote command execution, etc).
561 ### <a id="cluster-scenarios-master-satellite-clients"></a> Cluster with Master, Satellites and Remote Clients
563 You can combine "classic" cluster scenarios from HA to Master-Checker with the
564 Icinga 2 Remote Client modes. Each instance plays a certain role in that picture.
566 Imagine the following scenario:
568 * The master zone acts as High-Availability zone
569 * Remote satellite zones execute local checks and report them to the master
570 * All satellites query remote clients and receive check results (which they also replay to the master)
571 * All involved nodes share the same configuration logic: zones, endpoints, apilisteners
573 You'll need to think about the following:
575 * Deploy the entire configuration from the master to satellites and cascading remote clients? ("top down")
576 * Use local client configuration instead and report the inventory to satellites and cascading to the master? ("bottom up")
577 * Combine that with command execution brdiges on remote clients and also satellites
580 ### <a id="cluster-scenarios-security"></a> Security in Cluster Scenarios
582 While there are certain capabilities to ensure the safe communication between all
583 nodes (firewalls, policies, software hardening, etc) the Icinga 2 cluster also provides
584 additional security itself:
586 * [SSL certificates](13-distributed-monitoring-ha.md#manual-certificate-generation) are mandatory for cluster communication.
587 * Child zones only receive event updates (check results, commands, etc) for their configured updates.
588 * Zones cannot influence/interfere other zones. Each checked object is assigned to only one zone.
589 * All nodes in a zone trust each other.
590 * [Configuration sync](13-distributed-monitoring-ha.md#zone-config-sync-permissions) is disabled by default.
592 ### <a id="cluster-scenarios-features"></a> Features in Cluster Zones
594 Each cluster zone may use all available features. If you have multiple locations
595 or departments, they may write to their local database, or populate graphite.
596 Even further all commands are distributed amongst connected nodes. For example, you could
597 re-schedule a check or acknowledge a problem on the master, and it gets replicated to the
598 actual slave checker node.
602 > All features must be same on all endpoints inside an [HA zone](13-distributed-monitoring-ha.md#cluster-scenarios-high-availability).
603 > There are additional [High-Availability-enabled features](13-distributed-monitoring-ha.md#high-availability-features) available.
605 ### <a id="cluster-scenarios-distributed-zones"></a> Distributed Zones
607 That scenario fits if your instances are spread over the globe and they all report
608 to a master instance. Their network connection only works towards the master master
609 (or the master is able to connect, depending on firewall policies) which means
610 remote instances won't see each/connect to each other.
612 All events (check results, downtimes, comments, etc) are synced to the master node,
613 but the remote nodes can still run local features such as a web interface, reporting,
614 graphing, etc. in their own specified zone.
616 Imagine the following example with a master node in Nuremberg, and two remote DMZ
617 based instances in Berlin and Vienna. Additonally you'll specify
618 [global templates](13-distributed-monitoring-ha.md#zone-global-config-templates) available in all zones.
620 The configuration tree on the master instance `nuremberg` could look like this:
633 The configuration deployment will take care of automatically synchronising
634 the child zone configuration:
636 * The master node sends `zones.d/berlin` to the `berlin` child zone.
637 * The master node sends `zones.d/vienna` to the `vienna` child zone.
638 * The master node sends `zones.d/global-templates` to the `vienna` and `berlin` child zones.
640 The endpoint configuration would look like:
642 object Endpoint "nuremberg-master" {
643 host = "nuremberg.icinga.org"
646 object Endpoint "berlin-satellite" {
647 host = "berlin.icinga.org"
650 object Endpoint "vienna-satellite" {
651 host = "vienna.icinga.org"
654 The zones would look like:
656 object Zone "nuremberg" {
657 endpoints = [ "nuremberg-master" ]
660 object Zone "berlin" {
661 endpoints = [ "berlin-satellite" ]
665 object Zone "vienna" {
666 endpoints = [ "vienna-satellite" ]
670 object Zone "global-templates" {
674 The `nuremberg-master` zone will only execute local checks, and receive
675 check results from the satellite nodes in the zones `berlin` and `vienna`.
679 > The child zones `berlin` and `vienna` will get their configuration synchronised
680 > from the configuration master 'nuremberg'. The endpoints in the child
681 > zones **must not** have their `zones.d` directory populated if this endpoint
682 > [accepts synced configuration](13-distributed-monitoring-ha.md#zone-config-sync-permissions).
684 ### <a id="cluster-scenarios-load-distribution"></a> Load Distribution
686 If you are planning to off-load the checks to a defined set of remote workers
687 you can achieve that by:
689 * Deploying the configuration on all nodes.
690 * Let Icinga 2 distribute the load amongst all available nodes.
692 That way all remote check instances will receive the same configuration
693 but only execute their part. The master instance located in the `master` zone
694 can also execute checks, but you may also disable the `Checker` feature.
696 Configuration on the master node:
703 If you are planning to have some checks executed by a specific set of checker nodes
704 you have to define additional zones and define these check objects there.
708 object Endpoint "master-node" {
709 host = "master.icinga.org"
712 object Endpoint "checker1-node" {
713 host = "checker1.icinga.org"
716 object Endpoint "checker2-node" {
717 host = "checker2.icinga.org"
723 object Zone "master" {
724 endpoints = [ "master-node" ]
727 object Zone "checker" {
728 endpoints = [ "checker1-node", "checker2-node" ]
732 object Zone "global-templates" {
738 > The child zones `checker` will get its configuration synchronised
739 > from the configuration master 'master'. The endpoints in the child
740 > zone **must not** have their `zones.d` directory populated if this endpoint
741 > [accepts synced configuration](13-distributed-monitoring-ha.md#zone-config-sync-permissions).
743 ### <a id="cluster-scenarios-high-availability"></a> Cluster High Availability
745 High availability with Icinga 2 is possible by putting multiple nodes into
746 a dedicated [zone](13-distributed-monitoring-ha.md#configure-cluster-zones). All nodes will elect one
747 active master, and retry an election once the current active master is down.
749 Selected features provide advanced [HA functionality](13-distributed-monitoring-ha.md#high-availability-features).
750 Checks and notifications are load-balanced between nodes in the high availability
753 Connections from other zones will be accepted by all active and passive nodes
754 but all are forwarded to the current active master dealing with the check results,
757 object Zone "config-ha-master" {
758 endpoints = [ "icinga2a", "icinga2b", "icinga2c" ]
761 Two or more nodes in a high availability setup require an [initial cluster sync](13-distributed-monitoring-ha.md#initial-cluster-sync).
765 > Keep in mind that **only one node acts as configuration master** having the
766 > configuration files in the `zones.d` directory. All other nodes **must not**
767 > have that directory populated. Instead they are required to
768 > [accept synced configuration](13-distributed-monitoring-ha.md#zone-config-sync-permissions).
769 > Details in the [Configuration Sync Chapter](13-distributed-monitoring-ha.md#cluster-zone-config-sync).
771 ### <a id="cluster-scenarios-multiple-hierarchies"></a> Multiple Hierarchies
773 Your master zone collects all check results for reporting and graphing and also
774 does some sort of additional notifications.
775 The customers got their own instances in their local DMZ zones. They are limited to read/write
776 only their services, but replicate all events back to the master instance.
777 Within each DMZ there are additional check instances also serving interfaces for local
778 departments. The customers instances will collect all results, but also send them back to
779 your master instance.
780 Additionally the customers instance on the second level in the middle prohibits you from
781 sending commands to the subjacent department nodes. You're only allowed to receive the
782 results, and a subset of each customers configuration too.
784 Your master zone will generate global reports, aggregate alert notifications, and check
785 additional dependencies (for example, the customers internet uplink and bandwidth usage).
787 The customers zone instances will only check a subset of local services and delegate the rest
788 to each department. Even though it acts as configuration master with a master dashboard
789 for all departments managing their configuration tree which is then deployed to all
790 department instances. Furthermore the master NOC is able to see what's going on.
792 The instances in the departments will serve a local interface, and allow the administrators
793 to reschedule checks or acknowledge problems for their services.
796 ## <a id="high-availability-features"></a> High Availability for Icinga 2 features
798 All nodes in the same zone require the same features enabled for High Availability (HA)
801 By default the following features provide advanced HA functionality:
803 * [Checks](13-distributed-monitoring-ha.md#high-availability-checks) (load balanced, automated failover)
804 * [Notifications](13-distributed-monitoring-ha.md#high-availability-notifications) (load balanced, automated failover)
805 * [DB IDO](13-distributed-monitoring-ha.md#high-availability-db-ido) (Run-Once, automated failover)
807 ### <a id="high-availability-checks"></a> High Availability with Checks
809 All instances within the same zone (e.g. the `master` zone as HA cluster) must
810 have the `checker` feature enabled.
814 # icinga2 feature enable checker
816 All nodes in the same zone load-balance the check execution. When one instance shuts down
817 the other nodes will automatically take over the reamining checks.
819 ### <a id="high-availability-notifications"></a> High Availability with Notifications
821 All instances within the same zone (e.g. the `master` zone as HA cluster) must
822 have the `notification` feature enabled.
826 # icinga2 feature enable notification
828 Notifications are load balanced amongst all nodes in a zone. By default this functionality
830 If your nodes should notify independent from any other nodes (this will cause
831 duplicated notifications if not properly handled!), you can set `enable_ha = false`
832 in the [NotificationComponent](6-object-types.md#objecttype-notificationcomponent) feature.
834 ### <a id="high-availability-db-ido"></a> High Availability with DB IDO
836 All instances within the same zone (e.g. the `master` zone as HA cluster) must
837 have the DB IDO feature enabled.
839 Example DB IDO MySQL:
841 # icinga2 feature enable ido-mysql
843 By default the DB IDO feature only runs on one node. All other nodes in the same zone disable
844 the active IDO database connection at runtime. The node with the active DB IDO connection is
845 not necessarily the zone master.
849 > The DB IDO HA feature can be disabled by setting the `enable_ha` attribute to `false`
850 > for the [IdoMysqlConnection](6-object-types.md#objecttype-idomysqlconnection) or
851 > [IdoPgsqlConnection](6-object-types.md#objecttype-idopgsqlconnection) object on **all** nodes in the
854 > All endpoints will enable the DB IDO feature and connect to the configured
855 > database and dump configuration, status and historical data on their own.
857 If the instance with the active DB IDO connection dies, the HA functionality will
858 automatically elect a new DB IDO master.
860 The DB IDO feature will try to determine which cluster endpoint is currently writing
861 to the database and bail out if another endpoint is active. You can manually verify that
862 by running the following query:
864 icinga=> SELECT status_update_time, endpoint_name FROM icinga_programstatus;
865 status_update_time | endpoint_name
866 ------------------------+---------------
867 2014-08-15 15:52:26+02 | icinga2a
870 This is useful when the cluster connection between endpoints breaks, and prevents
871 data duplication in split-brain-scenarios. The failover timeout can be set for the
872 `failover_timeout` attribute, but not lower than 60 seconds.
875 ## <a id="cluster-add-node"></a> Add a new cluster endpoint
877 These steps are required for integrating a new cluster endpoint:
879 * generate a new [SSL client certificate](13-distributed-monitoring-ha.md#manual-certificate-generation)
880 * identify its location in the zones
881 * update the `zones.conf` file on each involved node ([endpoint](13-distributed-monitoring-ha.md#configure-cluster-endpoints), [zones](13-distributed-monitoring-ha.md#configure-cluster-zones))
882 * a new slave zone node requires updates for the master and slave zones
883 * verify if this endpoints requires [configuration synchronisation](13-distributed-monitoring-ha.md#cluster-zone-config-sync) enabled
884 * if the node requires the existing zone history: [initial cluster sync](13-distributed-monitoring-ha.md#initial-cluster-sync)
885 * add a [cluster health check](13-distributed-monitoring-ha.md#cluster-health-check)
887 ### <a id="initial-cluster-sync"></a> Initial Cluster Sync
889 In order to make sure that all of your cluster nodes have the same state you will
890 have to pick one of the nodes as your initial "master" and copy its state file
891 to all the other nodes.
893 You can find the state file in `/var/lib/icinga2/icinga2.state`. Before copying
894 the state file you should make sure that all your cluster nodes are properly shut
898 ## <a id="host-multiple-cluster-nodes"></a> Host With Multiple Cluster Nodes
900 Special scenarios might require multiple cluster nodes running on a single host.
901 By default Icinga 2 and its features will place their runtime data below the prefix
902 `LocalStateDir`. By default packages will set that path to `/var`.
903 You can either set that variable as constant configuration
904 definition in [icinga2.conf](4-configuring-icinga-2.md#icinga2-conf) or pass it as runtime variable to
907 # icinga2 -c /etc/icinga2/node1/icinga2.conf -DLocalStateDir=/opt/node1/var