1 # <a id="distributed-monitoring-high-availability"></a> Distributed Monitoring and High Availability
3 Building distributed environments with high availability included is fairly easy with Icinga 2.
4 The cluster feature is built-in and allows you to build many scenarios based on your requirements:
6 * [High Availability](12-distributed-monitoring-ha.md#cluster-scenarios-high-availability). All instances in the `Zone` run as Active/Active cluster.
7 * [Distributed Zones](12-distributed-monitoring-ha.md#cluster-scenarios-distributed-zones). A master zone and one or more satellites in their zones.
8 * [Load Distribution](12-distributed-monitoring-ha.md#cluster-scenarios-load-distribution). A configuration master and multiple checker satellites.
10 You can combine these scenarios into a global setup fitting your requirements.
12 Each instance got their own event scheduler, and does not depend on a centralized master
13 coordinating and distributing the events. In case of a cluster failure, all nodes
14 continue to run independently. Be alarmed when your cluster fails and a Split-Brain-scenario
15 is in effect - all alive instances continue to do their job, and history will begin to differ.
18 ## <a id="cluster-requirements"></a> Cluster Requirements
20 Before you start deploying, keep the following things in mind:
22 * Your [SSL CA and certificates](12-distributed-monitoring-ha.md#manual-certificate-generation) are mandatory for secure communication
23 * Get pen and paper or a drawing board and design your nodes and zones!
24 * all nodes in a cluster zone are providing high availability functionality and trust each other
25 * cluster zones can be built in a Top-Down-design where the child trusts the parent
26 * communication between zones happens bi-directional which means that a DMZ-located node can still reach the master node, or vice versa
27 * Update firewall rules and ACLs
28 * Decide whether to use the built-in [configuration syncronization](12-distributed-monitoring-ha.md#cluster-zone-config-sync) or use an external tool (Puppet, Ansible, Chef, Salt, etc) to manage the configuration deployment
33 > If you're looking for troubleshooting cluster problems, check the general
34 > [troubleshooting](16-troubleshooting.md#troubleshooting-cluster) section.
36 ## <a id="manual-certificate-generation"></a> Manual SSL Certificate Generation
38 Icinga 2 provides [CLI commands](8-cli-commands.md#cli-command-pki) assisting with CA
39 and node certificate creation for your Icinga 2 distributed setup.
43 > You can also use the master and client setup wizards to install the cluster nodes
44 > using CSR-Autosigning.
46 > The manual steps are helpful if you want to use your own and/or existing CA (for example
51 > You're free to use your own method to generated a valid ca and signed client
54 The first step is the creation of the certificate authority (CA) by running the
59 Now create a certificate and key file for each node running the following command
60 (replace `icinga2a` with the required hostname):
62 # icinga2 pki new-cert --cn icinga2a --key icinga2a.key --csr icinga2a.csr
63 # icinga2 pki sign-csr --csr icinga2a.csr --cert icinga2a.crt
65 Repeat the step for all nodes in your cluster scenario.
67 Save the CA key in a secure location in case you want to set up certificates for
68 additional nodes at a later time.
70 Navigate to the location of your newly generated certificate files, and manually
71 copy/transfer them to `/etc/icinga2/pki` in your Icinga 2 configuration folder.
75 > The certificate files must be readable by the user Icinga 2 is running as. Also,
76 > the private key file must not be world-readable.
78 Each node requires the following files in `/etc/icinga2/pki` (replace `fqdn-nodename` with
82 * <fqdn-nodename>.crt
83 * <fqdn-nodename>.key
85 If you're planning to use your existing CA and certificates please note that you *must not*
86 use wildcard certificates. The common name (CN) is mandatory for the cluster communication and
87 therefore must be unique for each connecting instance.
89 ### <a id="cluster-naming-convention"></a> Cluster Naming Convention
91 The SSL certificate common name (CN) will be used by the [ApiListener](6-object-types.md#objecttype-apilistener)
92 object to determine the local authority. This name must match the local [Endpoint](6-object-types.md#objecttype-endpoint)
97 # icinga2 pki new-cert --cn icinga2a --key icinga2a.key --csr icinga2a.csr
98 # icinga2 pki sign-csr --csr icinga2a.csr --cert icinga2a.crt
102 object Endpoint "icinga2a" {
103 host = "icinga2a.icinga.org"
106 The [Endpoint](6-object-types.md#objecttype-endpoint) name is further referenced as `endpoints` attribute on the
107 [Zone](6-object-types.md#objecttype-zone) object.
109 object Endpoint "icinga2b" {
110 host = "icinga2b.icinga.org"
113 object Zone "config-ha-master" {
114 endpoints = [ "icinga2a", "icinga2b" ]
117 Specifying the local node name using the [NodeName](12-distributed-monitoring-ha.md#configure-nodename) variable requires
118 the same name as used for the endpoint name and common name above. If not set, the FQDN is used.
120 const NodeName = "icinga2a"
123 ## <a id="cluster-configuration"></a> Cluster Configuration
125 The following section describe which configuration must be updated/created
126 in order to get your cluster running with basic functionality.
128 * [configure the node name](12-distributed-monitoring-ha.md#configure-nodename)
129 * [configure the ApiListener object](12-distributed-monitoring-ha.md#configure-apilistener-object)
130 * [configure cluster endpoints](12-distributed-monitoring-ha.md#configure-cluster-endpoints)
131 * [configure cluster zones](12-distributed-monitoring-ha.md#configure-cluster-zones)
133 Once you're finished with the basic setup the following section will
134 describe how to use [zone configuration synchronisation](12-distributed-monitoring-ha.md#cluster-zone-config-sync)
135 and configure [cluster scenarios](12-distributed-monitoring-ha.md#cluster-scenarios).
137 ### <a id="configure-nodename"></a> Configure the Icinga Node Name
139 Instead of using the default FQDN as node name you can optionally set
140 that value using the [NodeName](19-language-reference.md#constants) constant.
144 > Skip this step if your FQDN already matches the default `NodeName` set
145 > in `/etc/icinga2/constants.conf`.
147 This setting must be unique for each node, and must also match
148 the name of the local [Endpoint](6-object-types.md#objecttype-endpoint) object and the
149 SSL certificate common name as described in the
150 [cluster naming convention](12-distributed-monitoring-ha.md#cluster-naming-convention).
152 vim /etc/icinga2/constants.conf
154 /* Our local instance name. By default this is the server's hostname as returned by `hostname --fqdn`.
155 * This should be the common name from the API certificate.
157 const NodeName = "icinga2a"
160 Read further about additional [naming conventions](12-distributed-monitoring-ha.md#cluster-naming-convention).
162 Not specifying the node name will make Icinga 2 using the FQDN. Make sure that all
163 configured endpoint names and common names are in sync.
165 ### <a id="configure-apilistener-object"></a> Configure the ApiListener Object
167 The [ApiListener](6-object-types.md#objecttype-apilistener) object needs to be configured on
168 every node in the cluster with the following settings:
170 A sample config looks like:
172 object ApiListener "api" {
173 cert_path = SysconfDir + "/icinga2/pki/" + NodeName + ".crt"
174 key_path = SysconfDir + "/icinga2/pki/" + NodeName + ".key"
175 ca_path = SysconfDir + "/icinga2/pki/ca.crt"
177 accept_commands = true
180 You can simply enable the `api` feature using
182 # icinga2 feature enable api
184 Edit `/etc/icinga2/features-enabled/api.conf` if you require the configuration
185 synchronisation enabled for this node. Set the `accept_config` attribute to `true`.
187 If you want to use this node as [remote client for command execution](10-icinga2-client.md#icinga2-client-configuration-command-bridge)
188 set the `accept_commands` attribute to `true`.
192 > The certificate files must be readable by the user Icinga 2 is running as. Also,
193 > the private key file must not be world-readable.
195 ### <a id="configure-cluster-endpoints"></a> Configure Cluster Endpoints
197 `Endpoint` objects specify the `host` and `port` settings for the cluster node
198 connection information.
199 This configuration can be the same on all nodes in the cluster only containing
200 connection information.
202 A sample configuration looks like:
205 * Configure config master endpoint
208 object Endpoint "icinga2a" {
209 host = "icinga2a.icinga.org"
212 If this endpoint object is reachable on a different port, you must configure the
213 `ApiListener` on the local `Endpoint` object accordingly too.
215 If you don't want the local instance to connect to the remote instance, remove the
216 `host` attribute locally. Keep in mind that the configuration is now different amongst
217 all instances and point-of-view dependant.
219 ### <a id="configure-cluster-zones"></a> Configure Cluster Zones
221 `Zone` objects specify the endpoints located in a zone. That way your distributed setup can be
222 seen as zones connected together instead of multiple instances in that specific zone.
224 Zones can be used for [high availability](12-distributed-monitoring-ha.md#cluster-scenarios-high-availability),
225 [distributed setups](12-distributed-monitoring-ha.md#cluster-scenarios-distributed-zones) and
226 [load distribution](12-distributed-monitoring-ha.md#cluster-scenarios-load-distribution).
227 Furthermore zones are used for the [Icinga 2 remote client](10-icinga2-client.md#icinga2-client).
229 Each Icinga 2 `Endpoint` must be put into its respective `Zone`. In this example, you will
230 define the zone `config-ha-master` where the `icinga2a` and `icinga2b` endpoints
231 are located. The `check-satellite` zone consists of `icinga2c` only, but more nodes could
234 The `config-ha-master` zone acts as High-Availability setup - the Icinga 2 instances elect
235 one instance running a check, notification or feature (DB IDO), for example `icinga2a`. In case of
236 failure of the `icinga2a` instance, `icinga2b` will take over automatically.
238 object Zone "config-ha-master" {
239 endpoints = [ "icinga2a", "icinga2b" ]
242 The `check-satellite` zone is a separated location and only sends back their checkresults to
243 the defined parent zone `config-ha-master`.
245 object Zone "check-satellite" {
246 endpoints = [ "icinga2c" ]
247 parent = "config-ha-master"
251 ## <a id="cluster-zone-config-sync"></a> Zone Configuration Synchronisation
253 By default all objects for specific zones should be organized in
255 /etc/icinga2/zones.d/<zonename>
257 on the configuration master.
259 Your child zones and endpoint members **must not** have their config copied to `zones.d`.
260 The built-in configuration synchronisation takes care of that if your nodes accept
261 configuration from the parent zone. You can define that in the
262 [ApiListener](12-distributed-monitoring-ha.md#configure-apilistener-object) object by configuring the `accept_config`
263 attribute accordingly.
265 You should remove the sample config included in `conf.d` by commenting the `recursive_include`
266 statement in [icinga2.conf](5-configuring-icinga-2.md#icinga2-conf):
268 //include_recursive "conf.d"
270 Better use a dedicated directory name like `cluster` or similar, and include that
271 one if your nodes require local configuration not being synced to other nodes. That's
272 useful for local [health checks](12-distributed-monitoring-ha.md#cluster-health-check) for example.
276 > In a [high availability](12-distributed-monitoring-ha.md#cluster-scenarios-high-availability)
277 > setup only one assigned node can act as configuration master. All other zone
278 > member nodes **must not** have the `/etc/icinga2/zones.d` directory populated.
280 These zone packages are then distributed to all nodes in the same zone, and
281 to their respective target zone instances.
283 Each configured zone must exist with the same directory name. The parent zone
284 syncs the configuration to the child zones, if allowed using the `accept_config`
285 attribute of the [ApiListener](12-distributed-monitoring-ha.md#configure-apilistener-object) object.
287 Config on node `icinga2a`:
289 object Zone "master" {
290 endpoints = [ "icinga2a" ]
293 object Zone "checker" {
294 endpoints = [ "icinga2b" ]
305 Config on node `icinga2b`:
307 object Zone "master" {
308 endpoints = [ "icinga2a" ]
311 object Zone "checker" {
312 endpoints = [ "icinga2b" ]
317 EMPTY_IF_CONFIG_SYNC_ENABLED
319 If the local configuration is newer than the received update Icinga 2 will skip the synchronisation
324 > `zones.d` must not be included in [icinga2.conf](5-configuring-icinga-2.md#icinga2-conf). Icinga 2 automatically
325 > determines the required include directory. This can be overridden using the
326 > [global constant](19-language-reference.md#constants) `ZonesDir`.
328 ### <a id="zone-global-config-templates"></a> Global Configuration Zone for Templates
330 If your zone configuration setup shares the same templates, groups, commands, timeperiods, etc.
331 you would have to duplicate quite a lot of configuration objects making the merged configuration
332 on your configuration master unique.
336 > Only put templates, groups, etc into this zone. DO NOT add checkable objects such as
337 > hosts or services here. If they are checked by all instances globally, this will lead
338 > into duplicated check results and unclear state history. Not easy to troubleshoot too -
339 > you've been warned.
341 That is not necessary by defining a global zone shipping all those templates. By setting
342 `global = true` you ensure that this zone serving common configuration templates will be
343 synchronized to all involved nodes (only if they accept configuration though).
345 Config on configuration master:
357 In this example, the global zone is called `global-templates` and must be defined in
358 your zone configuration visible to all nodes.
360 object Zone "global-templates" {
366 > If the remote node does not have this zone configured, it will ignore the configuration
367 > update, if it accepts synchronized configuration.
369 If you don't require any global configuration, skip this setting.
371 ### <a id="zone-config-sync-permissions"></a> Zone Configuration Synchronisation Permissions
373 Each [ApiListener](6-object-types.md#objecttype-apilistener) object must have the `accept_config` attribute
374 set to `true` to receive configuration from the parent `Zone` members. Default value is `false`.
376 object ApiListener "api" {
377 cert_path = SysconfDir + "/icinga2/pki/" + NodeName + ".crt"
378 key_path = SysconfDir + "/icinga2/pki/" + NodeName + ".key"
379 ca_path = SysconfDir + "/icinga2/pki/ca.crt"
383 If `accept_config` is set to `false`, this instance won't accept configuration from remote
384 master instances anymore.
388 > Look into the [troubleshooting guides](16-troubleshooting.md#troubleshooting-cluster-config-sync) for debugging
389 > problems with the configuration synchronisation.
392 ## <a id="cluster-health-check"></a> Cluster Health Check
394 The Icinga 2 [ITL](7-icinga-template-library.md#icinga-template-library) ships an internal check command checking all configured
395 `EndPoints` in the cluster setup. The check result will become critical if
396 one or more configured nodes are not connected.
400 object Service "cluster" {
401 check_command = "cluster"
405 host_name = "icinga2a"
408 Each cluster node should execute its own local cluster health check to
409 get an idea about network related connection problems from different
412 Additionally you can monitor the connection from the local zone to the remote
415 Example for the `checker` zone checking the connection to the `master` zone:
417 object Service "cluster-zone-master" {
418 check_command = "cluster-zone"
421 vars.cluster_zone = "master"
423 host_name = "icinga2b"
427 ## <a id="cluster-scenarios"></a> Cluster Scenarios
429 All cluster nodes are full-featured Icinga 2 instances. You only need to enabled
430 the features for their role (for example, a `Checker` node only requires the `checker`
431 feature enabled, but not `notification` or `ido-mysql` features).
435 > There's a [Vagrant demo setup](https://github.com/Icinga/icinga-vagrant/tree/master/icinga2x-cluster)
436 > available featuring a two node cluster showcasing several aspects (config sync,
437 > remote command execution, etc).
439 ### <a id="cluster-scenarios-master-satellite-clients"></a> Cluster with Master, Satellites and Remote Clients
441 You can combine "classic" cluster scenarios from HA to Master-Checker with the
442 Icinga 2 Remote Client modes. Each instance plays a certain role in that picture.
444 Imagine the following scenario:
446 * The master zone acts as High-Availability zone
447 * Remote satellite zones execute local checks and report them to the master
448 * All satellites query remote clients and receive check results (which they also replay to the master)
449 * All involved nodes share the same configuration logic: zones, endpoints, apilisteners
451 You'll need to think about the following:
453 * Deploy the entire configuration from the master to satellites and cascading remote clients? ("top down")
454 * Use local client configuration instead and report the inventory to satellites and cascading to the master? ("bottom up")
455 * Combine that with command execution brdiges on remote clients and also satellites
460 ### <a id="cluster-scenarios-security"></a> Security in Cluster Scenarios
462 While there are certain capabilities to ensure the safe communication between all
463 nodes (firewalls, policies, software hardening, etc) the Icinga 2 cluster also provides
464 additional security itself:
466 * [SSL certificates](12-distributed-monitoring-ha.md#manual-certificate-generation) are mandatory for cluster communication.
467 * Child zones only receive event updates (check results, commands, etc) for their configured updates.
468 * Zones cannot influence/interfere other zones. Each checked object is assigned to only one zone.
469 * All nodes in a zone trust each other.
470 * [Configuration sync](12-distributed-monitoring-ha.md#zone-config-sync-permissions) is disabled by default.
472 ### <a id="cluster-scenarios-features"></a> Features in Cluster Zones
474 Each cluster zone may use all available features. If you have multiple locations
475 or departments, they may write to their local database, or populate graphite.
476 Even further all commands are distributed amongst connected nodes. For example, you could
477 re-schedule a check or acknowledge a problem on the master, and it gets replicated to the
478 actual slave checker node.
480 DB IDO on the left, graphite on the right side - works (if you disable
481 [DB IDO HA](12-distributed-monitoring-ha.md#high-availability-db-ido)).
482 Icinga Web 2 on the left, checker and notifications on the right side - works too.
483 Everything on the left and on the right side - make sure to deal with
484 [load-balanced notifications and checks](12-distributed-monitoring-ha.md#high-availability-features) in a
485 [HA zone](12-distributed-monitoring-ha.md#cluster-scenarios-high-availability).
488 ### <a id="cluster-scenarios-distributed-zones"></a> Distributed Zones
490 That scenario fits if your instances are spread over the globe and they all report
491 to a master instance. Their network connection only works towards the master master
492 (or the master is able to connect, depending on firewall policies) which means
493 remote instances won't see each/connect to each other.
495 All events (check results, downtimes, comments, etc) are synced to the master node,
496 but the remote nodes can still run local features such as a web interface, reporting,
497 graphing, etc. in their own specified zone.
499 Imagine the following example with a master node in Nuremberg, and two remote DMZ
500 based instances in Berlin and Vienna. Additonally you'll specify
501 [global templates](12-distributed-monitoring-ha.md#zone-global-config-templates) available in all zones.
503 The configuration tree on the master instance `nuremberg` could look like this:
516 The configuration deployment will take care of automatically synchronising
517 the child zone configuration:
519 * The master node sends `zones.d/berlin` to the `berlin` child zone.
520 * The master node sends `zones.d/vienna` to the `vienna` child zone.
521 * The master node sends `zones.d/global-templates` to the `vienna` and `berlin` child zones.
523 The endpoint configuration would look like:
525 object Endpoint "nuremberg-master" {
526 host = "nuremberg.icinga.org"
529 object Endpoint "berlin-satellite" {
530 host = "berlin.icinga.org"
533 object Endpoint "vienna-satellite" {
534 host = "vienna.icinga.org"
537 The zones would look like:
539 object Zone "nuremberg" {
540 endpoints = [ "nuremberg-master" ]
543 object Zone "berlin" {
544 endpoints = [ "berlin-satellite" ]
548 object Zone "vienna" {
549 endpoints = [ "vienna-satellite" ]
553 object Zone "global-templates" {
557 The `nuremberg-master` zone will only execute local checks, and receive
558 check results from the satellite nodes in the zones `berlin` and `vienna`.
562 > The child zones `berlin` and `vienna` will get their configuration synchronised
563 > from the configuration master 'nuremberg'. The endpoints in the child
564 > zones **must not** have their `zones.d` directory populated if this endpoint
565 > [accepts synced configuration](12-distributed-monitoring-ha.md#zone-config-sync-permissions).
567 ### <a id="cluster-scenarios-load-distribution"></a> Load Distribution
569 If you are planning to off-load the checks to a defined set of remote workers
570 you can achieve that by:
572 * Deploying the configuration on all nodes.
573 * Let Icinga 2 distribute the load amongst all available nodes.
575 That way all remote check instances will receive the same configuration
576 but only execute their part. The master instance located in the `master` zone
577 can also execute checks, but you may also disable the `Checker` feature.
579 Configuration on the master node:
586 If you are planning to have some checks executed by a specific set of checker nodes
587 you have to define additional zones and define these check objects there.
591 object Endpoint "master-node" {
592 host = "master.icinga.org"
595 object Endpoint "checker1-node" {
596 host = "checker1.icinga.org"
599 object Endpoint "checker2-node" {
600 host = "checker2.icinga.org"
606 object Zone "master" {
607 endpoints = [ "master-node" ]
610 object Zone "checker" {
611 endpoints = [ "checker1-node", "checker2-node" ]
615 object Zone "global-templates" {
621 > The child zones `checker` will get its configuration synchronised
622 > from the configuration master 'master'. The endpoints in the child
623 > zone **must not** have their `zones.d` directory populated if this endpoint
624 > [accepts synced configuration](12-distributed-monitoring-ha.md#zone-config-sync-permissions).
626 ### <a id="cluster-scenarios-high-availability"></a> Cluster High Availability
628 High availability with Icinga 2 is possible by putting multiple nodes into
629 a dedicated [zone](12-distributed-monitoring-ha.md#configure-cluster-zones). All nodes will elect one
630 active master, and retry an election once the current active master is down.
632 Selected features provide advanced [HA functionality](12-distributed-monitoring-ha.md#high-availability-features).
633 Checks and notifications are load-balanced between nodes in the high availability
636 Connections from other zones will be accepted by all active and passive nodes
637 but all are forwarded to the current active master dealing with the check results,
640 object Zone "config-ha-master" {
641 endpoints = [ "icinga2a", "icinga2b", "icinga2c" ]
644 Two or more nodes in a high availability setup require an [initial cluster sync](12-distributed-monitoring-ha.md#initial-cluster-sync).
648 > Keep in mind that **only one node acts as configuration master** having the
649 > configuration files in the `zones.d` directory. All other nodes **must not**
650 > have that directory populated. Instead they are required to
651 > [accept synced configuration](12-distributed-monitoring-ha.md#zone-config-sync-permissions).
652 > Details in the [Configuration Sync Chapter](12-distributed-monitoring-ha.md#cluster-zone-config-sync).
654 ### <a id="cluster-scenarios-multiple-hierarchies"></a> Multiple Hierarchies
656 Your master zone collects all check results for reporting and graphing and also
657 does some sort of additional notifications.
658 The customers got their own instances in their local DMZ zones. They are limited to read/write
659 only their services, but replicate all events back to the master instance.
660 Within each DMZ there are additional check instances also serving interfaces for local
661 departments. The customers instances will collect all results, but also send them back to
662 your master instance.
663 Additionally the customers instance on the second level in the middle prohibits you from
664 sending commands to the subjacent department nodes. You're only allowed to receive the
665 results, and a subset of each customers configuration too.
667 Your master zone will generate global reports, aggregate alert notifications, and check
668 additional dependencies (for example, the customers internet uplink and bandwidth usage).
670 The customers zone instances will only check a subset of local services and delegate the rest
671 to each department. Even though it acts as configuration master with a master dashboard
672 for all departments managing their configuration tree which is then deployed to all
673 department instances. Furthermore the master NOC is able to see what's going on.
675 The instances in the departments will serve a local interface, and allow the administrators
676 to reschedule checks or acknowledge problems for their services.
679 ## <a id="high-availability-features"></a> High Availability for Icinga 2 features
681 All nodes in the same zone require the same features enabled for High Availability (HA)
684 By default the following features provide advanced HA functionality:
686 * [Checks](12-distributed-monitoring-ha.md#high-availability-checks) (load balanced, automated failover)
687 * [Notifications](12-distributed-monitoring-ha.md#high-availability-notifications) (load balanced, automated failover)
688 * [DB IDO](12-distributed-monitoring-ha.md#high-availability-db-ido) (Run-Once, automated failover)
690 ### <a id="high-availability-checks"></a> High Availability with Checks
692 All nodes in the same zone load-balance the check execution. When one instance
693 fails the other nodes will automatically take over the reamining checks.
697 > If a node should not check anything, disable the `checker` feature explicitely and
700 # icinga2 feature disable checker
701 # service icinga2 reload
703 ### <a id="high-availability-notifications"></a> High Availability with Notifications
705 Notifications are load balanced amongst all nodes in a zone. By default this functionality
707 If your nodes should notify independent from any other nodes (this will cause
708 duplicated notifications if not properly handled!), you can set `enable_ha = false`
709 in the [NotificationComponent](6-object-types.md#objecttype-notificationcomponent) feature.
711 ### <a id="high-availability-db-ido"></a> High Availability with DB IDO
713 All instances within the same zone (e.g. the `master` zone as HA cluster) must
714 have the DB IDO feature enabled.
716 Example DB IDO MySQL:
718 # icinga2 feature enable ido-mysql
719 The feature 'ido-mysql' is already enabled.
721 By default the DB IDO feature only runs on one node. All other nodes in the same zone disable
722 the active IDO database connection at runtime. The node with the active DB IDO connection is
723 not necessarily the zone master.
727 > The DB IDO HA feature can be disabled by setting the `enable_ha` attribute to `false`
728 > for the [IdoMysqlConnection](6-object-types.md#objecttype-idomysqlconnection) or
729 > [IdoPgsqlConnection](6-object-types.md#objecttype-idopgsqlconnection) object on **all** nodes in the
732 > All endpoints will enable the DB IDO feature and connect to the configured
733 > database and dump configuration, status and historical data on their own.
735 If the instance with the active DB IDO connection dies, the HA functionality will
736 automatically elect a new DB IDO master.
738 The DB IDO feature will try to determine which cluster endpoint is currently writing
739 to the database and bail out if another endpoint is active. You can manually verify that
740 by running the following query:
742 icinga=> SELECT status_update_time, endpoint_name FROM icinga_programstatus;
743 status_update_time | endpoint_name
744 ------------------------+---------------
745 2014-08-15 15:52:26+02 | icinga2a
748 This is useful when the cluster connection between endpoints breaks, and prevents
749 data duplication in split-brain-scenarios. The failover timeout can be set for the
750 `failover_timeout` attribute, but not lower than 60 seconds.
753 ## <a id="cluster-add-node"></a> Add a new cluster endpoint
755 These steps are required for integrating a new cluster endpoint:
757 * generate a new [SSL client certificate](12-distributed-monitoring-ha.md#manual-certificate-generation)
758 * identify its location in the zones
759 * update the `zones.conf` file on each involved node ([endpoint](12-distributed-monitoring-ha.md#configure-cluster-endpoints), [zones](12-distributed-monitoring-ha.md#configure-cluster-zones))
760 * a new slave zone node requires updates for the master and slave zones
761 * verify if this endpoints requires [configuration synchronisation](12-distributed-monitoring-ha.md#cluster-zone-config-sync) enabled
762 * if the node requires the existing zone history: [initial cluster sync](12-distributed-monitoring-ha.md#initial-cluster-sync)
763 * add a [cluster health check](12-distributed-monitoring-ha.md#cluster-health-check)
765 ### <a id="initial-cluster-sync"></a> Initial Cluster Sync
767 In order to make sure that all of your cluster nodes have the same state you will
768 have to pick one of the nodes as your initial "master" and copy its state file
769 to all the other nodes.
771 You can find the state file in `/var/lib/icinga2/icinga2.state`. Before copying
772 the state file you should make sure that all your cluster nodes are properly shut
776 ## <a id="host-multiple-cluster-nodes"></a> Host With Multiple Cluster Nodes
778 Special scenarios might require multiple cluster nodes running on a single host.
779 By default Icinga 2 and its features will place their runtime data below the prefix
780 `LocalStateDir`. By default packages will set that path to `/var`.
781 You can either set that variable as constant configuration
782 definition in [icinga2.conf](5-configuring-icinga-2.md#icinga2-conf) or pass it as runtime variable to
785 # icinga2 -c /etc/icinga2/node1/icinga2.conf -DLocalStateDir=/opt/node1/var