1 ## <a id="cluster"></a> Cluster
3 An Icinga 2 cluster consists of two or more nodes and can reside on multiple
4 architectures. The base concept of Icinga 2 is the possibility to add additional
5 features using components. In case of a cluster setup you have to add the
6 cluster feature to all nodes. Before you start configuring the diffent nodes
7 it's necessary to setup the underlying communication layer based on SSL.
9 ### <a id="certificate-authority-certificates"></a> Certificate Authority and Certificates
11 Icinga 2 comes with two scripts helping you to create CA and node certificates
12 for you Icinga 2 Cluster.
14 The first step is the creation of CA using the following command:
18 Please make sure to export a variable containing an empty folder for the created
21 export ICINGA_CA="/root/icinga-ca"
23 In the next step you have to create a certificate and a key file for every node
24 using the following command:
26 icinga2-build-key icinga-node-1
28 Please create a certificate and a key file for every node in the Icinga 2
29 Cluster and save the CA key in case you want to set up certificates for
30 additional nodes at a later date.
32 ### <a id="enable-cluster-configuration"></a> Enable the Cluster Configuration
34 Until the cluster-component is moved into an independent feature you have to
35 enable the required libraries in the icinga2.conf configuration file:
39 ### <a id="configure-nodename"></a> Configure the Icinga Node Name
41 Instead of using the default FQDN as node name you can optionally set
42 that value using the [NodeName](#global-constants) constant.
43 This setting must be unique on each cluster node, and must also match
44 the name of the local [Endpoint](#objecttype-endpoint) object and the
45 SSL certificate common name.
47 const NodeName = "icinga-node-1"
49 Read further about additional [naming conventions](#cluster-naming-convention).
51 Not specifying the node name will default to FQDN. Make sure that all
52 configured endpoint names and set common names are in sync.
54 ### <a id="configure-clusterlistener-object"></a> Configure the ClusterListener Object
56 The ClusterListener needs to be configured on every node in the cluster with the
59 Configuration Setting |Value
60 -------------------------|------------------------------------
61 ca_path | path to ca.crt file
62 cert_path | path to server certificate
63 key_path | path to server key
64 bind_port | port for incoming and outgoing conns
65 peers | array of all reachable nodes
66 ------------------------- ------------------------------------
68 A sample config part can look like this:
71 * Load cluster library and configure ClusterListener using certificate files
75 object ClusterListener "cluster" {
76 ca_path = "/etc/icinga2/ca/ca.crt"
77 cert_path = "/etc/icinga2/ca/icinga-node-1.crt"
78 key_path = "/etc/icinga2/ca/icinga-node-1.key"
82 peers = [ "icinga-node-2" ]
85 The certificate files must be readable by the user Icinga 2 is running as. Also,
86 the private key file should not be world-readable.
88 Peers configures the direction used to connect multiple nodes together. If have
89 a three node cluster consisting of
95 and `node-3` is only reachable from `node-2`, you have to consider this in your
98 ### <a id="configure-cluster-endpoints"></a> Configure Cluster Endpoints
100 In addition to the configured port and hostname every endpoint can have specific
101 abilities to send configuration files to other nodes and limit the hosts allowed
102 to send configuration files.
104 Configuration Setting |Value
105 -------------------------|------------------------------------
108 accept_config | defines all nodes allowed to send configs
109 config_files | defines all files to be send to that node - MUST BE AN ABSOLUTE PATH
110 ------------------------- ------------------------------------
112 A sample config part can look like this:
115 * Configure config master endpoint
118 object Endpoint "icinga-node-1" {
119 host = "icinga-node-1.localdomain"
121 config_files = ["/etc/icinga2/conf.d/*.conf"]
124 If you update the configuration files on the configured file sender, it will
125 force a restart on all receiving nodes after validating the new config.
127 A sample config part for a config receiver endpoint can look like this:
130 * Configure config receiver endpoint
133 object Endpoint "icinga-node-2" {
134 host = "icinga-node-2.localdomain"
136 accept_config = [ "icinga-node-1" ]
139 By default these configuration files are saved in /var/lib/icinga2/cluster/config.
141 In order to load configuration files which were received from a remote Icinga 2
142 instance you will have to add the following include directive to your
143 `icinga2.conf` configuration file:
145 include_recursive LocalStateDir + "/lib/icinga2/cluster/config"
147 ### <a id="cluster-naming-convention"></a> Cluster Naming Convention
149 The SSL certificate common name (CN) will be used by the [ClusterListener](pbjecttype-clusterlistener)
150 object to determine the local authority. This name must match the local [Endpoint](#objecttype-endpoint)
155 # icinga2-build-key icinga-node-1
157 Common Name (e.g. server FQDN or YOUR name) [icinga-node-1]:
161 object Endpoint "icinga-node-1" {
162 host = "icinga-node-1.localdomain"
166 The [Endpoint](#objecttype-endpoint) name is further referenced as `peers` attribute on the
167 [ClusterListener](pbjecttype-clusterlistener) object.
169 object Endpoint "icinga-node-2" {
170 host = "icinga-node-2.localdomain"
174 object ClusterListener "cluster" {
175 ca_path = "/etc/icinga2/ca/ca.crt"
176 cert_path = "/etc/icinga2/ca/icinga-node-1.crt"
177 key_path = "/etc/icinga2/ca/icinga-node-1.key"
181 peers = [ "icinga-node-2" ]
184 Specifying the local node name using the [NodeName](#global-constants) variable requires
185 the same name as used for the endpoint name and common name above. If not set, the FQDN is used.
187 const NodeName = "icinga-node-1"
190 ### <a id="initial-cluster-sync"></a> Initial Cluster Sync
192 In order to make sure that all of your cluster nodes have the same state you will
193 have to pick one of the nodes as your initial "master" and copy its state file
194 to all the other nodes.
196 You can find the state file in `/var/lib/icinga2/icinga2.state`. Before copying
197 the state file you should make sure that all your cluster nodes are properly shut
201 ### <a id="assign-services-to-cluster-nodes"></a> Assign Services to Cluster Nodes
203 By default all services are distributed among the cluster nodes with the `Checker`
205 If you require specific services to be only executed by one or more checker nodes
206 within the cluster, you must define `authorities` as additional service object
207 attribute. Required Endpoints must be defined as array.
209 apply Service "dmz-oracledb" {
210 import "generic-service"
212 authorities = [ "icinga-node-1" ]
214 assign where "oracle" in host.groups
217 The most common use case is building a master-slave cluster. The master node
218 does not have the `checker` feature enabled, and the slave nodes are checking
219 services based on their location, inheriting from a global service template
220 defining the authorities.
222 ### <a id="cluster-health-check"></a> Cluster Health Check
224 The Icinga 2 [ITL](#itl) ships an internal check command checking all configured
225 `EndPoints` in the cluster setup. The check result will become critical if
226 one or more configured nodes are not connected.
230 apply Service "cluster" {
231 import "generic-service"
234 check_command = "cluster"
235 authorities = [ "icinga2a" ]
237 assign where host.name = "icinga2a"
240 Each cluster node should execute its own local cluster health check to
241 get an idea about network related connection problems from different
242 point of views. Use the `authorities` attribute to assign the service
243 check to the configured node.
245 ### <a id="host-multiple-cluster-nodes"></a> Host With Multiple Cluster Nodes
247 Special scenarios might require multiple cluster nodes running on a single host.
248 By default Icinga 2 and its features will drop their runtime data below the prefix
249 `LocalStateDir`. By default packages will set that path to `/var`.
250 You can either set that variable as constant configuration
251 definition in [icinga2.conf](#icinga2-conf) or pass it as runtime variable to
254 # icinga2 -c /etc/icinga2/node1/icinga2.conf -DLocalStateDir=/opt/node1/var
257 ### <a id="cluster-scenarios"></a> Cluster Scenarios
259 #### <a id="cluster-scenarios-features"></a> Features in Cluster
261 Each cluster instance may use available features. If you have multiple locations
262 or departments, they may write to their local database, or populate graphite.
263 Even further all commands are distributed (unless prohibited using [Domains](#domains)).
265 DB IDO on the left, graphite on the right side - works.
266 Icinga Web 2 on the left, checker and notifications on the right side - works too.
267 Everything on the left and on the right side - make sure to deal with duplicated notifications
268 and automated check distribution.
270 #### <a id="cluster-scenarios-location-based"></a> Location Based Cluster
272 That scenario fits if your instances are spread over the globe and they all report
273 to a central instance. Their network connection only works towards the central master
274 (or the master is able to connect, depending on firewall policies) which means
275 remote instances won't see each/connect to each other.
277 All events are synced to the central node, but the remote nodes can still run
278 local features such as a web interface, reporting, graphing, etc.
280 Imagine the following example with a central node in Nuremberg, and two remote DMZ
281 based instances in Berlin and Vienna. The configuration tree on the central instance
282 could look like this:
295 The configuration deployment should look like:
297 * The node `nuremberg` sends `conf.d/germany/berlin` to the `berlin` node.
298 * The node `nuremberg` sends `conf.d/austria/vienna` to the `vienna` node.
300 `conf.d/templates` is shared on all nodes.
302 The endpoint configuration on the `nuremberg` node would look like:
304 object Endpoint "nuremberg" {
305 host = "nuremberg.icinga.org"
309 object Endpoint "berlin" {
310 host = "berlin.icinga.org"
312 config_files_recursive = [ "/etc/icinga2/conf.d/templates",
313 "/etc/icinga2/conf.d/germany/berlin" ]
316 object Endpoint "vienna" {
317 host = "vienna.icinga.org"
319 config_files_recursive = [ "/etc/icinga2/conf.d/templates",
320 "/etc/icinga2/conf.d/austria/vienna" ]
323 Each remote node will only peer with the central `nuremberg` node. Therefore
324 only two endpoints are required for cluster connection. Furthermore the remote
325 node must include the received configuration by the cluster functionality.
327 Example for the configuration on the `berlin` node:
329 object Endpoint "nuremberg" {
330 host = "nuremberg.icinga.org"
334 object Endpoint "berlin" {
335 host = "berlin.icinga.org"
337 accept_config = [ "nuremberg" ]
340 include_recursive LocalStateDir + "/lib/icinga2/cluster/config"
342 Depenending on the network connectivity the connections can be either
343 established by the remote node or the central node.
345 Example for `berlin` node connecting to central `nuremberg` node:
349 object ClusterListener "berlin-cluster" {
350 ca_path = "/etc/icinga2/ca/ca.crt"
351 cert_path = "/etc/icinga2/ca/berlin.crt"
352 key_path = "/etc/icinga2/ca/berlin.key"
354 peers = [ "nuremberg" ]
357 Example for central `nuremberg` node connecting to remote nodes:
361 object ClusterListener "nuremberg-cluster" {
362 ca_path = "/etc/icinga2/ca/ca.crt"
363 cert_path = "/etc/icinga2/ca/nuremberg.crt"
364 key_path = "/etc/icinga2/ca/nuremberg.key"
366 peers = [ "berlin", "vienna" ]
369 The central node should not do any checks by itself. There's two possibilities to achieve
372 * Disable the `checker` feature
373 * Pin the service object configuration to the remote endpoints using the [authorities](#assign-services-to-cluster-nodes)
377 #### <a id="cluster-scenarios-load-distribution"></a> Load Distribution
379 If you are planning to off-load the checks to a defined set of remote workers
380 you can achieve that by:
382 * Deploying the configuration on all nodes.
383 * Let Icinga 2 distribute the load amongst all available nodes.
385 That way all remote check instances will receive the same configuration
386 but only execute their part. The central instance can also execute checks,
387 but you may also disable the `Checker` feature.
393 If you are planning to have some checks executed by a specific set of checker nodes
394 just pin them using the [authorities](#assign-services-to-cluster-nodes) attribute.
396 Example on the `central` node:
398 object Endpoint "central" {
399 host = "central.icinga.org"
403 object Endpoint "checker1" {
404 host = "checker1.icinga.org"
406 config_files_recursive = [ "/etc/icinga2/conf.d" ]
409 object Endpoint "checker2" {
410 host = "checker2.icinga.org"
412 config_files_recursive = [ "/etc/icinga2/conf.d" ]
415 object ClusterListener "central-cluster" {
416 ca_path = "/etc/icinga2/ca/ca.crt"
417 cert_path = "/etc/icinga2/ca/central.crt"
418 key_path = "/etc/icinga2/ca/central.key"
420 peers = [ "checker1", "checker2" ]
423 Example on `checker1` node:
425 object Endpoint "central" {
426 host = "central.icinga.org"
430 object Endpoint "checker1" {
431 host = "checker1.icinga.org"
433 accept_config = [ "central" ]
436 object Endpoint "checker2" {
437 host = "checker2.icinga.org"
439 accept_config = [ "central" ]
442 object ClusterListener "checker1-cluster" {
443 ca_path = "/etc/icinga2/ca/ca.crt"
444 cert_path = "/etc/icinga2/ca/checker1.crt"
445 key_path = "/etc/icinga2/ca/checker1.key"
450 #### <a id="cluster-scenarios-high-availability"></a> High Availability
452 Two nodes in a high availability setup require an [initial cluster sync](#initial-cluster-sync).
453 Furthermore the active master node should deploy the configuration to the
454 second node, if that does not already happen by your provisioning tool. It primarly
455 depends which features are enabled/used. It is still required that some failover
456 mechanism detects for example which instance will be the notification "master".
459 #### <a id="cluster-scenarios-multiple-hierachies"></a> Multiple Hierachies
461 Your central instance collects all check results for reporting and graphing and also
462 does some sort of additional notifications.
463 The customers got their own instances in their local DMZs. They are limited to read/write
464 only their services, but replicate all events back to the central instance.
465 Within each DMZ there are additional check instances also serving interfaces for local
466 departments. The customers instances will collect all results, but also send them back to
467 your central instance.
468 Additionally the customers instance on the second level in the middle prohibits you from
469 sending commands to the down below department nodes. You're only allowed to receive the
470 results, and a subset of each customers configuration too.
472 Your central instance will generate global reports, aggregate alert notifications and check
473 additional dependencies (for example, the customers internet uplink and bandwidth usage).
475 The customers instance will only check a subset of local services and delegate the rest
476 to each department. Even though it acts as configuration master with a central dashboard
477 for all departments managing their configuration tree which is then deployed to all
478 department instances. Furthermore the central NOC is able to see what's going on.
480 The instances in the departments will serve a local interface, and allow the administrators
481 to reschedule checks or acknowledge problems for their services.