1 # Monitoring Basics <a id="monitoring-basics"></a>
3 This part of the Icinga 2 documentation provides an overview of all the basic
4 monitoring concepts you need to know to run Icinga 2.
5 Keep in mind these examples are made with a Linux server. If you are
6 using Windows, you will need to change the services accordingly. See the [ITL reference](10-icinga-template-library.md#windows-plugins)
7 for further information.
9 ## Hosts and Services <a id="hosts-services"></a>
11 Icinga 2 can be used to monitor the availability of hosts and services. Hosts
12 and services can be virtually anything which can be checked in some way:
14 * Network services (HTTP, SMTP, SNMP, SSH, etc.)
18 * Other local or network-accessible services
20 Host objects provide a mechanism to group services that are running
21 on the same physical device.
23 Here is an example of a host object which defines two child services:
25 object Host "my-server1" {
27 check_command = "hostalive"
30 object Service "ping4" {
31 host_name = "my-server1"
32 check_command = "ping4"
35 object Service "http" {
36 host_name = "my-server1"
37 check_command = "http"
40 The example creates two services `ping4` and `http` which belong to the
43 It also specifies that the host should perform its own check using the `hostalive`
46 The `address` attribute is used by check commands to determine which network
47 address is associated with the host object.
49 Details on troubleshooting check problems can be found [here](15-troubleshooting.md#troubleshooting).
51 ### Host States <a id="host-states"></a>
53 Hosts can be in any of the following states:
56 ------------|--------------
57 UP | The host is available.
58 DOWN | The host is unavailable.
60 ### Service States <a id="service-states"></a>
62 Services can be in any of the following states:
65 ------------|--------------
66 OK | The service is working properly.
67 WARNING | The service is experiencing some problems but is still considered to be in working condition.
68 CRITICAL | The service is in a critical state.
69 UNKNOWN | The check could not determine the service's state.
71 ### Hard and Soft States <a id="hard-soft-states"></a>
73 When detecting a problem with a host/service Icinga re-checks the object a number of
74 times (based on the `max_check_attempts` and `retry_interval` settings) before sending
75 notifications. This ensures that no unnecessary notifications are sent for
76 transient failures. During this time the object is in a `SOFT` state.
78 After all re-checks have been executed and the object is still in a non-OK
79 state the host/service switches to a `HARD` state and notifications are sent.
82 ------------|--------------
83 HARD | The host/service's state hasn't recently changed.
84 SOFT | The host/service has recently changed state and is being re-checked.
86 ### Host and Service Checks <a id="host-service-checks"></a>
88 Hosts and services determine their state by running checks in a regular interval.
90 object Host "router" {
91 check_command = "hostalive"
95 The `hostalive` command is one of several built-in check commands. It sends ICMP
96 echo requests to the IP address specified in the `address` attribute to determine
97 whether a host is online.
99 A number of other [built-in check commands](10-icinga-template-library.md#icinga-template-library) are also
100 available. In addition to these commands the next few chapters will explain in
101 detail how to set up your own check commands.
104 ## Templates <a id="object-inheritance-using-templates"></a>
106 Templates may be used to apply a set of identical attributes to more than one
109 template Service "generic-service" {
110 max_check_attempts = 3
113 enable_perfdata = true
116 apply Service "ping4" {
117 import "generic-service"
119 check_command = "ping4"
121 assign where host.address
124 apply Service "ping6" {
125 import "generic-service"
127 check_command = "ping6"
129 assign where host.address6
133 In this example the `ping4` and `ping6` services inherit properties from the
134 template `generic-service`.
136 Objects as well as templates themselves can import an arbitrary number of
137 other templates. Attributes inherited from a template can be overridden in the
140 You can also import existing non-template objects. Note that templates
141 and objects share the same namespace, i.e. you can't define a template
142 that has the same name like an object.
145 ## Custom Attributes <a id="custom-attributes"></a>
147 In addition to built-in attributes you can define your own attributes:
149 object Host "localhost" {
153 Valid values for custom attributes include:
155 * [Strings](17-language-reference.md#string-literals), [numbers](17-language-reference.md#numeric-literals) and [booleans](17-language-reference.md#boolean-literals)
156 * [Arrays](17-language-reference.md#array) and [dictionaries](17-language-reference.md#dictionary)
157 * [Functions](03-monitoring-basics.md#custom-attributes-functions)
159 ### Functions as Custom Attributes <a id="custom-attributes-functions"></a>
161 Icinga 2 lets you specify [functions](17-language-reference.md#functions) for custom attributes.
162 The special case here is that whenever Icinga 2 needs the value for such a custom attribute it runs
163 the function and uses whatever value the function returns:
165 object CheckCommand "random-value" {
166 command = [ PluginDir + "/check_dummy", "0", "$text$" ]
168 vars.text = {{ Math.random() * 100 }}
171 This example uses the [abbreviated lambda syntax](17-language-reference.md#nullary-lambdas).
173 These functions have access to a number of variables:
175 Variable | Description
176 -------------|---------------
177 user | The User object (for notifications).
178 service | The Service object (for service checks/notifications/event handlers).
179 host | The Host object.
180 command | The command object (e.g. a CheckCommand object for checks).
184 vars.text = {{ host.check_interval }}
186 In addition to these variables the `macro` function can be used to retrieve the
187 value of arbitrary macro expressions:
190 if (macro("$address$") == "127.0.0.1") {
191 log("Running a check for localhost!")
197 The `resolve_arguments` can be used to resolve a command and its arguments much in
198 the same fashion Icinga does this for the `command` and `arguments` attributes for
199 commands. The `by_ssh` command uses this functionality to let users specify a
200 command and arguments that should be executed via SSH:
204 var command = macro("$by_ssh_command$")
205 var arguments = macro("$by_ssh_arguments$")
207 if (typeof(command) == String && !arguments) {
211 var escaped_args = []
212 for (arg in resolve_arguments(command, arguments)) {
213 escaped_args.add(escape_shell_arg(arg))
215 return escaped_args.join(" ")
220 Acessing object attributes at runtime inside these functions is described in the
221 [advanced topics](08-advanced-topics.md#access-object-attributes-at-runtime) chapter.
223 ## Runtime Macros <a id="runtime-macros"></a>
225 Macros can be used to access other objects' attributes at runtime. For example they
226 are used in command definitions to figure out which IP address a check should be
229 object CheckCommand "my-ping" {
230 command = [ PluginDir + "/check_ping", "-H", "$ping_address$" ]
233 "-w" = "$ping_wrta$,$ping_wpl$%"
234 "-c" = "$ping_crta$,$ping_cpl$%"
235 "-p" = "$ping_packets$"
238 vars.ping_address = "$address$"
246 vars.ping_packets = 5
249 object Host "router" {
250 check_command = "my-ping"
254 In this example we are using the `$address$` macro to refer to the host's `address`
257 We can also directly refer to custom attributes, e.g. by using `$ping_wrta$`. Icinga
258 automatically tries to find the closest match for the attribute you specified. The
259 exact rules for this are explained in the next section.
263 > When using the `$` sign as single character you must escape it with an
264 > additional dollar character (`$$`).
267 ### Evaluation Order <a id="macro-evaluation-order"></a>
269 When executing commands Icinga 2 checks the following objects in this order to look
270 up macros and their respective values:
272 1. User object (only for notifications)
276 5. Global custom attributes in the `Vars` constant
278 This execution order allows you to define default values for custom attributes
279 in your command objects.
281 Here's how you can override the custom attribute `ping_packets` from the previous
284 object Service "ping" {
285 host_name = "localhost"
286 check_command = "my-ping"
288 vars.ping_packets = 10 // Overrides the default value of 5 given in the command
291 If a custom attribute isn't defined anywhere, an empty value is used and a warning is
292 written to the Icinga 2 log.
294 You can also directly refer to a specific attribute -- thereby ignoring these evaluation
295 rules -- by specifying the full attribute name:
297 $service.vars.ping_wrta$
299 This retrieves the value of the `ping_wrta` custom attribute for the service. This
300 returns an empty value if the service does not have such a custom attribute no matter
301 whether another object such as the host has this attribute.
304 ### Host Runtime Macros <a id="host-runtime-macros"></a>
306 The following host custom attributes are available in all commands that are executed for
310 -----------------------------|--------------
311 host.name | The name of the host object.
312 host.display_name | The value of the `display_name` attribute.
313 host.state | The host's current state. Can be one of `UNREACHABLE`, `UP` and `DOWN`.
314 host.state_id | The host's current state. Can be one of `0` (up), `1` (down) and `2` (unreachable).
315 host.state_type | The host's current state type. Can be one of `SOFT` and `HARD`.
316 host.check_attempt | The current check attempt number.
317 host.max_check_attempts | The maximum number of checks which are executed before changing to a hard state.
318 host.last_state | The host's previous state. Can be one of `UNREACHABLE`, `UP` and `DOWN`.
319 host.last_state_id | The host's previous state. Can be one of `0` (up), `1` (down) and `2` (unreachable).
320 host.last_state_type | The host's previous state type. Can be one of `SOFT` and `HARD`.
321 host.last_state_change | The last state change's timestamp.
322 host.downtime_depth | The number of active downtimes.
323 host.duration_sec | The time since the last state change.
324 host.latency | The host's check latency.
325 host.execution_time | The host's check execution time.
326 host.output | The last check's output.
327 host.perfdata | The last check's performance data.
328 host.last_check | The timestamp when the last check was executed.
329 host.check_source | The monitoring instance that performed the last check.
330 host.num_services | Number of services associated with the host.
331 host.num_services_ok | Number of services associated with the host which are in an `OK` state.
332 host.num_services_warning | Number of services associated with the host which are in a `WARNING` state.
333 host.num_services_unknown | Number of services associated with the host which are in an `UNKNOWN` state.
334 host.num_services_critical | Number of services associated with the host which are in a `CRITICAL` state.
336 ### Service Runtime Macros <a id="service-runtime-macros"></a>
338 The following service macros are available in all commands that are executed for
342 ---------------------------|--------------
343 service.name | The short name of the service object.
344 service.display_name | The value of the `display_name` attribute.
345 service.check_command | The short name of the command along with any arguments to be used for the check.
346 service.state | The service's current state. Can be one of `OK`, `WARNING`, `CRITICAL` and `UNKNOWN`.
347 service.state_id | The service's current state. Can be one of `0` (ok), `1` (warning), `2` (critical) and `3` (unknown).
348 service.state_type | The service's current state type. Can be one of `SOFT` and `HARD`.
349 service.check_attempt | The current check attempt number.
350 service.max_check_attempts | The maximum number of checks which are executed before changing to a hard state.
351 service.last_state | The service's previous state. Can be one of `OK`, `WARNING`, `CRITICAL` and `UNKNOWN`.
352 service.last_state_id | The service's previous state. Can be one of `0` (ok), `1` (warning), `2` (critical) and `3` (unknown).
353 service.last_state_type | The service's previous state type. Can be one of `SOFT` and `HARD`.
354 service.last_state_change | The last state change's timestamp.
355 service.downtime_depth | The number of active downtimes.
356 service.duration_sec | The time since the last state change.
357 service.latency | The service's check latency.
358 service.execution_time | The service's check execution time.
359 service.output | The last check's output.
360 service.perfdata | The last check's performance data.
361 service.last_check | The timestamp when the last check was executed.
362 service.check_source | The monitoring instance that performed the last check.
364 ### Command Runtime Macros <a id="command-runtime-macros"></a>
366 The following custom attributes are available in all commands:
369 -----------------------|--------------
370 command.name | The name of the command object.
372 ### User Runtime Macros <a id="user-runtime-macros"></a>
374 The following custom attributes are available in all commands that are executed for
378 -----------------------|--------------
379 user.name | The name of the user object.
380 user.display_name | The value of the display_name attribute.
382 ### Notification Runtime Macros <a id="notification-runtime-macros"></a>
385 -----------------------|--------------
386 notification.type | The type of the notification.
387 notification.author | The author of the notification comment if existing.
388 notification.comment | The comment of the notification if existing.
390 ### Global Runtime Macros <a id="global-runtime-macros"></a>
392 The following macros are available in all executed commands:
395 -----------------------|--------------
396 icinga.timet | Current UNIX timestamp.
397 icinga.long_date_time | Current date and time including timezone information. Example: `2014-01-03 11:23:08 +0000`
398 icinga.short_date_time | Current date and time. Example: `2014-01-03 11:23:08`
399 icinga.date | Current date. Example: `2014-01-03`
400 icinga.time | Current time including timezone information. Example: `11:23:08 +0000`
401 icinga.uptime | Current uptime of the Icinga 2 process.
403 The following macros provide global statistics:
406 ----------------------------------|--------------
407 icinga.num_services_ok | Current number of services in state 'OK'.
408 icinga.num_services_warning | Current number of services in state 'Warning'.
409 icinga.num_services_critical | Current number of services in state 'Critical'.
410 icinga.num_services_unknown | Current number of services in state 'Unknown'.
411 icinga.num_services_pending | Current number of pending services.
412 icinga.num_services_unreachable | Current number of unreachable services.
413 icinga.num_services_flapping | Current number of flapping services.
414 icinga.num_services_in_downtime | Current number of services in downtime.
415 icinga.num_services_acknowledged | Current number of acknowledged service problems.
416 icinga.num_hosts_up | Current number of hosts in state 'Up'.
417 icinga.num_hosts_down | Current number of hosts in state 'Down'.
418 icinga.num_hosts_unreachable | Current number of unreachable hosts.
419 icinga.num_hosts_pending | Current number of pending hosts.
420 icinga.num_hosts_flapping | Current number of flapping hosts.
421 icinga.num_hosts_in_downtime | Current number of hosts in downtime.
422 icinga.num_hosts_acknowledged | Current number of acknowledged host problems.
425 ## Apply Rules <a id="using-apply"></a>
427 Several object types require an object relation, e.g. [Service](09-object-types.md#objecttype-service),
428 [Notification](09-object-types.md#objecttype-notification), [Dependency](09-object-types.md#objecttype-dependency),
429 [ScheduledDowntime](09-object-types.md#objecttype-scheduleddowntime) objects.
430 If you for example create a service object you have to specify the [host_name](09-object-types.md#objecttype-service)
431 attribute and reference an existing host attribute.
433 object Service "ping4" {
434 check_command = "ping4"
435 host_name = "icinga2-client1.localdomain"
438 This isn't comfortable when managing a huge set of configuration objects which could
439 [match](03-monitoring-basics.md#using-apply-expressions) on a common pattern.
441 Instead you want to use **[apply](17-language-reference.md#apply) rules**.
443 If you want basic monitoring for all your hosts, add a `ping4` service apply rule
444 for all hosts which have the `address` attribute specified. Just one rule for 1000 hosts
445 instead of 1000 service objects. Apply rules will automatically generate them for you.
447 apply Service "ping4" {
448 check_command = "ping4"
449 assign where host.address
452 More explanations on assign where expressions can be found [here](03-monitoring-basics.md#using-apply-expressions).
454 Before you start with apply rules keep the following in mind:
456 * Define the best match.
457 * A set of unique [custom attributes](03-monitoring-basics.md#custom-attributes) for these hosts/services?
458 * Or [group](03-monitoring-basics.md#groups) memberships, e.g. a host being a member of a hostgroup which should have a service set?
459 * A generic pattern [match](18-library-reference.md#global-functions-match) on the host/service name?
460 * [Multiple expressions combined](03-monitoring-basics.md#using-apply-expressions) with `&&` or `||` [operators](17-language-reference.md#expression-operators)
461 * All expressions must return a boolean value (an empty string is equal to `false` e.g.)
463 More specific object type requirements are described in these chapters:
465 * [Apply services to hosts](03-monitoring-basics.md#using-apply-services)
466 * [Apply notifications to hosts and services](03-monitoring-basics.md#using-apply-notifications)
467 * [Apply dependencies to hosts and services](03-monitoring-basics.md#using-apply-dependencies)
468 * [Apply scheduled downtimes to hosts and services](03-monitoring-basics.md#using-apply-scheduledowntimes)
470 You can set/override object attributes in apply rules using the respectively available
471 objects in that scope (host and/or service objects).
473 vars.application_type = host.vars.application_type
475 [Custom attributes](03-monitoring-basics.md#custom-attributes) can also store nested dictionaries and arrays. That way you can use them
476 for not only matching for their existence or values in apply expressions, but also assign
477 ("inherit") their values into the generated objected from apply rules.
479 A more advanced example is to use [apply rules with for loops on arrays or
480 dictionaries](03-monitoring-basics.md#using-apply-for) provided by
481 [custom atttributes](03-monitoring-basics.md#custom-attributes) or groups.
485 > Building configuration in that dynamic way requires detailed information
486 > of the generated objects. Use the `object list` [CLI command](11-cli-commands.md#cli-command-object)
487 > after successful [configuration validation](11-cli-commands.md#config-validation).
490 ### Apply Rules Expressions <a id="using-apply-expressions"></a>
492 You can use simple or advanced combinations of apply rule expressions. Each
493 expression must evaluate into the boolean `true` value. An empty string
494 will be for instance interpreted as `false`. In a similar fashion undefined
495 attributes will return `false`.
499 assign where host.vars.attribute_does_not_exist
501 Multiple `assign where` condition rows are evaluated as `OR` condition.
503 You can combine multiple expressions for matching only a subset of objects. In some cases,
504 you want to be able to add more than one assign/ignore where expression which matches
505 a specific condition. To achieve this you can use the logical `and` and `or` operators.
507 #### Apply Rules Expressions Examples <a id="using-apply-expressions-examples"></a>
509 Assign a service to a specific host in a host group [array](18-library-reference.md#array-type) using the [in operator](17-language-reference.md#expression-operators):
511 assign where "hostgroup-dev" in host.groups
513 Assign an object when a custom attribute is [equal](17-language-reference.md#expression-operators) to a value:
515 assign where host.vars.application_type == "database"
517 assign where service.vars.sms_notify == true
519 Assign an object if a dictionary [contains](18-library-reference.md#dictionary-contains) a given key:
521 assign where host.vars.app_dict.contains("app")
523 Match the host name by either using a [case insensitive match](18-library-reference.md#global-functions-match):
525 assign where match("webserver*", host.name)
527 Match the host name by using a [regular expression](18-library-reference.md#global-functions-regex). Please note the [escaped](17-language-reference.md#string-literals-escape-sequences) backslash character:
529 assign where regex("^webserver-[\\d+]", host.name)
532 [Match](18-library-reference.md#global-functions-match) all `*mysql*` patterns in the host name and (`&&`) custom attribute `prod_mysql_db`
533 matches the `db-*` pattern. All hosts with the custom attribute `test_server` set to `true`
534 should be ignored, or any host name ending with `*internal` pattern.
536 object HostGroup "mysql-server" {
537 display_name = "MySQL Server"
539 assign where match("*mysql*", host.name) && match("db-*", host.vars.prod_mysql_db)
540 ignore where host.vars.test_server == true
541 ignore where match("*internal", host.name)
544 Similar example for advanced notification apply rule filters: If the service
545 attribute `notes` [matches](18-library-reference.md#global-functions-match) the `has gold support 24x7` string `AND` one of the
546 two condition passes, either the `customer` host custom attribute is set to `customer-xy`
547 `OR` the host custom attribute `always_notify` is set to `true`.
549 The notification is ignored for services whose host name ends with `*internal`
550 `OR` the `priority` custom attribute is [less than](17-language-reference.md#expression-operators) `2`.
552 template Notification "cust-xy-notification" {
553 users = [ "noc-xy", "mgmt-xy" ]
554 command = "mail-service-notification"
557 apply Notification "notify-cust-xy-mysql" to Service {
558 import "cust-xy-notification"
560 assign where match("*has gold support 24x7*", service.notes) && (host.vars.customer == "customer-xy" || host.vars.always_notify == true)
561 ignore where match("*internal", host.name) || (service.vars.priority < 2 && host.vars.is_clustered == true)
564 More advanced examples are covered [here](08-advanced-topics.md#use-functions-assign-where).
566 ### Apply Services to Hosts <a id="using-apply-services"></a>
568 The sample configuration already includes a detailed example in [hosts.conf](04-configuring-icinga-2.md#hosts-conf)
569 and [services.conf](04-configuring-icinga-2.md#services-conf) for this use case.
571 The example for `ssh` applies a service object to all hosts with the `address`
572 attribute being defined and the custom attribute `os` set to the string `Linux` in `vars`.
574 apply Service "ssh" {
575 import "generic-service"
577 check_command = "ssh"
579 assign where host.address && host.vars.os == "Linux"
582 Other detailed examples are used in their respective chapters, for example
583 [apply services with custom command arguments](03-monitoring-basics.md#command-passing-parameters).
585 ### Apply Notifications to Hosts and Services <a id="using-apply-notifications"></a>
587 Notifications are applied to specific targets (`Host` or `Service`) and work in a similar
591 apply Notification "mail-noc" to Service {
592 import "mail-service-notification"
594 user_groups = [ "noc" ]
596 assign where host.vars.notification.mail
600 In this example the `mail-noc` notification will be created as object for all services having the
601 `notification.mail` custom attribute defined. The notification command is set to `mail-service-notification`
602 and all members of the user group `noc` will get notified.
604 It is also possible to generally apply a notification template and dynamically overwrite values from
605 the template by checking for custom attributes. This can be achieved by using [conditional statements](17-language-reference.md#conditional-statements):
607 apply Notification "host-mail-noc" to Host {
608 import "mail-host-notification"
610 // replace interval inherited from `mail-host-notification` template with new notfication interval set by a host custom attribute
611 if (host.vars.notification_interval) {
612 interval = host.vars.notification_interval
615 // same with notification period
616 if (host.vars.notification_period) {
617 period = host.vars.notification_period
620 // Send SMS instead of email if the host's custom attribute `notification_type` is set to `sms`
621 if (host.vars.notification_type == "sms") {
622 command = "sms-host-notification"
624 command = "mail-host-notification"
627 user_groups = [ "noc" ]
629 assign where host.address
632 In the example above, the notification template `mail-host-notification`, which contains all relevant
633 notification settings, is applied on all host objects where the `host.address` is defined.
634 Each host object is then checked for custom attributes (`host.vars.notification_interval`,
635 `host.vars.notification_period` and `host.vars.notification_type`). Depending if the custom
636 attibute is set or which value it has, the value from the notification template is dynamically
639 The corresponding host object could look like this:
641 object Host "host1" {
642 import "host-linux-prod"
643 display_name = "host1"
644 address = "192.168.1.50"
645 vars.notification_interval = 1h
646 vars.notification_period = "24x7"
647 vars.notification_type = "sms"
650 ### Apply Dependencies to Hosts and Services <a id="using-apply-dependencies"></a>
652 Detailed examples can be found in the [dependencies](03-monitoring-basics.md#dependencies) chapter.
654 ### Apply Recurring Downtimes to Hosts and Services <a id="using-apply-scheduledowntimes"></a>
656 The sample configuration includes an example in [downtimes.conf](04-configuring-icinga-2.md#downtimes-conf).
658 Detailed examples can be found in the [recurring downtimes](08-advanced-topics.md#recurring-downtimes) chapter.
661 ### Using Apply For Rules <a id="using-apply-for"></a>
663 Next to the standard way of using [apply rules](03-monitoring-basics.md#using-apply)
664 there is the requirement of applying objects based on a set (array or
665 dictionary) using [apply for](17-language-reference.md#apply-for) expressions.
667 The sample configuration already includes a detailed example in [hosts.conf](04-configuring-icinga-2.md#hosts-conf)
668 and [services.conf](04-configuring-icinga-2.md#services-conf) for this use case.
670 Take the following example: A host provides the snmp oids for different service check
671 types. This could look like the following example:
673 object Host "router-v6" {
674 check_command = "hostalive"
677 vars.oids["if01"] = "1.1.1.1.1"
678 vars.oids["temp"] = "1.1.1.1.2"
679 vars.oids["bgp"] = "1.1.1.1.5"
682 Now we want to create service checks for `if01` and `temp`, but not `bgp`.
683 Furthermore we want to pass the snmp oid stored as dictionary value to the
684 custom attribute called `vars.snmp_oid` -- this is the command argument required
685 by the [snmp](10-icinga-template-library.md#plugin-check-command-snmp) check command.
686 The service's `display_name` should be set to the identifier inside the dictionary.
688 apply Service for (identifier => oid in host.vars.oids) {
689 check_command = "snmp"
690 display_name = identifier
693 ignore where identifier == "bgp" //don't generate service for bgp checks
696 Icinga 2 evaluates the `apply for` rule for all objects with the custom attribute
697 `oids` set. It then iterates over all list items inside the `for` loop and evaluates the
698 `assign/ignore where` expressions. You can access the loop variable
699 in these expressions, e.g. for ignoring certain values.
700 In this example we'd ignore the `bgp` identifier and avoid generating an unwanted service.
701 We could extend the configuration by also matching the `oid` value on certain
702 [regex](18-library-reference.md#global-functions-regex)/[wildcard match](18-library-reference.md#global-functions-match) patterns for example.
706 > You don't need an `assign where` expression only checking for existance
707 > of the custom attribute.
709 That way you'll save duplicated apply rules by combining them into one
710 generic `apply for` rule generating the object name with or without a prefix.
713 #### Apply For and Custom Attribute Override <a id="using-apply-for-custom-attribute-override"></a>
715 Imagine a different more advanced example: You are monitoring your network device (host)
716 with many interfaces (services). The following requirements/problems apply:
718 * Each interface service check should be named with a prefix and a name defined in your host object (which could be generated from your CMDB, etc.)
719 * Each interface has its own vlan tag
720 * Some interfaces have QoS enabled
721 * Additional attributes such as `display_name` or `notes`, `notes_url` and `action_url` must be
722 dynamically generated
725 Tip: Define the snmp community as global constant in your [constants.conf](04-configuring-icinga-2.md#constants-conf) file.
727 const IftrafficSnmpCommunity = "public"
729 By defining the `interfaces` dictionary with three example interfaces on the `cisco-catalyst-6509-34`
730 host object, you'll make sure to pass the [custom attribute](03-monitoring-basics.md#custom-attributes)
731 storage required by the for loop in the service apply rule.
733 object Host "cisco-catalyst-6509-34" {
734 import "generic-host"
735 display_name = "Catalyst 6509 #34 VIE21"
736 address = "127.0.1.4"
738 /* "GigabitEthernet0/2" is the interface name,
739 * and key name in service apply for later on
741 vars.interfaces["GigabitEthernet0/2"] = {
742 /* define all custom attributes with the
743 * same name required for command parameters/arguments
744 * in service apply (look into your CheckCommand definition)
746 iftraffic_units = "g"
747 iftraffic_community = IftrafficSnmpCommunity
748 iftraffic_bandwidth = 1
752 vars.interfaces["GigabitEthernet0/4"] = {
753 iftraffic_units = "g"
754 //iftraffic_community = IftrafficSnmpCommunity
755 iftraffic_bandwidth = 1
759 vars.interfaces["MgmtInterface1"] = {
760 iftraffic_community = IftrafficSnmpCommunity
762 interface_address = "127.99.0.100" #special management ip
766 You can also omit the `"if-"` string, then all generated service names are directly
767 taken from the `if_name` variable value.
769 The config dictionary contains all key-value pairs for the specific interface in one
770 loop cycle, like `iftraffic_units`, `vlan`, and `qos` for the specified interface.
772 You can either map the custom attributes from the `interface_config` dictionary to
773 local custom attributes stashed into `vars`. If the names match the required command
774 argument parameters already (for example `iftraffic_units`), you could also add the
775 `interface_config` dictionary to the `vars` dictionary using the `+=` operator.
777 After `vars` is fully populated, all object attributes can be set calculated from
778 provided host attributes. For strings, you can use string concatention with the `+` operator.
780 You can also specify the display_name, check command, interval, notes, notes_url, action_url, etc.
781 attributes that way. Attribute strings can be [concatenated](17-language-reference.md#expression-operators),
782 for example for adding a more detailed service `display_name`.
784 This example also uses [if conditions](17-language-reference.md#conditional-statements)
785 if specific values are not set, adding a local default value.
786 The other way around you can override specific custom attributes inherited from a service template if set.
788 /* loop over the host.vars.interfaces dictionary
789 * for (key => value in dict) means `interface_name` as key
790 * and `interface_config` as value. Access config attributes
791 * with the indexer (`.`) character.
793 apply Service "if-" for (interface_name => interface_config in host.vars.interfaces) {
794 import "generic-service"
795 check_command = "iftraffic"
796 display_name = "IF-" + interface_name
798 /* use the key as command argument (no duplication of values in host.vars.interfaces) */
799 vars.iftraffic_interface = interface_name
801 /* map the custom attributes as command arguments */
802 vars.iftraffic_units = interface_config.iftraffic_units
803 vars.iftraffic_community = interface_config.iftraffic_community
805 /* the above can be achieved in a shorter fashion if the names inside host.vars.interfaces
806 * are the _exact_ same as required as command parameter by the check command
809 vars += interface_config
811 /* set a default value for units and bandwidth */
812 if (interface_config.iftraffic_units == "") {
813 vars.iftraffic_units = "m"
815 if (interface_config.iftraffic_bandwidth == "") {
816 vars.iftraffic_bandwidth = 1
818 if (interface_config.vlan == "") {
819 vars.vlan = "not set"
821 if (interface_config.qos == "") {
825 /* set the global constant if not explicitely
826 * not provided by the `interfaces` dictionary on the host
828 if (len(interface_config.iftraffic_community) == 0 || len(vars.iftraffic_community) == 0) {
829 vars.iftraffic_community = IftrafficSnmpCommunity
832 /* Calculate some additional object attributes after populating the `vars` dictionary */
833 notes = "Interface check for " + interface_name + " (units: '" + interface_config.iftraffic_units + "') in VLAN '" + vars.vlan + "' with ' QoS '" + vars.qos + "'"
834 notes_url = "https://foreman.company.com/hosts/" + host.name
835 action_url = "http://snmp.checker.company.com/" + host.name + "/if-" + interface_name
840 This example makes use of the [check_iftraffic](https://exchange.icinga.com/exchange/iftraffic) plugin.
841 The `CheckCommand` definition can be found in the
842 [contributed plugin check commands](10-icinga-template-library.md#plugin-contrib-command-iftraffic)
843 -- make sure to include them in your [icinga2 configuration file](04-configuring-icinga-2.md#icinga2-conf).
848 > Building configuration in that dynamic way requires detailed information
849 > of the generated objects. Use the `object list` [CLI command](11-cli-commands.md#cli-command-object)
850 > after successful [configuration validation](11-cli-commands.md#config-validation).
852 Verify that the apply-for-rule successfully created the service objects with the
853 inherited custom attributes:
856 # icinga2 object list --type Service --name *catalyst*
858 Object 'cisco-catalyst-6509-34!if-GigabitEthernet0/2' of type 'Service':
861 % = modified in '/etc/icinga2/conf.d/iftraffic.conf', lines 59:3-59:26
862 * iftraffic_bandwidth = 1
863 * iftraffic_community = "public"
864 % = modified in '/etc/icinga2/conf.d/iftraffic.conf', lines 53:3-53:65
865 * iftraffic_interface = "GigabitEthernet0/2"
866 % = modified in '/etc/icinga2/conf.d/iftraffic.conf', lines 49:3-49:43
867 * iftraffic_units = "g"
868 % = modified in '/etc/icinga2/conf.d/iftraffic.conf', lines 52:3-52:57
873 Object 'cisco-catalyst-6509-34!if-GigabitEthernet0/4' of type 'Service':
876 % = modified in '/etc/icinga2/conf.d/iftraffic.conf', lines 59:3-59:26
877 * iftraffic_bandwidth = 1
878 * iftraffic_community = "public"
879 % = modified in '/etc/icinga2/conf.d/iftraffic.conf', lines 53:3-53:65
880 % = modified in '/etc/icinga2/conf.d/iftraffic.conf', lines 79:5-79:53
881 * iftraffic_interface = "GigabitEthernet0/4"
882 % = modified in '/etc/icinga2/conf.d/iftraffic.conf', lines 49:3-49:43
883 * iftraffic_units = "g"
884 % = modified in '/etc/icinga2/conf.d/iftraffic.conf', lines 52:3-52:57
888 Object 'cisco-catalyst-6509-34!if-MgmtInterface1' of type 'Service':
891 % = modified in '/etc/icinga2/conf.d/iftraffic.conf', lines 59:3-59:26
892 * iftraffic_bandwidth = 1
893 % = modified in '/etc/icinga2/conf.d/iftraffic.conf', lines 66:5-66:32
894 * iftraffic_community = "public"
895 % = modified in '/etc/icinga2/conf.d/iftraffic.conf', lines 53:3-53:65
896 * iftraffic_interface = "MgmtInterface1"
897 % = modified in '/etc/icinga2/conf.d/iftraffic.conf', lines 49:3-49:43
898 * iftraffic_units = "m"
899 % = modified in '/etc/icinga2/conf.d/iftraffic.conf', lines 52:3-52:57
900 % = modified in '/etc/icinga2/conf.d/iftraffic.conf', lines 63:5-63:30
901 * interface_address = "127.99.0.100"
903 % = modified in '/etc/icinga2/conf.d/iftraffic.conf', lines 72:5-72:24
907 ### Use Object Attributes in Apply Rules <a id="using-apply-object-attributes"></a>
909 Since apply rules are evaluated after the generic objects, you
910 can reference existing host and/or service object attributes as
911 values for any object attribute specified in that apply rule.
913 object Host "opennebula-host" {
914 import "generic-host"
917 vars.hosting["xyz"] = {
919 customer_name = "Customer xyz"
921 support_contract = "gold"
923 vars.hosting["abc"] = {
925 customer_name = "Customer xyz"
927 support_contract = "silver"
931 apply Service for (customer => config in host.vars.hosting) {
932 import "generic-service"
933 check_command = "ping4"
935 vars.qos = "disabled"
939 vars.http_uri = "/" + vars.customer + "/" + config.http_uri
941 display_name = "Shop Check for " + vars.customer_name + "-" + vars.customer_id
943 notes = "Support contract: " + vars.support_contract + " for Customer " + vars.customer_name + " (" + vars.customer_id + ")."
945 notes_url = "https://foreman.company.com/hosts/" + host.name
946 action_url = "http://snmp.checker.company.com/" + host.name + "/" + vars.customer_id
949 ## Groups <a id="groups"></a>
951 A group is a collection of similar objects. Groups are primarily used as a
952 visualization aid in web interfaces.
954 Group membership is defined at the respective object itself. If
955 you have a hostgroup name `windows` for example, and want to assign
956 specific hosts to this group for later viewing the group on your
957 alert dashboard, first create a HostGroup object:
959 object HostGroup "windows" {
960 display_name = "Windows Servers"
963 Then add your hosts to this group:
965 template Host "windows-server" {
966 groups += [ "windows" ]
969 object Host "mssql-srv1" {
970 import "windows-server"
972 vars.mssql_port = 1433
975 object Host "mssql-srv2" {
976 import "windows-server"
978 vars.mssql_port = 1433
981 This can be done for service and user groups the same way:
983 object UserGroup "windows-mssql-admins" {
984 display_name = "Windows MSSQL Admins"
987 template User "generic-windows-mssql-users" {
988 groups += [ "windows-mssql-admins" ]
991 object User "win-mssql-noc" {
992 import "generic-windows-mssql-users"
994 email = "noc@example.com"
997 object User "win-mssql-ops" {
998 import "generic-windows-mssql-users"
1000 email = "ops@example.com"
1003 ### Group Membership Assign <a id="group-assign-intro"></a>
1005 Instead of manually assigning each object to a group you can also assign objects
1006 to a group based on their attributes:
1008 object HostGroup "prod-mssql" {
1009 display_name = "Production MSSQL Servers"
1011 assign where host.vars.mssql_port && host.vars.prod_mysql_db
1012 ignore where host.vars.test_server == true
1013 ignore where match("*internal", host.name)
1016 In this example all hosts with the `vars` attribute `mssql_port`
1017 will be added as members to the host group `mssql`. However, all
1018 hosts [matching](18-library-reference.md#global-functions-match) the string `\*internal`
1019 or with the `test_server` attribute set to `true` are **not** added to this group.
1021 Details on the `assign where` syntax can be found in the
1022 [Language Reference](17-language-reference.md#apply).
1024 ## Notifications <a id="alert-notifications"></a>
1026 Notifications for service and host problems are an integral part of your
1029 When a host or service is in a downtime, a problem has been acknowledged or
1030 the dependency logic determined that the host/service is unreachable, no
1031 notifications are sent. You can configure additional type and state filters
1032 refining the notifications being actually sent.
1034 There are many ways of sending notifications, e.g. by email, XMPP,
1035 IRC, Twitter, etc. On its own Icinga 2 does not know how to send notifications.
1036 Instead it relies on external mechanisms such as shell scripts to notify users.
1037 More notification methods are listed in the [addons and plugins](13-addons.md#notification-scripts-interfaces)
1040 A notification specification requires one or more users (and/or user groups)
1041 who will be notified in case of problems. These users must have all custom
1042 attributes defined which will be used in the `NotificationCommand` on execution.
1044 The user `icingaadmin` in the example below will get notified only on `WARNING` and
1045 `CRITICAL` states and `problem` and `recovery` notification types.
1047 object User "icingaadmin" {
1048 display_name = "Icinga 2 Admin"
1049 enable_notifications = true
1050 states = [ OK, Warning, Critical ]
1051 types = [ Problem, Recovery ]
1052 email = "icinga@localhost"
1055 If you don't set the `states` and `types` configuration attributes for the `User`
1056 object, notifications for all states and types will be sent.
1058 Details on troubleshooting notification problems can be found [here](15-troubleshooting.md#troubleshooting).
1060 **Note**: Make sure that the [notification](11-cli-commands.md#enable-features) feature is enabled
1061 in order to execute notification commands.
1063 You should choose which information you (and your notified users) are interested in
1064 case of emergency, and also which information does not provide any value to you and
1067 An example notification command is explained [here](03-monitoring-basics.md#notification-commands).
1069 You can add all shared attributes to a `Notification` template which is inherited
1070 to the defined notifications. That way you'll save duplicated attributes in each
1071 `Notification` object. Attributes can be overridden locally.
1073 template Notification "generic-notification" {
1076 command = "mail-service-notification"
1078 states = [ Warning, Critical, Unknown ]
1079 types = [ Problem, Acknowledgement, Recovery, Custom, FlappingStart,
1080 FlappingEnd, DowntimeStart, DowntimeEnd, DowntimeRemoved ]
1085 The time period `24x7` is included as example configuration with Icinga 2.
1087 Use the `apply` keyword to create `Notification` objects for your services:
1089 apply Notification "notify-cust-xy-mysql" to Service {
1090 import "generic-notification"
1092 users = [ "noc-xy", "mgmt-xy" ]
1094 assign where match("*has gold support 24x7*", service.notes) && (host.vars.customer == "customer-xy" || host.vars.always_notify == true
1095 ignore where match("*internal", host.name) || (service.vars.priority < 2 && host.vars.is_clustered == true)
1099 Instead of assigning users to notifications, you can also add the `user_groups`
1100 attribute with a list of user groups to the `Notification` object. Icinga 2 will
1101 send notifications to all group members.
1103 **Note**: Only users who have been notified of a problem before (`Warning`, `Critical`, `Unknown`
1104 states for services, `Down` for hosts) will receive `Recovery` notifications.
1106 ### Notification Escalations <a id="notification-escalations"></a>
1108 When a problem notification is sent and a problem still exists at the time of re-notification
1109 you may want to escalate the problem to the next support level. A different approach
1110 is to configure the default notification by email, and escalate the problem via SMS
1111 if not already solved.
1113 You can define notification start and end times as additional configuration
1114 attributes making the `Notification` object a so-called `notification escalation`.
1115 Using templates you can share the basic notification attributes such as users or the
1116 `interval` (and override them for the escalation then).
1118 Using the example from above, you can define additional users being escalated for SMS
1119 notifications between start and end time.
1121 object User "icinga-oncall-2nd-level" {
1122 display_name = "Icinga 2nd Level"
1124 vars.mobile = "+1 555 424642"
1127 object User "icinga-oncall-1st-level" {
1128 display_name = "Icinga 1st Level"
1130 vars.mobile = "+1 555 424642"
1133 Define an additional [NotificationCommand](03-monitoring-basics.md#notification-commands) for SMS notifications.
1137 > The example is not complete as there are many different SMS providers.
1138 > Please note that sending SMS notifications will require an SMS provider
1139 > or local hardware with an active SIM card.
1141 object NotificationCommand "sms-notification" {
1143 PluginDir + "/send_sms_notification",
1148 The two new notification escalations are added onto the local host
1149 and its service `ping4` using the `generic-notification` template.
1150 The user `icinga-oncall-2nd-level` will get notified by SMS (`sms-notification`
1151 command) after `30m` until `1h`.
1155 > The `interval` was set to 15m in the `generic-notification`
1156 > template example. Lower that value in your escalations by using a secondary
1157 > template or by overriding the attribute directly in the `notifications` array
1158 > position for `escalation-sms-2nd-level`.
1160 If the problem does not get resolved nor acknowledged preventing further notifications,
1161 the `escalation-sms-1st-level` user will be escalated `1h` after the initial problem was
1162 notified, but only for one hour (`2h` as `end` key for the `times` dictionary).
1164 apply Notification "mail" to Service {
1165 import "generic-notification"
1167 command = "mail-notification"
1168 users = [ "icingaadmin" ]
1170 assign where service.name == "ping4"
1173 apply Notification "escalation-sms-2nd-level" to Service {
1174 import "generic-notification"
1176 command = "sms-notification"
1177 users = [ "icinga-oncall-2nd-level" ]
1184 assign where service.name == "ping4"
1187 apply Notification "escalation-sms-1st-level" to Service {
1188 import "generic-notification"
1190 command = "sms-notification"
1191 users = [ "icinga-oncall-1st-level" ]
1198 assign where service.name == "ping4"
1201 ### Notification Delay <a id="notification-delay"></a>
1203 Sometimes the problem in question should not be announced when the notification is due
1204 (the object reaching the `HARD` state), but after a certain period. In Icinga 2
1205 you can use the `times` dictionary and set `begin = 15m` as key and value if you want to
1206 postpone the notification window for 15 minutes. Leave out the `end` key -- if not set,
1207 Icinga 2 will not check against any end time for this notification. Make sure to
1208 specify a relatively low notification `interval` to get notified soon enough again.
1210 apply Notification "mail" to Service {
1211 import "generic-notification"
1213 command = "mail-notification"
1214 users = [ "icingaadmin" ]
1218 times.begin = 15m // delay notification window
1220 assign where service.name == "ping4"
1223 ### Disable Re-notifications <a id="disable-renotification"></a>
1225 If you prefer to be notified only once, you can disable re-notifications by setting the
1226 `interval` attribute to `0`.
1228 apply Notification "notify-once" to Service {
1229 import "generic-notification"
1231 command = "mail-notification"
1232 users = [ "icingaadmin" ]
1234 interval = 0 // disable re-notification
1236 assign where service.name == "ping4"
1239 ### Notification Filters by State and Type <a id="notification-filters-state-type"></a>
1241 If there are no notification state and type filter attributes defined at the `Notification`
1242 or `User` object, Icinga 2 assumes that all states and types are being notified.
1244 Available state and type filters for notifications are:
1246 template Notification "generic-notification" {
1248 states = [ OK, Warning, Critical, Unknown ]
1249 types = [ Problem, Acknowledgement, Recovery, Custom, FlappingStart,
1250 FlappingEnd, DowntimeStart, DowntimeEnd, DowntimeRemoved ]
1253 If you are familiar with Icinga 1.x `notification_options`, please note that they have been split
1254 into type and state to allow more fine granular filtering for example on downtimes and flapping.
1255 You can filter for acknowledgements and custom notifications too.
1258 ## Commands <a id="commands"></a>
1260 Icinga 2 uses three different command object types to specify how
1261 checks should be performed, notifications should be sent, and
1262 events should be handled.
1264 ### Check Commands <a id="check-commands"></a>
1266 [CheckCommand](09-object-types.md#objecttype-checkcommand) objects define the command line how
1269 [CheckCommand](09-object-types.md#objecttype-checkcommand) objects are referenced by
1270 [Host](09-object-types.md#objecttype-host) and [Service](09-object-types.md#objecttype-service) objects
1271 using the `check_command` attribute.
1275 > Make sure that the [checker](11-cli-commands.md#enable-features) feature is enabled in order to
1278 #### Integrate the Plugin with a CheckCommand Definition <a id="command-plugin-integration"></a>
1280 Unless you have done so already, download your check plugin and put it
1281 into the [PluginDir](04-configuring-icinga-2.md#constants-conf) directory. The following example uses the
1282 `check_mysql` plugin contained in the Monitoring Plugins package.
1284 The plugin path and all command arguments are made a list of
1285 double-quoted string arguments for proper shell escaping.
1287 Call the `check_disk` plugin with the `--help` parameter to see
1288 all available options. Our example defines warning (`-w`) and
1289 critical (`-c`) thresholds for the disk usage. Without any
1290 partition defined (`-p`) it will check all local partitions.
1292 icinga@icinga2 $ /usr/lib64/nagios/plugins/check_mysql --help
1294 This program tests connections to a MySQL server
1297 check_mysql [-d database] [-H host] [-P port] [-s socket]
1298 [-u user] [-p password] [-S] [-l] [-a cert] [-k key]
1299 [-C ca-cert] [-D ca-dir] [-L ciphers] [-f optfile] [-g group]
1301 Next step is to understand how [command parameters](03-monitoring-basics.md#command-passing-parameters)
1302 are being passed from a host or service object, and add a [CheckCommand](09-object-types.md#objecttype-checkcommand)
1303 definition based on these required parameters and/or default values.
1305 Please continue reading in the [plugins section](05-service-monitoring.md#service-monitoring-plugins) for additional integration examples.
1307 #### Passing Check Command Parameters from Host or Service <a id="command-passing-parameters"></a>
1309 Check command parameters are defined as custom attributes which can be accessed as runtime macros
1310 by the executed check command.
1312 The check command parameters for ITL provided plugin check command definitions are documented
1313 [here](10-icinga-template-library.md#icinga-template-library), for example
1314 [disk](10-icinga-template-library.md#plugin-check-command-disk).
1316 In order to practice passing command parameters you should [integrate your own plugin](03-monitoring-basics.md#command-plugin-integration).
1318 The following example will use `check_mysql` provided by the [Monitoring Plugins installation](02-getting-started.md#setting-up-check-plugins).
1320 Define the default check command custom attributes, for example `mysql_user` and `mysql_password`
1321 (freely definable naming schema) and optional their default threshold values. You can
1322 then use these custom attributes as runtime macros for [command arguments](03-monitoring-basics.md#command-arguments)
1323 on the command line.
1327 > Use a common command type as prefix for your command arguments to increase
1328 > readability. `mysql_user` helps understanding the context better than just
1329 > `user` as argument.
1331 The default custom attributes can be overridden by the custom attributes
1332 defined in the host or service using the check command `my-mysql`. The custom attributes
1333 can also be inherited from a parent template using additive inheritance (`+=`).
1335 # vim /etc/icinga2/conf.d/commands.conf
1337 object CheckCommand "my-mysql" {
1338 command = [ PluginDir + "/check_mysql" ] //constants.conf -> const PluginDir
1341 "-H" = "$mysql_host$"
1344 value = "$mysql_user$"
1346 "-p" = "$mysql_password$"
1347 "-P" = "$mysql_port$"
1348 "-s" = "$mysql_socket$"
1349 "-a" = "$mysql_cert$"
1350 "-d" = "$mysql_database$"
1351 "-k" = "$mysql_key$"
1352 "-C" = "$mysql_ca_cert$"
1353 "-D" = "$mysql_ca_dir$"
1354 "-L" = "$mysql_ciphers$"
1355 "-f" = "$mysql_optfile$"
1356 "-g" = "$mysql_group$"
1358 set_if = "$mysql_check_slave$"
1359 description = "Check if the slave thread is running properly."
1362 set_if = "$mysql_ssl$"
1363 description = "Use ssl encryption"
1367 vars.mysql_check_slave = false
1368 vars.mysql_ssl = false
1369 vars.mysql_host = "$address$"
1372 The check command definition also sets `mysql_host` to the `$address$` default value. You can override
1373 this command parameter if for example your MySQL host is not running on the same server's ip address.
1375 Make sure pass all required command parameters, such as `mysql_user`, `mysql_password` and `mysql_database`.
1376 `MysqlUsername` and `MysqlPassword` are specified as [global constants](04-configuring-icinga-2.md#constants-conf)
1379 # vim /etc/icinga2/conf.d/services.conf
1381 apply Service "mysql-icinga-db-health" {
1382 import "generic-service"
1384 check_command = "my-mysql"
1386 vars.mysql_user = MysqlUsername
1387 vars.mysql_password = MysqlPassword
1389 vars.mysql_database = "icinga"
1390 vars.mysql_host = "192.168.33.11"
1392 assign where match("icinga2*", host.name)
1393 ignore where host.vars.no_health_check == true
1397 Take a different example: The example host configuration in [hosts.conf](04-configuring-icinga-2.md#hosts-conf)
1398 also applies an `ssh` service check. Your host's ssh port is not the default `22`, but set to `2022`.
1399 You can pass the command parameter as custom attribute `ssh_port` directly inside the service apply rule
1400 inside [services.conf](04-configuring-icinga-2.md#services-conf):
1402 apply Service "ssh" {
1403 import "generic-service"
1405 check_command = "ssh"
1406 vars.ssh_port = 2022 //custom command parameter
1408 assign where (host.address || host.address6) && host.vars.os == "Linux"
1411 If you prefer this being configured at the host instead of the service, modify the host configuration
1412 object instead. The runtime macro resolving order is described [here](03-monitoring-basics.md#macro-evaluation-order).
1414 object Host NodeName {
1416 vars.ssh_port = 2022
1419 #### Passing Check Command Parameters Using Apply For <a id="command-passing-parameters-apply-for"></a>
1421 The host `localhost` with the generated services from the `basic-partitions` dictionary (see
1422 [apply for](03-monitoring-basics.md#using-apply-for) for details) checks a basic set of disk partitions
1423 with modified custom attributes (warning thresholds at `10%`, critical thresholds at `5%`
1426 The custom attribute `disk_partition` can either hold a single string or an array of
1427 string values for passing multiple partitions to the `check_disk` check plugin.
1429 object Host "my-server" {
1430 import "generic-host"
1431 address = "127.0.0.1"
1434 vars.local_disks["basic-partitions"] = {
1435 disk_partitions = [ "/", "/tmp", "/var", "/home" ]
1439 apply Service for (disk => config in host.vars.local_disks) {
1440 import "generic-service"
1441 check_command = "my-disk"
1445 vars.disk_wfree = "10%"
1446 vars.disk_cfree = "5%"
1450 More details on using arrays in custom attributes can be found in
1451 [this chapter](03-monitoring-basics.md#custom-attributes).
1454 #### Command Arguments <a id="command-arguments"></a>
1456 By defining a check command line using the `command` attribute Icinga 2
1457 will resolve all macros in the static string or array. Sometimes it is
1458 required to extend the arguments list based on a met condition evaluated
1459 at command execution. Or making arguments optional -- only set if the
1460 macro value can be resolved by Icinga 2.
1462 object CheckCommand "check_http" {
1463 command = [ PluginDir + "/check_http" ]
1466 "-H" = "$http_vhost$"
1467 "-I" = "$http_address$"
1469 "-p" = "$http_port$"
1471 set_if = "$http_ssl$"
1474 set_if = "$http_sni$"
1477 value = "$http_auth_pair$"
1478 description = "Username:password on sites with basic authentication"
1481 set_if = "$http_ignore_body$"
1483 "-r" = "$http_expect_body_regex$"
1484 "-w" = "$http_warn_time$"
1485 "-c" = "$http_critical_time$"
1486 "-e" = "$http_expect$"
1489 vars.http_address = "$address$"
1490 vars.http_ssl = false
1491 vars.http_sni = false
1494 The example shows the `check_http` check command defining the most common
1495 arguments. Each of them is optional by default and will be omitted if
1496 the value is not set. For example, if the service calling the check command
1497 does not have `vars.http_port` set, it won't get added to the command
1500 If the `vars.http_ssl` custom attribute is set in the service, host or command
1501 object definition, Icinga 2 will add the `-S` argument based on the `set_if`
1502 numeric value to the command line. String values are not supported.
1504 If the macro value cannot be resolved, Icinga 2 will not add the defined argument
1505 to the final command argument array. Empty strings for macro values won't omit
1508 That way you can use the `check_http` command definition for both, with and
1509 without SSL enabled checks saving you duplicated command definitions.
1511 Details on all available options can be found in the
1512 [CheckCommand object definition](09-object-types.md#objecttype-checkcommand).
1515 #### Environment Variables <a id="command-environment-variables"></a>
1517 The `env` command object attribute specifies a list of environment variables with values calculated
1518 from either runtime macros or custom attributes which should be exported as environment variables
1519 prior to executing the command.
1521 This is useful for example for hiding sensitive information on the command line output
1522 when passing credentials to database checks:
1524 object CheckCommand "mysql-health" {
1526 PluginDir + "/check_mysql"
1530 "-H" = "$mysql_address$"
1531 "-d" = "$mysql_database$"
1534 vars.mysql_address = "$address$"
1535 vars.mysql_database = "icinga"
1536 vars.mysql_user = "icinga_check"
1537 vars.mysql_pass = "password"
1539 env.MYSQLUSER = "$mysql_user$"
1540 env.MYSQLPASS = "$mysql_pass$"
1545 ### Notification Commands <a id="notification-commands"></a>
1547 [NotificationCommand](09-object-types.md#objecttype-notificationcommand)
1548 objects define how notifications are delivered to external interfaces
1549 (email, XMPP, IRC, Twitter, etc.).
1550 [NotificationCommand](09-object-types.md#objecttype-notificationcommand)
1551 objects are referenced by [Notification](09-object-types.md#objecttype-notification)
1552 objects using the `command` attribute.
1556 > Make sure that the [notification](11-cli-commands.md#enable-features) feature is enabled
1557 > in order to execute notification commands.
1559 While it's possible to specify an entire notification command right
1560 in the NotificationCommand object it is generally advisable to create a
1561 shell script in the `/etc/icinga2/scripts` directory and have the
1562 NotificationCommand object refer to that.
1564 A fresh Icinga 2 install comes with with two example scripts for host
1565 and service notifications by email. Based on the Icinga 2 runtime macros
1566 (such as `$service.output$` for the current check output) it's possible
1567 to send email to the user(s) associated with the notification itself
1568 (`$user.email$`). Feel free to take these scripts as a starting point
1569 for your own individual notification solution - and keep in mind that
1570 nearly everything is technically possible.
1572 Information needed to generate notifications is passed to the scripts as
1573 arguments. The NotificationCommand objects `mail-host-notification` and
1574 `mail-service-notification` correspond to the shell scripts
1575 `mail-host-notification.sh` and `mail-service-notification.sh` in
1576 `/etc/icinga2/scripts` and define default values for arguments. These
1577 defaults can always be overwritten locally.
1581 > This example requires the `mail` binary installed on the Icinga 2
1584 #### Notification Commands in 2.7 <a id="notification-command-2-7"></a>
1586 Icinga 2 v2.7.0 introduced new notification scripts which support both
1587 environment variables and command line parameters.
1589 Therefore the `NotificationCommand` objects inside the [commands.conf](04-configuring-icinga-2.md#commands-conf)
1590 and `Notification` apply rules inside the [notifications.conf](04-configuring-icinga-2.md#notifications-conf)
1591 configuration files have been updated. Your configuration needs to be
1592 updated next to the notification scripts themselves.
1596 > Several parameters have been changed. Please review the notification
1597 > script parameters and configuration objects before updating your production
1600 The safest way is to incorporate the configuration updates from
1601 v2.7.0 inside the [commands.conf](04-configuring-icinga-2.md#commands-conf) and [notifications.conf](04-configuring-icinga-2.md#notifications-conf)
1602 configuration files.
1604 A quick-fix is shown below:
1607 @@ -5,7 +5,8 @@ object NotificationCommand "mail-host-notification" {
1610 NOTIFICATIONTYPE = "$notification.type$"
1611 - HOSTALIAS = "$host.display_name$"
1612 + HOSTNAME = "$host.name$"
1613 + HOSTDISPLAYNAME = "$host.display_name$"
1614 HOSTADDRESS = "$address$"
1615 HOSTSTATE = "$host.state$"
1616 LONGDATETIME = "$icinga.long_date_time$"
1617 @@ -22,8 +23,9 @@ object NotificationCommand "mail-service-notification" {
1620 NOTIFICATIONTYPE = "$notification.type$"
1621 - SERVICEDESC = "$service.name$"
1622 - HOSTALIAS = "$host.display_name$"
1623 + SERVICENAME = "$service.name$"
1624 + HOSTNAME = "$host.name$"
1625 + HOSTDISPLAYNAME = "$host.display_name$"
1626 HOSTADDRESS = "$address$"
1627 SERVICESTATE = "$service.state$"
1628 LONGDATETIME = "$icinga.long_date_time$"
1632 #### mail-host-notification <a id="mail-host-notification"></a>
1634 The `mail-host-notification` NotificationCommand object uses the
1635 example notification script located in `/etc/icinga2/scripts/mail-host-notification.sh`.
1637 Here is a quick overview of the arguments that can be used. See also [host runtime
1638 macros](03-monitoring-basics.md#-host-runtime-macros) for further
1642 -------------------------------|---------------------------------------
1643 `notification_address` | **Required.** The host's IPv4 address. Defaults to `$address$`.
1644 `notification_date` | **Required.** Date and time. Defaults to `$icinga.long_date_time$`.
1645 `notification_hostname` | **Required.** The host's `FQDN`. Defaults to `$host.name$`.
1646 `notification_hostdisplayname` | **Required.** The host's display name. Defaults to `$host.display_name$`.
1647 `notification_hostoutput` | **Required.** Output from host check. Defaults to `$host.output$`.
1648 `notification_useremail` | **Required.** The notification's recipient(s). Defaults to `$user.email$`.
1649 `notification_hoststate` | **Required.** Current state of host. Defaults to `$host.state$`.
1650 `notification_type` | **Required.** Type of notification. Defaults to `$notification.type$`.
1651 `notification_address6` | **Optional.** The host's IPv6 address. Defaults to `$address6$`.
1652 `notification_author` | **Optional.** Comment author. Defaults to `$notification.author$`.
1653 `notification_comment` | **Optional.** Comment text. Defaults to `$notification.comment$`.
1654 `notification_from` | **Optional.** Define a valid From: string (e.g. `"Icinga 2 Host Monitoring <icinga@example.com>"`). Requires `GNU mailutils` (Debian/Ubuntu) or `mailx` (RHEL/SUSE).
1655 `notification_icingaweb2url` | **Optional.** Define URL to your Icinga Web 2 (e.g. `"https://www.example.com/icingaweb2"`)
1656 `notification_logtosyslog` | **Optional.** Set `true` to log notification events to syslog; useful for debugging. Defaults to `false`.
1658 #### mail-service-notification <a id="mail-service-notification"></a>
1660 The `mail-service-notification` NotificationCommand object uses the
1661 example notification script located in `/etc/icinga2/scripts/mail-service-notification.sh`.
1663 Here is a quick overview of the arguments that can be used. See also [service runtime
1664 macros](03-monitoring-basics.md#-service-runtime-macros) for further
1668 ----------------------------------|---------------------------------------
1669 `notification_address` | **Required.** The host's IPv4 address. Defaults to `$address$`.
1670 `notification_date` | **Required.** Date and time. Defaults to `$icinga.long_date_time$`.
1671 `notification_hostname` | **Required.** The host's `FQDN`. Defaults to `$host.name$`.
1672 `notification_servicename` | **Required.** The service name. Defaults to `$service.name$`.
1673 `notification_hostdisplayname` | **Required.** Host display name. Defaults to `$host.display_name$`.
1674 `notification_servicedisplayname` | **Required.** Service display name. Defaults to `$service.display_name$`.
1675 `notification_serviceoutput` | **Required.** Output from service check. Defaults to `$service.output$`.
1676 `notification_useremail` | **Required.** The notification's recipient(s). Defaults to `$user.email$`.
1677 `notification_servicestate` | **Required.** Current state of host. Defaults to `$service.state$`.
1678 `notification_type` | **Required.** Type of notification. Defaults to `$notification.type$`.
1679 `notification_address6` | **Optional.** The host's IPv6 address. Defaults to `$address6$`.
1680 `notification_author` | **Optional.** Comment author. Defaults to `$notification.author$`.
1681 `notification_comment` | **Optional.** Comment text. Defaults to `$notification.comment$`.
1682 `notification_from` | **Optional.** Define a valid From: string (e.g. `"Icinga 2 Host Monitoring <icinga@example.com>"`). Requires `GNU mailutils` (Debian/Ubuntu) or `mailx` (RHEL/SUSE).
1683 `notification_icingaweb2url` | **Optional.** Define URL to your Icinga Web 2 (e.g. `"https://www.example.com/icingaweb2"`)
1684 `notification_logtosyslog` | **Optional.** Set `true` to log notification events to syslog; useful for debugging. Defaults to `false`.
1686 ### Event Commands <a id="event-commands"></a>
1688 Unlike notifications, event commands for hosts/services are called on every
1689 check execution if one of these conditions matches:
1691 * The host/service is in a [soft state](03-monitoring-basics.md#hard-soft-states)
1692 * The host/service state changes into a [hard state](03-monitoring-basics.md#hard-soft-states)
1693 * The host/service state recovers from a [soft or hard state](03-monitoring-basics.md#hard-soft-states) to [OK](03-monitoring-basics.md#service-states)/[Up](03-monitoring-basics.md#host-states)
1695 [EventCommand](09-object-types.md#objecttype-eventcommand) objects are referenced by
1696 [Host](09-object-types.md#objecttype-host) and [Service](09-object-types.md#objecttype-service) objects
1697 with the `event_command` attribute.
1699 Therefore the `EventCommand` object should define a command line
1700 evaluating the current service state and other service runtime attributes
1701 available through runtime variables. Runtime macros such as `$service.state_type$`
1702 and `$service.state$` will be processed by Icinga 2 and help with fine-granular
1705 If the host/service is located on a client as [command endpoint](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint)
1706 the event command will be executed on the client itself (similar to the check
1709 Common use case scenarios are a failing HTTP check which requires an immediate
1710 restart via event command. Another example would be an application that is not
1711 responding and therefore requires a restart. You can also use event handlers
1712 to forward more details on state changes and events than the typical notification
1715 #### Use Event Commands to Send Information from the Master <a id="event-command-send-information-from-master"></a>
1717 This example sends a web request from the master node to an external tool
1718 for every event triggered on a `businessprocess` service.
1720 Define an [EventCommand](09-object-types.md#objecttype-eventcommand)
1721 object `send_to_businesstool` which sends state changes to the external tool.
1723 object EventCommand "send_to_businesstool" {
1732 value ="$businesstool_url$"
1735 "-d" = "$businesstool_message$"
1738 vars.businesstool_url = "http://localhost:8080/businesstool"
1739 vars.businesstool_message = "$host.name$ $service.name$ $service.state$ $service.state_type$ $service.check_attempt$"
1742 Set the `event_command` attribute to `send_to_businesstool` on the Service.
1744 object Service "businessprocess" {
1745 host_name = "businessprocess"
1747 check_command = "icingacli-businessprocess"
1748 vars.icingacli_businessprocess_process = "icinga"
1749 vars.icingacli_businessprocess_config = "training"
1751 event_command = "send_to_businesstool"
1754 In order to test this scenario you can run:
1758 This allows to catch the web request. You can also enable the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output)
1759 and search for the event command execution log message.
1761 tail -f /var/log/icinga2/debug.log | grep EventCommand
1763 Feed in a check result via REST API action [process-check-result](12-icinga2-api.md#icinga2-api-actions-process-check-result)
1764 or via Icinga Web 2.
1769 PUT /businesstool HTTP/1.1
1770 User-Agent: curl/7.29.0
1771 Host: localhost:8080
1774 Content-Type: application/x-www-form-urlencoded
1776 businessprocess businessprocess CRITICAL SOFT 1
1779 #### Use Event Commands to Restart Service Daemon via Command Endpoint on Linux <a id="event-command-restart-service-daemon-command-endpoint-linux"></a>
1781 This example triggers a restart of the `httpd` service on the local system
1782 when the `procs` service check executed via Command Endpoint fails. It only
1783 triggers if the service state is `Critical` and attempts to restart the
1784 service before a notification is sent.
1788 * Icinga 2 as client on the remote node
1789 * icinga user with sudo permissions to the httpd daemon
1791 Example on CentOS 7:
1794 icinga ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart httpd
1796 Note: Distributions might use a different name. On Debian/Ubuntu the service is called `apache2`.
1798 Define an [EventCommand](09-object-types.md#objecttype-eventcommand) object `restart_service`
1799 which allows to trigger local service restarts. Put it into a [global zone](06-distributed-monitoring.md#distributed-monitoring-global-zone-config-sync)
1800 to sync its configuration to all clients.
1802 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/global-templates/eventcommands.conf
1804 object EventCommand "restart_service" {
1805 command = [ PluginDir + "/restart_service" ]
1808 "-s" = "$service.state$"
1809 "-t" = "$service.state_type$"
1810 "-a" = "$service.check_attempt$"
1811 "-S" = "$restart_service$"
1814 vars.restart_service = "$procs_command$"
1817 This event command triggers the following script which restarts the service.
1818 The script only is executed if the service state is `CRITICAL`. Warning and Unknown states
1819 are ignored as they indicate not an immediate failure.
1821 [root@icinga2-client1.localdomain /]# vim /usr/lib64/nagios/plugins/restart_service
1825 while getopts "s:t:a:S:" opt; do
1828 servicestate=$OPTARG
1831 servicestatetype=$OPTARG
1834 serviceattempt=$OPTARG
1842 if ( [ -z $servicestate ] || [ -z $servicestatetype ] || [ -z $serviceattempt ] || [ -z $service ] ); then
1843 echo "USAGE: $0 -s servicestate -z servicestatetype -a serviceattempt -S service"
1846 # Only restart on the third attempt of a critical event
1847 if ( [ $servicestate == "CRITICAL" ] && [ $servicestatetype == "SOFT" ] && [ $serviceattempt -eq 3 ] ); then
1848 sudo /usr/bin/systemctl restart $service
1852 [root@icinga2-client1.localdomain /]# chmod +x /usr/lib64/nagios/plugins/restart_service
1855 Add a service on the master node which is executed via command endpoint on the client.
1856 Set the `event_command` attribute to `restart_service`, the name of the previously defined
1857 EventCommand object.
1859 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/icinga2-client1.localdomain.conf
1861 object Service "Process httpd" {
1862 check_command = "procs"
1863 event_command = "restart_service"
1864 max_check_attempts = 4
1866 host_name = "icinga2-client1.localdomain"
1867 command_endpoint = "icinga2-client1.localdomain"
1869 vars.procs_command = "httpd"
1870 vars.procs_warning = "1:10"
1871 vars.procs_critical = "1:"
1874 In order to test this configuration just stop the `httpd` on the remote host `icinga2-client1.localdomain`.
1876 [root@icinga2-client1.localdomain /]# systemctl stop httpd
1878 You can enable the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) and search for the
1879 executed command line.
1881 [root@icinga2-client1.localdomain /]# tail -f /var/log/icinga2/debug.log | grep restart_service
1884 #### Use Event Commands to Restart Service Daemon via Command Endpoint on Windows <a id="event-command-restart-service-daemon-command-endpoint-windows"></a>
1886 This example triggers a restart of the `httpd` service on the remote system
1887 when the `service-windows` service check executed via Command Endpoint fails.
1888 It only triggers if the service state is `Critical` and attempts to restart the
1889 service before a notification is sent.
1893 * Icinga 2 as client on the remote node
1894 * Icinga 2 service with permissions to execute Powershell scripts (which is the default)
1896 Define an [EventCommand](09-object-types.md#objecttype-eventcommand) object `restart_service-windows`
1897 which allows to trigger local service restarts. Put it into a [global zone](06-distributed-monitoring.md#distributed-monitoring-global-zone-config-sync)
1898 to sync its configuration to all clients.
1900 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/global-templates/eventcommands.conf
1902 object EventCommand "restart_service-windows" {
1904 "C:\\Windows\\SysWOW64\\WindowsPowerShell\\v1.0\\powershell.exe",
1905 PluginDir + "/restart_service.ps1"
1909 "-ServiceState" = "$service.state$"
1910 "-ServiceStateType" = "$service.state_type$"
1911 "-ServiceAttempt" = "$service.check_attempt$"
1912 "-Service" = "$restart_service$"
1915 value = "$$LASTEXITCODE"
1919 vars.restart_service = "$service_win_service$"
1922 This event command triggers the following script which restarts the service.
1923 The script only is executed if the service state is `CRITICAL`. Warning and Unknown states
1924 are ignored as they indicate not an immediate failure.
1926 Add the `restart_service.ps1` Powershell script into `C:\Program Files\Icinga2\sbin`:
1929 [string]$Service = '',
1930 [string]$ServiceState = '',
1931 [string]$ServiceStateType = '',
1932 [int]$ServiceAttempt = ''
1935 if (!$Service -Or !$ServiceState -Or !$ServiceStateType -Or !$ServiceAttempt) {
1936 $scriptName = GCI $MyInvocation.PSCommandPath | Select -Expand Name;
1937 Write-Host "USAGE: $scriptName -ServiceState servicestate -ServiceStateType servicestatetype -ServiceAttempt serviceattempt -Service service" -ForegroundColor red;
1941 # Only restart on the third attempt of a critical event
1942 if ($ServiceState -eq "CRITICAL" -And $ServiceStateType -eq "SOFT" -And $ServiceAttempt -eq 3) {
1943 Restart-Service $Service;
1948 Add a service on the master node which is executed via command endpoint on the client.
1949 Set the `event_command` attribute to `restart_service-windows`, the name of the previously defined
1950 EventCommand object.
1952 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/icinga2-client2.localdomain.conf
1954 object Service "Service httpd" {
1955 check_command = "service-windows"
1956 event_command = "restart_service-windows"
1957 max_check_attempts = 4
1959 host_name = "icinga2-client2.localdomain"
1960 command_endpoint = "icinga2-client2.localdomain"
1962 vars.service_win_service = "httpd"
1965 In order to test this configuration just stop the `httpd` on the remote host `icinga2-client1.localdomain`.
1969 You can enable the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) and search for the
1970 executed command line in `C:\ProgramData\icinga2\var\log\icinga2\debug.log`.
1973 #### Use Event Commands to Restart Service Daemon via SSH <a id="event-command-restart-service-daemon-ssh"></a>
1975 This example triggers a restart of the `httpd` daemon
1976 via SSH when the `http` service check fails.
1980 * SSH connection allowed (firewall, packet filters)
1981 * icinga user with public key authentication
1982 * icinga user with sudo permissions to restart the httpd daemon.
1986 # ls /home/icinga/.ssh/
1990 icinga ALL=(ALL) NOPASSWD: /etc/init.d/apache2 restart
1992 Define a generic [EventCommand](09-object-types.md#objecttype-eventcommand) object `event_by_ssh`
1993 which can be used for all event commands triggered using SSH:
1995 [root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/local_eventcommands.conf
1997 /* pass event commands through ssh */
1998 object EventCommand "event_by_ssh" {
1999 command = [ PluginDir + "/check_by_ssh" ]
2002 "-H" = "$event_by_ssh_address$"
2003 "-p" = "$event_by_ssh_port$"
2004 "-C" = "$event_by_ssh_command$"
2005 "-l" = "$event_by_ssh_logname$"
2006 "-i" = "$event_by_ssh_identity$"
2008 set_if = "$event_by_ssh_quiet$"
2010 "-w" = "$event_by_ssh_warn$"
2011 "-c" = "$event_by_ssh_crit$"
2012 "-t" = "$event_by_ssh_timeout$"
2015 vars.event_by_ssh_address = "$address$"
2016 vars.event_by_ssh_quiet = false
2019 The actual event command only passes the `event_by_ssh_command` attribute.
2020 The `event_by_ssh_service` custom attribute takes care of passing the correct
2021 daemon name, while `test $service.state_id$ -gt 0` makes sure that the daemon
2022 is only restarted when the service is not in an `OK` state.
2024 object EventCommand "event_by_ssh_restart_service" {
2025 import "event_by_ssh"
2027 //only restart the daemon if state > 0 (not-ok)
2028 //requires sudo permissions for the icinga user
2029 vars.event_by_ssh_command = "test $service.state_id$ -gt 0 && sudo systemctl restart $event_by_ssh_service$"
2033 Now set the `event_command` attribute to `event_by_ssh_restart_service` and tell it
2034 which service should be restarted using the `event_by_ssh_service` attribute.
2036 apply Service "http" {
2037 import "generic-service"
2038 check_command = "http"
2040 event_command = "event_by_ssh_restart_service"
2041 vars.event_by_ssh_service = "$host.vars.httpd_name$"
2043 //vars.event_by_ssh_logname = "icinga"
2044 //vars.event_by_ssh_identity = "/home/icinga/.ssh/id_rsa.pub"
2046 assign where host.vars.httpd_name
2049 Specify the `httpd_name` custom attribute on the host to assign the
2050 service and set the event handler service.
2052 object Host "remote-http-host" {
2053 import "generic-host"
2054 address = "192.168.1.100"
2056 vars.httpd_name = "apache2"
2059 In order to test this configuration just stop the `httpd` on the remote host `icinga2-client1.localdomain`.
2061 [root@icinga2-client1.localdomain /]# systemctl stop httpd
2063 You can enable the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) and search for the
2064 executed command line.
2066 [root@icinga2-client1.localdomain /]# tail -f /var/log/icinga2/debug.log | grep by_ssh
2068 ## Dependencies <a id="dependencies"></a>
2070 Icinga 2 uses host and service [Dependency](09-object-types.md#objecttype-dependency) objects
2071 for determining their network reachability.
2073 A service can depend on a host, and vice versa. A service has an implicit
2074 dependency (parent) to its host. A host to host dependency acts implicitly
2075 as host parent relation.
2076 When dependencies are calculated, not only the immediate parent is taken into
2077 account but all parents are inherited.
2079 The `parent_host_name` and `parent_service_name` attributes are mandatory for
2080 service dependencies, `parent_host_name` is required for host dependencies.
2081 [Apply rules](03-monitoring-basics.md#using-apply) will allow you to
2082 [determine these attributes](03-monitoring-basics.md#dependencies-apply-custom-attributes) in a more
2083 dynamic fashion if required.
2085 parent_host_name = "core-router"
2086 parent_service_name = "uplink-port"
2088 Notifications are suppressed by default if a host or service becomes unreachable.
2089 You can control that option by defining the `disable_notifications` attribute.
2091 disable_notifications = false
2093 If the dependency should be triggered in the parent object's soft state, you
2094 need to set `ignore_soft_states` to `false`.
2096 The dependency state filter must be defined based on the parent object being
2097 either a host (`Up`, `Down`) or a service (`OK`, `Warning`, `Critical`, `Unknown`).
2099 The following example will make the dependency fail and trigger it if the parent
2100 object is **not** in one of these states:
2102 states = [ OK, Critical, Unknown ]
2104 Rephrased: If the parent service object changes into the `Warning` state, this
2105 dependency will fail and render all child objects (hosts or services) unreachable.
2107 You can determine the child's reachability by querying the `is_reachable` attribute
2108 in for example [DB IDO](23-appendix.md#schema-db-ido-extensions).
2110 ### Implicit Dependencies for Services on Host <a id="dependencies-implicit-host-service"></a>
2112 Icinga 2 automatically adds an implicit dependency for services on their host. That way
2113 service notifications are suppressed when a host is `DOWN` or `UNREACHABLE`. This dependency
2114 does not overwrite other dependencies and implicitely sets `disable_notifications = true` and
2115 `states = [ Up ]` for all service objects.
2117 Service checks are still executed. If you want to prevent them from happening, you can
2118 apply the following dependency to all services setting their host as `parent_host_name`
2119 and disabling the checks. `assign where true` matches on all `Service` objects.
2121 apply Dependency "disable-host-service-checks" to Service {
2122 disable_checks = true
2126 ### Dependencies for Network Reachability <a id="dependencies-network-reachability"></a>
2128 A common scenario is the Icinga 2 server behind a router. Checking internet
2129 access by pinging the Google DNS server `google-dns` is a common method, but
2130 will fail in case the `dsl-router` host is down. Therefore the example below
2131 defines a host dependency which acts implicitly as parent relation too.
2133 Furthermore the host may be reachable but ping probes are dropped by the
2134 router's firewall. In case the `dsl-router`'s `ping4` service check fails, all
2135 further checks for the `ping4` service on host `google-dns` service should
2136 be suppressed. This is achieved by setting the `disable_checks` attribute to `true`.
2138 object Host "dsl-router" {
2139 import "generic-host"
2140 address = "192.168.1.1"
2143 object Host "google-dns" {
2144 import "generic-host"
2148 apply Service "ping4" {
2149 import "generic-service"
2151 check_command = "ping4"
2153 assign where host.address
2156 apply Dependency "internet" to Host {
2157 parent_host_name = "dsl-router"
2158 disable_checks = true
2159 disable_notifications = true
2161 assign where host.name != "dsl-router"
2164 apply Dependency "internet" to Service {
2165 parent_host_name = "dsl-router"
2166 parent_service_name = "ping4"
2167 disable_checks = true
2169 assign where host.name != "dsl-router"
2172 ### Apply Dependencies based on Custom Attributes <a id="dependencies-apply-custom-attributes"></a>
2174 You can use [apply rules](03-monitoring-basics.md#using-apply) to set parent or
2175 child attributes, e.g. `parent_host_name` to other objects'
2178 A common example are virtual machines hosted on a master. The object
2179 name of that master is auto-generated from your CMDB or VMWare inventory
2180 into the host's custom attributes (or a generic template for your
2183 Define your master host object:
2186 object Host "master.example.com" {
2187 import "generic-host"
2190 Add a generic template defining all common host attributes:
2192 /* generic template for your virtual machines */
2193 template Host "generic-vm" {
2194 import "generic-host"
2197 Add a template for all hosts on your example.com cloud setting
2198 custom attribute `vm_parent` to `master.example.com`:
2200 template Host "generic-vm-example.com" {
2202 vars.vm_parent = "master.example.com"
2205 Define your guest hosts:
2207 object Host "www.example1.com" {
2208 import "generic-vm-master.example.com"
2211 object Host "www.example2.com" {
2212 import "generic-vm-master.example.com"
2215 Apply the host dependency to all child hosts importing the
2216 `generic-vm` template and set the `parent_host_name`
2217 to the previously defined custom attribute `host.vars.vm_parent`.
2219 apply Dependency "vm-host-to-parent-master" to Host {
2220 parent_host_name = host.vars.vm_parent
2221 assign where "generic-vm" in host.templates
2224 You can extend this example, and make your services depend on the
2225 `master.example.com` host too. Their local scope allows you to use
2226 `host.vars.vm_parent` similar to the example above.
2228 apply Dependency "vm-service-to-parent-master" to Service {
2229 parent_host_name = host.vars.vm_parent
2230 assign where "generic-vm" in host.templates
2233 That way you don't need to wait for your guest hosts becoming
2234 unreachable when the master host goes down. Instead the services
2235 will detect their reachability immediately when executing checks.
2239 > This method with setting locally scoped variables only works in
2240 > apply rules, but not in object definitions.
2243 ### Dependencies for Agent Checks <a id="dependencies-agent-checks"></a>
2245 Another classic example are agent based checks. You would define a health check
2246 for the agent daemon responding to your requests, and make all other services
2247 querying that daemon depend on that health check.
2249 The following configuration defines two nrpe based service checks `nrpe-load`
2250 and `nrpe-disk` applied to the host `nrpe-server` [matched](18-library-reference.md#global-functions-match)
2251 by its name. The health check is defined as `nrpe-health` service.
2253 apply Service "nrpe-health" {
2254 import "generic-service"
2255 check_command = "nrpe"
2256 assign where match("nrpe-*", host.name)
2259 apply Service "nrpe-load" {
2260 import "generic-service"
2261 check_command = "nrpe"
2262 vars.nrpe_command = "check_load"
2263 assign where match("nrpe-*", host.name)
2266 apply Service "nrpe-disk" {
2267 import "generic-service"
2268 check_command = "nrpe"
2269 vars.nrpe_command = "check_disk"
2270 assign where match("nrpe-*", host.name)
2273 object Host "nrpe-server" {
2274 import "generic-host"
2275 address = "192.168.1.5"
2278 apply Dependency "disable-nrpe-checks" to Service {
2279 parent_service_name = "nrpe-health"
2282 disable_checks = true
2283 disable_notifications = true
2284 assign where service.check_command == "nrpe"
2285 ignore where service.name == "nrpe-health"
2288 The `disable-nrpe-checks` dependency is applied to all services
2289 on the `nrpe-service` host using the `nrpe` check_command attribute
2290 but not the `nrpe-health` service itself.