The `address` attribute is used by check commands to determine which network
address is associated with the host object.
-Details on troubleshooting check problems can be found [here](#troubleshooting).
+Details on troubleshooting check problems can be found [here](16-troubleshooting.md#troubleshooting).
### <a id="host-states"></a> Host States
### <a id="host-service-checks"></a> Host and Service Checks
-Hosts and Services determine their state from a check result returned from a check
-execution to the Icinga 2 application. By default the `generic-host` example template
-will define `hostalive` as host check. If your host is unreachable for ping, you should
-consider using a different check command, for instance the `http` check command, or if
-there is no check available, the `dummy` check command.
+Hosts and services determine their state by running checks in a regular interval.
- object Host "uncheckable-host" {
- check_command = "dummy"
- vars.dummy_state = 1
- vars.dummy_text = "Pretending to be OK."
+ object Host "router" {
+ check_command = "hostalive"
+ address = "10.0.0.1"
}
-Service checks could also use a `dummy` check, but the common strategy is to
-[integrate an existing plugin](#command-plugin-integration) as
-[check command](#check-commands) and [reference](#command-passing-parameters)
-that in your [Service](#objecttype-service) object definition.
-
-## <a id="configuration-best-practice"></a> Configuration Best Practice
-
-The [Getting Started](#getting-started) chapter already introduced various aspects
-of the Icinga 2 configuration language. If you are ready to configure additional
-hosts, services, notifications, dependencies, etc, you should think about the
-requirements first and then decide for a possible strategy.
-
-There are many ways of creating Icinga 2 configuration objects:
-
-* Manually with your preferred editor, for example vi(m), nano, notepad, etc.
-* Generated by a configuration management tool such as Puppet, Chef, Ansible, etc.
-* A configuration addon for Icinga 2
-* A custom exporter script from your CMDB or inventory tool
-* your own.
-
-In order to find the best strategy for your own configuration, ask yourself the following questions:
-
-* Do your hosts share a common group of services (for example linux hosts with disk, load, etc checks)?
-* Only a small set of users receives notifications and escalations for all hosts/services?
+The `hostalive` command is one of several built-in check commands. It sends ICMP
+echo requests to the IP address specified in the `address` attribute to determine
+whether a host is online.
-If you can at least answer one of these questions with yes, look for the [apply rules](#using-apply) logic
-instead of defining objects on a per host and service basis.
+A number of other [built-in check commands](7-icinga-template-library.md#plugin-check-commands) are also
+available. In addition to these commands the next few chapters will explain in
+detail how to set up your own check commands.
-* You are required to define specific configuration for each host/service?
-* Does your configuration generation tool already know about the host-service-relationship?
-Then you should look for the object specific configuration setting `host_name` etc accordingly.
-
-Finding the best files and directory tree for your configuration is up to you. Make sure that
-the [icinga2.conf](#icinga2-conf) configuration file includes them, and then think about:
-
-* tree-based on locations, hostgroups, specific host attributes with sub levels of directories.
-* flat `hosts.conf`, `services.conf`, etc files for rule based configuration.
-* generated configuration with one file per host and a global configuration for groups, users, etc.
-* one big file generated from an external application (probably a bad idea for maintaining changes).
-* your own.
-
-In either way of choosing the right strategy you should additionally check the following:
-
-* Are there any specific attributes describing the host/service you could set as `vars` custom attributes?
-You can later use them for applying assign/ignore rules, or export them into external interfaces.
-* Put hosts into hostgroups, services into servicegroups and use these attributes for your apply rules.
-* Use templates to store generic attributes for your objects and apply rules making your configuration more readable.
-Details can be found in the [using templates](#using-templates) chapter.
-* Apply rules may overlap. Keep a central place (for example, `services.conf` or `notifications.conf`) storing
-the configuration instead of defining apply rules deep in your configuration tree.
-* Every plugin used as check, notification or event command requires a `Command` definition.
-Further details can be looked up in the [check commands](#check-commands) chapter.
-
-If you happen to have further questions, do not hesitate to join the [community support channels](https://support.icinga.org)
-and ask community members for their experience and best practices.
-
-
-### <a id="object-inheritance-using-templates"></a> Object Inheritance Using Templates
+## <a id="object-inheritance-using-templates"></a> Templates
Templates may be used to apply a set of identical attributes to more than one
object:
enable_perfdata = true
}
- object Service "ping4" {
+ apply Service "ping4" {
import "generic-service"
- host_name = "localhost"
check_command = "ping4"
+
+ assign where host.address
}
- object Service "ping6" {
+ apply Service "ping6" {
import "generic-service"
- host_name = "localhost"
check_command = "ping6"
+
+ assign where host.address6
}
+
In this example the `ping4` and `ping6` services inherit properties from the
template `generic-service`.
Objects as well as templates themselves can import an arbitrary number of
-templates. Attributes inherited from a template can be overridden in the
+other templates. Attributes inherited from a template can be overridden in the
object if necessary.
-### <a id="using-apply"></a> Apply objects based on rules
+You can also import existing non-template objects. Note that templates
+and objects share the same namespace, i.e. you can't define a template
+that has the same name like an object.
-Instead of assigning each object (`Service`, `Notification`, `Dependency`, `ScheduledDowntime`)
-based on attribute identifiers for example `host_name` objects can be [applied](#apply).
-Detailed scenario examples are used in their respective chapters, for example
-[apply services with custom command arguments](#using-apply-services-command-arguments).
+## <a id="custom-attributes"></a> Custom Attributes
-#### <a id="using-apply-services"></a> Apply Services to Hosts
+In addition to built-in attributes you can define your own attributes:
- apply Service "load" {
- import "generic-service"
+ object Host "localhost" {
+ vars.ssh_port = 2222
+ }
- check_command = "load"
+Valid values for custom attributes include:
- assign where "linux-server" in host.groups
- ignore where host.vars.no_load_check
- }
+* Strings and numbers
+* Arrays and dictionaries
+* Functions
-In this example the `load` service will be created as object for all hosts in the `linux-server`
-host group. If the `no_load_check` custom attribute is set, the host will be
-ignored.
+### <a id="custom-attributes-functions"></a> Functions as Custom Attributes
-#### <a id="using-apply-notifications"></a> Apply Notifications to Hosts and Services
+Icinga lets you specify functions for custom attributes. The special case here
+is that whenever Icinga needs the value for such a custom attribute it runs
+the function and uses whatever value the function returns:
-Notifications are applied to specific targets (`Host` or `Service`) and work in a similar
-manner:
+ object CheckCommand "random-value" {
+ import "plugin-check-command"
- apply Notification "mail-noc" to Service {
- import "mail-service-notification"
- command = "mail-service-notification"
- user_groups = [ "noc" ]
+ command = [ PluginDir + "/check_dummy", "0", "$text$" ]
- assign where service.vars.sla == "24x7"
+ vars.text = {{ Math.random() * 100 }}
}
-In this example the `mail-noc` notification will be created as object for all services having the
-`sla` custom attribute set to `24x7`. The notification command is set to `mail-service-notification`
-and all members of the user group `noc` will get notified.
+This example uses the [abbreviated lambda syntax](19-language-reference.md#nullary-lambdas).
-#### <a id="using-apply-dependencies"></a> Apply Dependencies to Hosts and Services
+These functions have access to a number of variables:
-Detailed examples can be found in the [dependencies](#dependencies) chapter.
-
-### <a id="using-apply-scheduledowntimes"></a> Apply Recurring Downtimes to Hosts and Services
+ Variable | Description
+ -------------|---------------
+ user | The User object (for notifications).
+ service | The Service object (for service checks/notifications/event handlers).
+ host | The Host object.
+ command | The command object (e.g. a CheckCommand object for checks).
-Detailed examples can be found in the [recurring downtimes](#recurring-downtimes) chapter.
+Here's an example:
+ vars.text = {{ host.check_interval }}
-### <a id="groups"></a> Groups
+In addition to these variables the `macro` function can be used to retrieve the
+value of arbitrary macro expressions:
-Groups are used for combining hosts, services, and users into
-accessible configuration attributes and views in external (web)
-interfaces.
+ vars.text = {{
+ if (macro("$address$") == "127.0.0.1") {
+ log("Running a check for localhost!")
+ }
-Group membership is defined at the respective object itself. If
-you have a hostgroup name `windows` for example, and want to assign
-specific hosts to this group for later viewing the group on your
-alert dashboard, first create the hostgroup:
+ return "Some text"
+ }}
- object HostGroup "windows" {
- display_name = "Windows Servers"
- }
+The [Object Accessor Functions](20-library-reference.md#object-accessor-functions) can be used to retrieve references
+to other objects by name.
-Then add your hosts to this hostgroup
+## <a id="runtime-macros"></a> Runtime Macros
- template Host "windows-server" {
- groups += [ "windows" ]
- }
+Macros can be used to access other objects' attributes at runtime. For example they
+are used in command definitions to figure out which IP address a check should be
+run against:
- object Host "mssql-srv1" {
- import "windows-server"
+ object CheckCommand "my-ping" {
+ import "plugin-check-command"
- vars.mssql_port = 1433
- }
+ command = [ PluginDir + "/check_ping", "-H", "$ping_address$" ]
- object Host "mssql-srv2" {
- import "windows-server"
+ arguments = {
+ "-w" = "$ping_wrta$,$ping_wpl$%"
+ "-c" = "$ping_crta$,$ping_cpl$%"
+ "-p" = "$ping_packets$"
+ }
- vars.mssql_port = 1433
- }
+ vars.ping_wrta = 100
+ vars.ping_wpl = 5
-This can be done for service and user groups the same way. Additionally
-the user groups are associated as attributes in `Notification` objects.
+ vars.ping_crta = 250
+ vars.ping_cpl = 10
- object UserGroup "windows-mssql-admins" {
- display_name = "Windows MSSQL Admins"
+ vars.ping_packets = 5
}
- template User "generic-windows-mssql-users" {
- groups += [ "windows-mssql-admins" ]
+ object Host "router" {
+ check_command = "my-ping"
+ address = "10.0.0.1"
}
- object User "win-mssql-noc" {
- import "generic-windows-mssql-users"
+In this example we are using the `$address$` macro to refer to the host's `address`
+attribute.
- email = "noc@example.com"
- }
+We can also directly refer to custom attributes, e.g. by using `$ping_wrta$`. Icinga
+automatically tries to find the closest match for the attribute you specified. The
+exact rules for this are explained in the next section.
- object User "win-mssql-ops" {
- import "generic-windows-mssql-users"
- email = "ops@example.com"
- }
+### <a id="macro-evaluation-order"></a> Evaluation Order
+
+When executing commands Icinga 2 checks the following objects in this order to look
+up macros and their respective values:
+
+1. User object (only for notifications)
+2. Service object
+3. Host object
+4. Command object
+5. Global custom attributes in the `Vars` constant
+
+This execution order allows you to define default values for custom attributes
+in your command objects.
-#### <a id="group-assign"></a> Group Membership Assign
+Here's how you can override the custom attribute `ping_packets` from the previous
+example:
-If there is a certain number of hosts, services, or users matching a pattern
-it's reasonable to assign the group object to these members.
-Details on the `assign where` syntax can be found [here](#apply)
+ object Service "ping" {
+ host_name = "localhost"
+ check_command = "my-ping"
- object HostGroup "mssql" {
- display_name = "MSSQL Servers"
- assign where host.vars.mssql_port
+ vars.ping_packets = 10 // Overrides the default value of 5 given in the command
}
-In this inherited example from above all hosts with the `vars` attribute `mssql_port`
-set will be added as members to the host group `mssql`.
+If a custom attribute isn't defined anywhere an empty value is used and a warning is
+written to the Icinga 2 log.
-## <a id="notifications"></a> Notifications
+You can also directly refer to a specific attribute - thereby ignoring these evaluation
+rules - by specifying the full attribute name:
-Notifications for service and host problems are an integral part of your
-monitoring setup.
+ $service.vars.ping_wrta$
-When a host or service is in a downtime, a problem has been acknowledged or
-the dependency logic determined that the host/service is unreachable, no
-notifications are sent. You can configure additional type and state filters
-refining the notifications being actually sent.
+This retrieves the value of the `ping_wrta` custom attribute for the service. This
+returns an empty value if the server does not have such a custom attribute no matter
+whether another object such as the host has this attribute.
-There are many ways of sending notifications, e.g. by e-mail, XMPP,
-IRC, Twitter, etc. On its own Icinga 2 does not know how to send notifications.
-Instead it relies on external mechanisms such as shell scripts to notify users.
-A notification specification requires one or more users (and/or user groups)
-who will be notified in case of problems. These users must have all custom
-attributes defined which will be used in the `NotificationCommand` on execution.
+### <a id="host-runtime-macros"></a> Host Runtime Macros
-The user `icingaadmin` in the example below will get notified only on `WARNING` and
-`CRITICAL` states and `problem` and `recovery` notification types.
+The following host custom attributes are available in all commands that are executed for
+hosts or services:
- object User "icingaadmin" {
- display_name = "Icinga 2 Admin"
- enable_notifications = true
- states = [ OK, Warning, Critical ]
- types = [ Problem, Recovery ]
- email = "icinga@localhost"
- }
+ Name | Description
+ -----------------------------|--------------
+ host.name | The name of the host object.
+ host.display_name | The value of the `display_name` attribute.
+ host.state | The host's current state. Can be one of `UNREACHABLE`, `UP` and `DOWN`.
+ host.state_id | The host's current state. Can be one of `0` (up), `1` (down) and `2` (unreachable).
+ host.state_type | The host's current state type. Can be one of `SOFT` and `HARD`.
+ host.check_attempt | The current check attempt number.
+ host.max_check_attempts | The maximum number of checks which are executed before changing to a hard state.
+ host.last_state | The host's previous state. Can be one of `UNREACHABLE`, `UP` and `DOWN`.
+ host.last_state_id | The host's previous state. Can be one of `0` (up), `1` (down) and `2` (unreachable).
+ host.last_state_type | The host's previous state type. Can be one of `SOFT` and `HARD`.
+ host.last_state_change | The last state change's timestamp.
+ host.downtime_depth | The number of active downtimes.
+ host.duration_sec | The time since the last state change.
+ host.latency | The host's check latency.
+ host.execution_time | The host's check execution time.
+ host.output | The last check's output.
+ host.perfdata | The last check's performance data.
+ host.last_check | The timestamp when the last check was executed.
+ host.check_source | The monitoring instance that performed the last check.
+ host.num_services | Number of services associated with the host.
+ host.num_services_ok | Number of services associated with the host which are in an `OK` state.
+ host.num_services_warning | Number of services associated with the host which are in a `WARNING` state.
+ host.num_services_unknown | Number of services associated with the host which are in an `UNKNOWN` state.
+ host.num_services_critical | Number of services associated with the host which are in a `CRITICAL` state.
-If you don't set the `states` and `types` configuration attributes for the `User`
-object, notifications for all states and types will be sent.
+### <a id="service-runtime-macros"></a> Service Runtime Macros
-Details on troubleshooting notification problems can be found [here](#troubleshooting).
+The following service macros are available in all commands that are executed for
+services:
-> **Note**
->
-> Make sure that the [notification](#features) feature is enabled on your master instance
-> in order to execute notification commands.
+ Name | Description
+ ---------------------------|--------------
+ service.name | The short name of the service object.
+ service.display_name | The value of the `display_name` attribute.
+ service.check_command | The short name of the command along with any arguments to be used for the check.
+ service.state | The service's current state. Can be one of `OK`, `WARNING`, `CRITICAL` and `UNKNOWN`.
+ service.state_id | The service's current state. Can be one of `0` (ok), `1` (warning), `2` (critical) and `3` (unknown).
+ service.state_type | The service's current state type. Can be one of `SOFT` and `HARD`.
+ service.check_attempt | The current check attempt number.
+ service.max_check_attempts | The maximum number of checks which are executed before changing to a hard state.
+ service.last_state | The service's previous state. Can be one of `OK`, `WARNING`, `CRITICAL` and `UNKNOWN`.
+ service.last_state_id | The service's previous state. Can be one of `0` (ok), `1` (warning), `2` (critical) and `3` (unknown).
+ service.last_state_type | The service's previous state type. Can be one of `SOFT` and `HARD`.
+ service.last_state_change | The last state change's timestamp.
+ service.downtime_depth | The number of active downtimes.
+ service.duration_sec | The time since the last state change.
+ service.latency | The service's check latency.
+ service.execution_time | The service's check execution time.
+ service.output | The last check's output.
+ service.perfdata | The last check's performance data.
+ service.last_check | The timestamp when the last check was executed.
+ service.check_source | The monitoring instance that performed the last check.
-You should choose which information you (and your notified users) are interested in
-case of emergency, and also which information does not provide any value to you and
-your environment.
+### <a id="command-runtime-macros"></a> Command Runtime Macros
-An example notification command is explained [here](#notification-commands).
+The following custom attributes are available in all commands:
-You can add all shared attributes to a `Notification` template which is inherited
-to the defined notifications. That way you'll save duplicated attributes in each
-`Notification` object. Attributes can be overridden locally.
+ Name | Description
+ -----------------------|--------------
+ command.name | The name of the command object.
- template Notification "generic-notification" {
- interval = 15m
+### <a id="user-runtime-macros"></a> User Runtime Macros
- command = "mail-service-notification"
+The following custom attributes are available in all commands that are executed for
+users:
- states = [ Warning, Critical, Unknown ]
- types = [ Problem, Acknowledgement, Recovery, Custom, FlappingStart,
- FlappingEnd, DowntimeStart, DowntimeEnd, DowntimeRemoved ]
+ Name | Description
+ -----------------------|--------------
+ user.name | The name of the user object.
+ user.display_name | The value of the display_name attribute.
- period = "24x7"
- }
+### <a id="notification-runtime-macros"></a> Notification Runtime Macros
-The time period `24x7` is shipped as example configuration with Icinga 2.
+ Name | Description
+ -----------------------|--------------
+ notification.type | The type of the notification.
+ notification.author | The author of the notification comment, if existing.
+ notification.comment | The comment of the notification, if existing.
-Use the `apply` keyword to create `Notification` objects for your services:
+### <a id="global-runtime-macros"></a> Global Runtime Macros
- apply Notification "mail" to Service {
- import "generic-notification"
+The following macros are available in all executed commands:
- command = "mail-notification"
- users = [ "icingaadmin" ]
+ Name | Description
+ -----------------------|--------------
+ icinga.timet | Current UNIX timestamp.
+ icinga.long_date_time | Current date and time including timezone information. Example: `2014-01-03 11:23:08 +0000`
+ icinga.short_date_time | Current date and time. Example: `2014-01-03 11:23:08`
+ icinga.date | Current date. Example: `2014-01-03`
+ icinga.time | Current time including timezone information. Example: `11:23:08 +0000`
+ icinga.uptime | Current uptime of the Icinga 2 process.
- assign where service.name == "mysql"
- }
+The following macros provide global statistics:
-Instead of assigning users to notifications, you can also add the `user_groups`
-attribute with a list of user groups to the `Notification` object. Icinga 2 will
-send notifications to all group members.
+ Name | Description
+ ----------------------------------|--------------
+ icinga.num_services_ok | Current number of services in state 'OK'.
+ icinga.num_services_warning | Current number of services in state 'Warning'.
+ icinga.num_services_critical | Current number of services in state 'Critical'.
+ icinga.num_services_unknown | Current number of services in state 'Unknown'.
+ icinga.num_services_pending | Current number of pending services.
+ icinga.num_services_unreachable | Current number of unreachable services.
+ icinga.num_services_flapping | Current number of flapping services.
+ icinga.num_services_in_downtime | Current number of services in downtime.
+ icinga.num_services_acknowledged | Current number of acknowledged service problems.
+ icinga.num_hosts_up | Current number of hosts in state 'Up'.
+ icinga.num_hosts_down | Current number of hosts in state 'Down'.
+ icinga.num_hosts_unreachable | Current number of unreachable hosts.
+ icinga.num_hosts_flapping | Current number of flapping hosts.
+ icinga.num_hosts_in_downtime | Current number of hosts in downtime.
+ icinga.num_hosts_acknowledged | Current number of acknowledged host problems.
-### <a id="notification-escalations"></a> Notification Escalations
-When a problem notification is sent and a problem still exists at the time of re-notification
-you may want to escalate the problem to the next support level. A different approach
-is to configure the default notification by email, and escalate the problem via SMS
-if not already solved.
-You can define notification start and end times as additional configuration
-attributes making the `Notification` object a so-called `notification escalation`.
-Using templates you can share the basic notification attributes such as users or the
-`interval` (and override them for the escalation then).
-Using the example from above, you can define additional users being escalated for SMS
-notifications between start and end time.
+## <a id="using-apply"></a> Apply Rules
- object User "icinga-oncall-2nd-level" {
- display_name = "Icinga 2nd Level"
+Instead of assigning each object ([Service](6-object-types.md#objecttype-service),
+[Notification](6-object-types.md#objecttype-notification), [Dependency](6-object-types.md#objecttype-dependency),
+[ScheduledDowntime](6-object-types.md#objecttype-scheduleddowntime))
+based on attribute identifiers for example `host_name` objects can be [applied](19-language-reference.md#apply).
- vars.mobile = "+1 555 424642"
- }
-
- object User "icinga-oncall-1st-level" {
- display_name = "Icinga 1st Level"
-
- vars.mobile = "+1 555 424642"
- }
+Before you start using the apply rules keep the following in mind:
-Define an additional `NotificationCommand` for SMS notifications.
+* Define the best match.
+ * A set of unique [custom attributes](#custom-attributes-apply) for these hosts/services?
+ * Or [group](3-monitoring-basics.md#groups) memberships, e.g. a host being a member of a hostgroup, applying services to it?
+ * A generic pattern [match](19-language-reference.md#function-calls) on the host/service name?
+ * [Multiple expressions combined](3-monitoring-basics.md#using-apply-expressions) with `&&` or `||` [operators](19-language-reference.md#expression-operators)
+* All expressions must return a boolean value (an empty string is equal to `false` e.g.)
> **Note**
>
-> The example is not complete as there are many different SMS providers.
-> Please note that sending SMS notifications will require an SMS provider
-> or local hardware with a SIM card active.
+> You can set/override object attributes in apply rules using the respectively available
+> objects in that scope (host and/or service objects).
- object NotificationCommand "sms-notification" {
- command = [
- PluginDir + "/send_sms_notification",
- "$mobile$",
- "..."
- }
+[Custom attributes](3-monitoring-basics.md#custom-attributes) can also store nested dictionaries and arrays. That way you can use them
+for not only matching for their existance or values in apply expressions, but also assign
+("inherit") their values into the generated objected from apply rules.
-The two new notification escalations are added onto the host `localhost`
-and its service `ping4` using the `generic-notification` template.
-The user `icinga-oncall-2nd-level` will get notified by SMS (`sms-notification`
-command) after `30m` until `1h`.
+* [Apply services to hosts](3-monitoring-basics.md#using-apply-services)
+* [Apply notifications to hosts and services](3-monitoring-basics.md#using-apply-notifications)
+* [Apply dependencies to hosts and services](3-monitoring-basics.md#using-apply-scheduledowntimes)
+* [Apply scheduled downtimes to hosts and services](3-monitoring-basics.md#using-apply-scheduledowntimes)
-> **Note**
->
-> The `interval` was set to 15m in the `generic-notification`
-> template example. Lower that value in your escalations by using a secondary
-> template or by overriding the attribute directly in the `notifications` array
-> position for `escalation-sms-2nd-level`.
+A more advanced example is using [apply with for loops on arrays or
+dictionaries](#using-apply-for) for example provided by
+[custom atttributes](#custom-attributes-apply) or groups.
-If the problem does not get resolved nor acknowledged preventing further notifications
-the `escalation-sms-1st-level` user will be escalated `1h` after the initial problem was
-notified, but only for one hour (`2h` as `end` key for the `times` dictionary).
+> **Tip**
+>
+> Building configuration in that dynamic way requires detailed information
+> of the generated objects. Use the `object list` [CLI command](8-cli-commands.md#cli-command-object)
+> after successful [configuration validation](8-cli-commands.md#config-validation).
- apply Notification "mail" to Service {
- import "generic-notification"
- command = "mail-notification"
- users = [ "icingaadmin" ]
+### <a id="using-apply-expressions"></a> Apply Rules Expressions
- assign where service.name == "ping4"
- }
+You can use simple or advanced combinations of apply rule expressions. Each
+expression must evaluate into the boolean `true` value. An empty string
+will be for instance interpreted as `false`. In a similar fashion undefined
+attributes will return `false`.
- apply Notification "escalation-sms-2nd-level" to Service {
- import "generic-notification"
+Returns `false`:
- command = "sms-notification"
- users = [ "icinga-oncall-2nd-level" ]
+ assign where host.vars.attribute_does_not_exist
- times = {
- begin = 30m
- end = 1h
- }
+Multiple `assign where` condition rows are evaluated as `OR` condition.
- assign where service.name == "ping4"
- }
+You can combine multiple expressions for matching only a subset of objects. In some cases,
+you want to be able to add more than one assign/ignore where expression which matches
+a specific condition. To achieve this you can use the logical `and` and `or` operators.
- apply Notification "escalation-sms-1st-level" to Service {
- import "generic-notification"
- command = "sms-notification"
- users = [ "icinga-oncall-1st-level" ]
+Match all `*mysql*` patterns in the host name and (`&&`) custom attribute `prod_mysql_db`
+matches the `db-*` pattern. All hosts with the custom attribute `test_server` set to `true`
+should be ignored, or any host name ending with `*internal` pattern.
- times = {
- begin = 1h
- end = 2h
- }
+ object HostGroup "mysql-server" {
+ display_name = "MySQL Server"
- assign where service.name == "ping4"
+ assign where match("*mysql*", host.name) && match("db-*", host.vars.prod_mysql_db)
+ ignore where host.vars.test_server == true
+ ignore where match("*internal", host.name)
}
-### <a id="notification-delay"></a> Notification Delay
-
-Sometimes the problem in question should not be notified when the notification is due
-(the object reaching the `HARD` state) but a defined time duration afterwards. In Icinga 2
-you can use the `times` dictionary and set `begin = 15m` as key and value if you want to
-postpone the first notification for 15 minutes. Leave out the `end` key - if not set,
-Icinga 2 will not check against any end time for this notification.
+Similar example for advanced notification apply rule filters: If the service
+attribute `notes` contains the `has gold support 24x7` string `AND` one of the
+two condition passes: Either the `customer` host custom attribute is set to `customer-xy`
+`OR` the host custom attribute `always_notify` is set to `true`.
- apply Notification "mail" to Service {
- import "generic-notification"
+The notification is ignored for services whose host name ends with `*internal`
+`OR` the `priority` custom attribute is [less than](19-language-reference.md#expression-operators) `2`.
- command = "mail-notification"
- users = [ "icingaadmin" ]
+ template Notification "cust-xy-notification" {
+ users = [ "noc-xy", "mgmt-xy" ]
+ command = "mail-service-notification"
+ }
- times.begin = 15m // delay first notification
+ apply Notification "notify-cust-xy-mysql" to Service {
+ import "cust-xy-notification"
- assign where service.name == "ping4"
+ assign where match("*has gold support 24x7*", service.notes) && (host.vars.customer == "customer-xy" || host.vars.always_notify == true
+ ignore where match("*internal", host.name) || (service.vars.priority < 2 && host.vars.is_clustered == true)
}
-### <a id="notification-filters-state-type"></a> Notification Filters by State and Type
-
-If there are no notification state and type filter attributes defined at the `Notification`
-or `User` object Icinga 2 assumes that all states and types are being notified.
-Available state and type filters for notifications are:
- template Notification "generic-notification" {
- states = [ Warning, Critical, Unknown ]
- types = [ Problem, Acknowledgement, Recovery, Custom, FlappingStart,
- FlappingEnd, DowntimeStart, DowntimeEnd, DowntimeRemoved ]
- }
+### <a id="using-apply-services"></a> Apply Services to Hosts
-If you are familiar with Icinga 1.x `notification_options` please note that they have been split
-into type and state to allow more fine granular filtering for example on downtimes and flapping.
-You can filter for acknowledgements and custom notifications too.
+The sample configuration already includes a detailed example in [hosts.conf](5-configuring-icinga-2.md#hosts-conf)
+and [services.conf](5-configuring-icinga-2.md#services-conf) for this use case.
+The example for `ssh` applies a service object to all hosts with the `address`
+attribute being defined and the custom attribute `os` set to the string `Linux` in `vars`.
-## <a id="timeperiods"></a> Time Periods
+ apply Service "ssh" {
+ import "generic-service"
-Time Periods define time ranges in Icinga where event actions are
-triggered, for example whether a service check is executed or not within
-the `check_period` attribute. Or a notification should be sent to
-users or not, filtered by the `period` and `notification_period`
-configuration attributes for `Notification` and `User` objects.
+ check_command = "ssh"
-> **Note**
->
-> If you are familar with Icinga 1.x - these time period definitions
-> are called `legacy timeperiods` in Icinga 2.
->
-> An Icinga 2 legacy timeperiod requires the `ITL` provided template
->`legacy-timeperiod`.
-
-The `TimePeriod` attribute `ranges` may contain multiple directives,
-including weekdays, days of the month, and calendar dates.
-These types may overlap/override other types in your ranges dictionary.
-
-The descending order of precedence is as follows:
-
-* Calendar date (2008-01-01)
-* Specific month date (January 1st)
-* Generic month date (Day 15)
-* Offset weekday of specific month (2nd Tuesday in December)
-* Offset weekday (3rd Monday)
-* Normal weekday (Tuesday)
-
-If you don't set any `check_period` or `notification_period` attribute
-on your configuration objects Icinga 2 assumes `24x7` as time period
-as shown below.
-
- object TimePeriod "24x7" {
- import "legacy-timeperiod"
-
- display_name = "Icinga 2 24x7 TimePeriod"
- ranges = {
- "monday" = "00:00-24:00"
- "tuesday" = "00:00-24:00"
- "wednesday" = "00:00-24:00"
- "thursday" = "00:00-24:00"
- "friday" = "00:00-24:00"
- "saturday" = "00:00-24:00"
- "sunday" = "00:00-24:00"
- }
+ assign where host.address && host.vars.os == "Linux"
}
-If your operation staff should only be notified during workhours
-create a new timeperiod named `workhours` defining a work day from
-09:00 to 17:00.
- object TimePeriod "workhours" {
- import "legacy-timeperiod"
+Other detailed scenario examples are used in their respective chapters, for example
+[apply services with custom command arguments](#using-apply-services-command-arguments).
- display_name = "Icinga 2 8x5 TimePeriod"
- ranges = {
- "monday" = "09:00-17:00"
- "tuesday" = "09:00-17:00"
- "wednesday" = "09:00-17:00"
- "thursday" = "09:00-17:00"
- "friday" = "09:00-17:00"
- }
- }
+### <a id="using-apply-notifications"></a> Apply Notifications to Hosts and Services
-Use the `period` attribute to assign time periods to
-`Notification` and `Dependency` objects:
+Notifications are applied to specific targets (`Host` or `Service`) and work in a similar
+manner:
- object Notification "mail" {
- import "generic-notification"
- host_name = "localhost"
+ apply Notification "mail-noc" to Service {
+ import "mail-service-notification"
- command = "mail-notification"
- users = [ "icingaadmin" ]
- period = "workhours"
+ user_groups = [ "noc" ]
+
+ assign where host.vars.notification.mail
}
-## <a id="commands"></a> Commands
+In this example the `mail-noc` notification will be created as object for all services having the
+`notification.mail` custom attribute defined. The notification command is set to `mail-service-notification`
+and all members of the user group `noc` will get notified.
-Icinga 2 uses three different command object types to specify how
-checks should be performed, notifications should be sent, and
-events should be handled.
+### <a id="using-apply-dependencies"></a> Apply Dependencies to Hosts and Services
-### <a id="command-environment-variables"></a> Environment Variables for Commands
+Detailed examples can be found in the [dependencies](3-monitoring-basics.md#dependencies) chapter.
-Please check [Runtime Custom Attributes as Environment Variables](#runtime-custom-attribute-env-vars).
+### <a id="using-apply-scheduledowntimes"></a> Apply Recurring Downtimes to Hosts and Services
+The sample confituration includes an example in [downtimes.conf](5-configuring-icinga-2.md#downtimes-conf).
-### <a id="check-commands"></a> Check Commands
+Detailed examples can be found in the [recurring downtimes](4-advanced-topics.md#recurring-downtimes) chapter.
-`CheckCommand` objects define the command line how a check is called.
-> **Note**
->
-> Make sure that the [checker](#features) feature is enabled in order to
-> execute checks.
+### <a id="using-apply-for"></a> Using Apply For Rules
-#### <a id="command-plugin-integration"></a> Integrate the Plugin with a CheckCommand Definition
+Next to the standard way of using apply rules there is the requirement of generating
+apply rules objects based on set (array or dictionary). That way you'll save quite
+of a lot of duplicated apply rules by combining them into one generic generating
+the object name with or without a prefix.
-`CheckCommand` objects require the [ITL template](#itl-plugin-check-command)
-`plugin-check-command` to support native plugin based check methods.
+The sample configuration already includes a detailed example in [hosts.conf](5-configuring-icinga-2.md#hosts-conf)
+and [services.conf](5-configuring-icinga-2.md#services-conf) for this use case.
-Unless you have done so already, download your check plugin and put it
-into the `PluginDir` directory. The following example uses the
-`check_disk` plugin shipped with the Monitoring Plugins package.
+Imagine a different example: You are monitoring your switch (hosts) with many
+interfaces (services). The following requirements/problems apply:
-The plugin path and all command arguments are made a list of
-double-quoted string arguments for proper shell escaping.
+* Each interface service check should be named with a prefix and a running number
+* Each interface has its own vlan tag
+* Some interfaces have QoS enabled
+* Additional attributes such as `display_name` or `notes, `notes_url` and `action_url` must be
+dynamically generated
-Call the `check_disk` plugin with the `--help` parameter to see
-all available options. Our example defines warning (`-w`) and
-critical (`-c`) thresholds for the disk usage. Without any
-partition defined (`-p`) it will check all local partitions.
+By defining the `interfaces` dictionary with three example interfaces on the `core-switch`
+host object, you'll make sure to pass the storage required by the for loop in the service apply
+rule.
- icinga@icinga2 $ /usr/lib/nagios/plugins/check_disk --help
- ...
- This plugin checks the amount of used disk space on a mounted file system
- and generates an alert if free space is less than one of the threshold values
+ object Host "core-switch" {
+ import "generic-host"
+ address = "127.0.0.1"
- Usage:
- check_disk -w limit -c limit [-W limit] [-K limit] {-p path | -x device}
- [-C] [-E] [-e] [-f] [-g group ] [-k] [-l] [-M] [-m] [-R path ] [-r path ]
- [-t timeout] [-u unit] [-v] [-X type] [-N type]
- ...
+ vars.interfaces["0"] = {
+ port = 1
+ vlan = "internal"
+ address = "127.0.0.2"
+ qos = "enabled"
+ }
+ vars.interfaces["1"] = {
+ port = 2
+ vlan = "mgmt"
+ address = "127.0.1.2"
+ }
+ vars.interfaces["2"] = {
+ port = 3
+ vlan = "remote"
+ address = "127.0.2.2"
+ }
+ }
-> **Note**
->
-> Don't execute plugins as `root` and always use the absolute path to the plugin! Trust us.
+You can also omit the `"if-"` string, then all generated service names are directly
+taken from the `if_name` variable value.
-Next step is to understand how command parameters are being passed from
-a host or service object, and add a `CheckCommand` definition based on these
-required parameters and/or default values.
+The config dictionary contains all key-value pairs for the specific interface in one
+loop cycle, like `port`, `vlan`, `address` and `qos` for the `0` interface.
-#### <a id="command-passing-parameters"></a> Passing Check Command Parameters from Host or Service
+By defining a default value for the custom attribute `qos` in the `vars` dictionary
+before adding the `config` dictionary we''ll ensure that this attribute is always defined.
-Unlike Icinga 1.x check command parameters are defined as custom attributes
-which can be accessed as runtime macros by the executed check command.
+After `vars` is fully populated, all object attributes can be set. For strings, you can use
+string concatention with the `+` operator.
-Define the default check command custom attribute `disk_wfree` and `disk_cfree`
-(freely definable naming schema) and their default threshold values. You can
-then use these custom attributes as runtime macros for [command arguments](#command-arguments)
-on the command line.
+You can also specifiy the check command that way.
-The default custom attributes can be overridden by the custom attributes
-defined in the service using the check command `my-disk`. The custom attributes
-can also be inherited from a parent template using additive inheritance (`+=`).
+ apply Service "if-" for (if_name => config in host.vars.interfaces) {
+ import "generic-service"
+ check_command = "ping4"
+ vars.qos = "disabled"
+ vars += config
- object CheckCommand "my-disk" {
- import "plugin-check-command"
+ display_name = "if-" + if_name + "-" + vars.vlan
- command = [ PluginDir + "/check_disk" ]
+ notes = "Interface check for Port " + string(vars.port) + " in VLAN " + vars.vlan + " on Address " + vars.address + " QoS " + vars.qos
+ notes_url = "http://foreman.company.com/hosts/" + host.name
+ action_url = "http://snmp.checker.company.com/" + host.name + "if-" + if_name
+ }
- arguments = {
- "-w" = "$disk_wfree$%"
- "-c" = "$disk_cfree$%"
- }
+Note that numbers must be explicitely casted to string when adding to strings.
+This can be achieved by wrapping them into the [string()](19-language-reference.md#function-calls) function.
- vars.disk_wfree = 20
- vars.disk_cfree = 10
- }
+> **Tip**
+>
+> Building configuration in that dynamic way requires detailed information
+> of the generated objects. Use the `object list` [CLI command](8-cli-commands.md#cli-command-object)
+> after successful [configuration validation](8-cli-commands.md#config-validation).
-The host `localhost` with the service `my-disk` checks all disks with modified
-custom attributes (warning thresholds at `10%`, critical thresholds at `5%`
-free disk space).
+### <a id="using-apply-object attributes"></a> Use Object Attributes in Apply Rules
- object Host "localhost" {
+Since apply rules are evaluated after the generic objects, you
+can reference existing host and/or service object attributes as
+values for any object attribute specified in that apply rule.
+
+ object Host "opennebula-host" {
import "generic-host"
+ address = "10.1.1.2"
- address = "127.0.0.1"
- address6 = "::1"
+ vars.hosting["xyz"] = {
+ http_uri = "/shop"
+ customer_name = "Customer xyz"
+ customer_id = "7568"
+ support_contract = "gold"
+ }
+ vars.hosting["abc"] = {
+ http_uri = "/shop"
+ customer_name = "Customer xyz"
+ customer_id = "7568"
+ support_contract = "silver"
+ }
}
- object Service "my-disk" {
+ apply Service for (customer => config in host.vars.hosting) {
import "generic-service"
+ check_command = "ping4"
- host_name = "localhost"
- check_command = "my-disk"
-
- vars.disk_wfree = 10
- vars.disk_cfree = 5
- }
-
-#### <a id="command-arguments"></a> Command Arguments
+ vars.qos = "disabled"
-By defining a check command line using the `command` attribute Icinga 2
-will resolve all macros in the static string or array. Sometimes it is
-required to extend the arguments list based on a met condition evaluated
-at command execution. Or making arguments optional - only set if the
-macro value can be resolved by Icinga 2.
+ vars += config
- object CheckCommand "check_http" {
- import "plugin-check-command"
+ vars.http_uri = "/" + vars.customer + "/" + config.http_uri
- command = [ PluginDir + "/check_http" ]
+ display_name = "Shop Check for " + vars.customer_name + "-" + vars.customer_id
- arguments = {
- "-H" = "$http_vhost$"
- "-I" = "$http_address$"
- "-u" = "$http_uri$"
- "-p" = "$http_port$"
- "-S" = {
- set_if = "$http_ssl$"
- }
- "--sni" = {
- set_if = "$http_sni$"
- }
- "-a" = {
- value = "$http_auth_pair$"
- description = "Username:password on sites with basic authentication"
- }
- "--no-body" = {
- set_if = "$http_ignore_body$"
- }
- "-r" = "$http_expect_body_regex$"
- "-w" = "$http_warn_time$"
- "-c" = "$http_critical_time$"
- "-e" = "$http_expect$"
- }
+ notes = "Support contract: " + vars.support_contract + " for Customer " + vars.customer_name + " (" + vars.customer_id + ")."
- vars.http_address = "$address$"
- vars.http_ssl = false
- vars.http_sni = false
+ notes_url = "http://foreman.company.com/hosts/" + host.name
+ action_url = "http://snmp.checker.company.com/" + host.name + "/" + vars.customer_id
}
-The example shows the `check_http` check command defining the most common
-arguments. Each of them is optional by default and will be omitted if
-the value is not set. For example if the service calling the check command
-does not have `vars.http_port` set, it won't get added to the command
-line.
-If the `vars.http_ssl` custom attribute is set in the service, host or command
-object definition, Icinga 2 will add the `-S` argument based on the `set_if`
-option to the command line.
-That way you can use the `check_http` command definition for both, with and
-without SSL enabled checks saving you duplicated command definitions.
-
-Details on all available options can be found in the
-[CheckCommand object definition](#objecttype-checkcommand).
+## <a id="groups"></a> Groups
-### <a id="using-apply-services-command-arguments"></a> Apply Services with custom Command Arguments
+A group is a collection of similar objects. Groups are primarily used as a
+visualization aid in web interfaces.
-Imagine the following scenario: The `my-host1` host is reachable using the default port 22, while
-the `my-host2` host requires a different port on 2222. Both hosts are in the hostgroup `my-linux-servers`.
+Group membership is defined at the respective object itself. If
+you have a hostgroup name `windows` for example, and want to assign
+specific hosts to this group for later viewing the group on your
+alert dashboard, first create a HostGroup object:
- object HostGroup "my-linux-servers" {
- display_name = "Linux Servers"
- assign where host.vars.os == "Linux"
+ object HostGroup "windows" {
+ display_name = "Windows Servers"
}
- /* this one has port 22 opened */
- object Host "my-host1" {
- import "generic-host"
- address = "129.168.1.50"
- vars.os = "Linux"
- }
+Then add your hosts to this group:
- /* this one listens on a different ssh port */
- object Host "my-host2" {
- import "generic-host"
- address = "129.168.2.50"
- vars.os = "Linux"
- vars.custom_ssh_port = 2222
+ template Host "windows-server" {
+ groups += [ "windows" ]
}
-All hosts in the `my-linux-servers` hostgroup should get the `my-ssh` service applied based on an
-[apply rule](#apply). The optional `ssh_port` command argument should be inherited from the host
-the service is applied to. If not set, the check command `my-ssh` will omit the argument.
-
- object CheckCommand "my-ssh" {
- import "plugin-check-command"
+ object Host "mssql-srv1" {
+ import "windows-server"
- command = [ PluginDir + "/check_ssh" ]
+ vars.mssql_port = 1433
+ }
- arguments = {
- "-p" = "$ssh_port$"
- "host" = {
- value = "$ssh_address$"
- skip_key = true
- order = -1
- }
- }
+ object Host "mssql-srv2" {
+ import "windows-server"
- vars.ssh_address = "$address$"
+ vars.mssql_port = 1433
}
- /* apply ssh service */
- apply Service "my-ssh" {
- import "generic-service"
- check_command = "my-ssh"
-
- //set the command argument for ssh port with a custom host attribute, if set
- vars.ssh_port = "$host.vars.custom_ssh_port$"
+This can be done for service and user groups the same way:
- assign where "my-linux-servers" in host.groups
+ object UserGroup "windows-mssql-admins" {
+ display_name = "Windows MSSQL Admins"
}
-The `my-host1` will get the `my-ssh` service checking on the default port:
+ template User "generic-windows-mssql-users" {
+ groups += [ "windows-mssql-admins" ]
+ }
- [2014-05-26 21:52:23 +0200] notice/Process: Running command '/usr/lib/nagios/plugins/check_ssh', '129.168.1.50': PID 27281
+ object User "win-mssql-noc" {
+ import "generic-windows-mssql-users"
-The `my-host2` will inherit the `custom_ssh_port` variable to the service and execute a different command:
+ email = "noc@example.com"
+ }
- [2014-05-26 21:51:32 +0200] notice/Process: Running command '/usr/lib/nagios/plugins/check_ssh', '-p', '2222', '129.168.2.50': PID 26956
+ object User "win-mssql-ops" {
+ import "generic-windows-mssql-users"
-
-### <a id="notification-commands"></a> Notification Commands
-
-`NotificationCommand` objects define how notifications are delivered to external
-interfaces (E-Mail, XMPP, IRC, Twitter, etc).
-
-`NotificationCommand` objects require the [ITL template](#itl-plugin-notification-command)
-`plugin-notification-command` to support native plugin-based notifications.
-
-> **Note**
->
-> Make sure that the [notification](#features) feature is enabled on your master instance
-> in order to execute notification commands.
-
-Below is an example using runtime macros from Icinga 2 (such as `$service.output$` for
-the current check output) sending an email to the user(s) associated with the
-notification itself (`$user.email$`).
-
-If you want to specify default values for some of the custom attribute definitions,
-you can add a `vars` dictionary as shown for the `CheckCommand` object.
-
- object NotificationCommand "mail-service-notification" {
- import "plugin-notification-command"
-
- command = [ SysconfDir + "/icinga2/scripts/mail-notification.sh" ]
-
- env = {
- NOTIFICATIONTYPE = "$notification.type$"
- SERVICEDESC = "$service.name$"
- HOSTALIAS = "$host.display_name$"
- HOSTADDRESS = "$address$"
- SERVICESTATE = "$service.state$"
- LONGDATETIME = "$icinga.long_date_time$"
- SERVICEOUTPUT = "$service.output$"
- NOTIFICATIONAUTHORNAME = "$notification.author$"
- NOTIFICATIONCOMMENT = "$notification.comment$"
- HOSTDISPLAYNAME = "$host.display_name$"
- SERVICEDISPLAYNAME = "$service.display_name$"
- USEREMAIL = "$user.email$"
- }
- }
-
-The command attribute in the `mail-service-notification` command refers to the following
-shell script. The macros specified in the `env` array are exported
-as environment variables and can be used in the notification script:
-
- #!/usr/bin/env bash
- template=$(cat <<TEMPLATE
- ***** Icinga *****
-
- Notification Type: $NOTIFICATIONTYPE
-
- Service: $SERVICEDESC
- Host: $HOSTALIAS
- Address: $HOSTADDRESS
- State: $SERVICESTATE
-
- Date/Time: $LONGDATETIME
-
- Additional Info: $SERVICEOUTPUT
-
- Comment: [$NOTIFICATIONAUTHORNAME] $NOTIFICATIONCOMMENT
- TEMPLATE
- )
-
- /usr/bin/printf "%b" $template | mail -s "$NOTIFICATIONTYPE - $HOSTDISPLAYNAME - $SERVICEDISPLAYNAME is $SERVICESTATE" $USEREMAIL
-
-> **Note**
->
-> This example is for `exim` only. Requires changes for `sendmail` and
-> other MTAs.
-
-While it's possible to specify the entire notification command right
-in the NotificationCommand object it is generally advisable to create a
-shell script in the `/etc/icinga2/scripts` directory and have the
-NotificationCommand object refer to that.
-
-### <a id="event-commands"></a> Event Commands
-
-Unlike notifications event commands are called on every host/service execution
-if one of these conditions match:
-
-* The host/service is in a [soft state](#hard-soft-states)
-* The host/service state changes into a [hard state](#hard-soft-states)
-* The host/service state recovers from a [soft or hard state](#hard-soft-states) to [OK](#service-states)/[Up](#host-states)
-
-Therefore the `EventCommand` object should define a command line
-evaluating the current service state and other service runtime attributes
-available through runtime vars. Runtime macros such as `$service.state_type$`
-and `$service.state$` will be processed by Icinga 2 helping on fine-granular
-events being triggered.
-
-Common use case scenarios are a failing HTTP check requiring an immediate
-restart via event command, or if an application is locked and requires
-a restart upon detection.
-
-`EventCommand` objects require the ITL template `plugin-event-command`
-to support native plugin based checks.
-
-When the event command is triggered on a service state change, it will
-send a check result using the `process_check_result` script forcibly
-changing the service state back to `OK` (`-r 0`) providing some debug
-information in the check output (`-o`).
-
- object EventCommand "plugin-event-process-check-result" {
- import "plugin-event-command"
-
- command = [
- PluginDir + "/process_check_result",
- "-H", "$host.name$",
- "-S", "$service.name$",
- "-c", RunDir + "/icinga2/cmd/icinga2.cmd",
- "-r", "0",
- "-o", "Event Handler triggered in state '$service.state$' with output '$service.output$'."
- ]
- }
-
-
-## <a id="dependencies"></a> Dependencies
-
-Icinga 2 uses host and service [Dependency](#objecttype-dependency) objects
-for determing their network reachability.
-The `parent_host_name` and `parent_service_name` attributes are mandatory for
-service dependencies, `parent_host_name` is required for host dependencies.
-
-A service can depend on a host, and vice versa. A service has an implicit
-dependency (parent) to its host. A host to host dependency acts implicitly
-as host parent relation.
-When dependencies are calculated, not only the immediate parent is taken into
-account but all parents are inherited.
-
-Notifications are suppressed if a host or service becomes unreachable.
-
-### <a id="dependencies-implicit-host-service"></a> Implicit Dependencies for Services on Host
-
-Icinga 2 automatically adds an implicit dependency for services on their host. That way
-service notifications are suppressed when a host is `DOWN` or `UNREACHABLE`. This dependency
-does not overwrite other dependencies and implicitely sets `disable_notifications = true` and
-`states = [ Up ]` for all service objects.
-
-Service checks are still executed. If you want to prevent them from happening, you can
-apply the following dependency to all services setting their host as `parent_host_name`
-and disabling the checks. `assign where true` matches on all `Service` objects.
-
- apply Dependency "disable-host-service-checks" to Service {
- disable_checks = true
- assign where true
- }
-
-### <a id="dependencies-network-reachability"></a> Dependencies for Network Reachability
-
-A common scenario is the Icinga 2 server behind a router. Checking internet
-access by pinging the Google DNS server `google-dns` is a common method, but
-will fail in case the `dsl-router` host is down. Therefore the example below
-defines a host dependency which acts implicitly as parent relation too.
-
-Furthermore the host may be reachable but ping probes are dropped by the
-router's firewall. In case the `dsl-router``ping4` service check fails, all
-further checks for the `ping4` service on host `google-dns` service should
-be suppressed. This is achieved by setting the `disable_checks` attribute to `true`.
-
- object Host "dsl-router" {
- address = "192.168.1.1"
- }
-
- object Host "google-dns" {
- address = "8.8.8.8"
- }
-
- apply Service "ping4" {
- import "generic-service"
-
- check_command = "ping4"
-
- assign where host.address
- }
-
- apply Dependency "internet" to Host {
- parent_host_name = "dsl-router"
- disable_checks = true
- disable_notifications = true
-
- assign where host.name != "dsl-router"
- }
-
- apply Dependency "internet" to Service {
- parent_host_name = "dsl-router"
- parent_service_name = "ping4"
- disable_checks = true
-
- assign where host.name != "dsl-router"
- }
-
-
-### <a id="dependencies-agent-checks"></a> Dependencies for Agent Checks
-
-Another classic example are agent based checks. You would define a health check
-for the agent daemon responding to your requests, and make all other services
-querying that daemon depend on that health check.
-
-The following configuration defines two nrpe based service checks `nrpe-load`
-and `nrpe-disk` applied to the `nrpe-server`. The health check is defined as
-`nrpe-health` service.
-
- apply Service "nrpe-health" {
- import "generic-service"
- check_command = "nrpe"
- assign where match("nrpe-*", host.name)
- }
-
- apply Service "nrpe-load" {
- import "generic-service"
- check_command = "nrpe"
- vars.nrpe_command = "check_load"
- assign where match("nrpe-*", host.name)
+ email = "ops@example.com"
}
- apply Service "nrpe-disk" {
- import "generic-service"
- check_command = "nrpe"
- vars.nrpe_command = "check_disk"
- assign where match("nrpe-*", host.name)
- }
+### <a id="group-assign-intro"></a> Group Membership Assign
- object Host "nrpe-server" {
- import "generic-host"
- address = "192.168.1.5"
- }
+Instead of manually assigning each object to a group you can also assign objects
+to a group based on their attributes:
- apply Dependency "disable-nrpe-checks" to Service {
- parent_service_name = "nrpe-health"
+ object HostGroup "prod-mssql" {
+ display_name = "Production MSSQL Servers"
- states = [ OK ]
- disable_checks = true
- disable_notifications = true
- assign where service.check_command == "nrpe"
- ignore where service.name == "nrpe-health"
+ assign where host.vars.mssql_port && host.vars.prod_mysql_db
+ ignore where host.vars.test_server == true
+ ignore where match("*internal", host.name)
}
-The `disable-nrpe-checks` dependency is applied to all services
-on the `nrpe-service` host using the `nrpe` check_command attribute
-but not the `nrpe-health` service itself.
-
+In this example all hosts with the `vars` attribute `mssql_port`
+will be added as members to the host group `mssql`. However, all `*internal`
+hosts or with the `test_server` attribute set to `true` are not added to this
+group.
-## <a id="downtimes"></a> Downtimes
+Details on the `assign where` syntax can be found in the
+[Language Reference](19-language-reference.md#apply)
-Downtimes can be scheduled for planned server maintenance or
-any other targetted service outage you are aware of in advance.
-
-Downtimes will suppress any notifications, and may trigger other
-downtimes too. If the downtime was set by accident, or the duration
-exceeds the maintenance, you can manually cancel the downtime.
-Planned downtimes will also be taken into account for SLA reporting
-tools calculating the SLAs based on the state and downtime history.
-
-Multiple downtimes for a single object may overlap. This is useful
-when you want to extend your maintenance window taking longer than expected.
-If there are multiple downtimes triggered for one object, the overall downtime depth
-will be greater than `1`.
-
-
-If the downtime was scheduled after the problem changed to a critical hard
-state triggering a problem notification, and the service recovers during
-the downtime window, the recovery notification won't be suppressed.
-
-### <a id="fixed-flexible-downtimes"></a> Fixed and Flexible Downtimes
-
-A `fixed` downtime will be activated at the defined start time, and
-removed at the end time. During this time window the service state
-will change to `NOT-OK` and then actually trigger the downtime.
-Notifications are suppressed and the downtime depth is incremented.
-
-Common scenarios are a planned distribution upgrade on your linux
-servers, or database updates in your warehouse. The customer knows
-about a fixed downtime window between 23:00 and 24:00. After 24:00
-all problems should be alerted again. Solution is simple -
-schedule a `fixed` downtime starting at 23:00 and ending at 24:00.
-
-Unlike a `fixed` downtime, a `flexible` downtime will be triggered
-by the state change in the time span defined by start and end time,
-and then last for the specified duration in minutes.
-
-Imagine the following scenario: Your service is frequently polled
-by users trying to grab free deleted domains for immediate registration.
-Between 07:30 and 08:00 the impact will hit for 15 minutes and generate
-a network outage visible to the monitoring. The service is still alive,
-but answering too slow to Icinga 2 service checks.
-For that reason, you may want to schedule a downtime between 07:30 and
-08:00 with a duration of 15 minutes. The downtime will then last from
-its trigger time until the duration is over. After that, the downtime
-is removed (may happen before or after the actual end time!).
-
-### <a id="scheduling-downtime"></a> Scheduling a downtime
-
-This can either happen through a web interface or by sending an [external command](#external-commands)
-to the external command pipe provided by the `ExternalCommandListener` configuration.
-
-Fixed downtimes require a start and end time (a duration will be ignored).
-Flexible downtimes need a start and end time for the time span, and a duration
-independent from that time span.
-
-### <a id="triggered-downtimes"></a> Triggered Downtimes
-
-This is optional when scheduling a downtime. If there is already a downtime
-scheduled for a future maintenance, the current downtime can be triggered by
-that downtime. This renders useful if you have scheduled a host downtime and
-are now scheduling a child host's downtime getting triggered by the parent
-downtime on NOT-OK state change.
+## <a id="notifications"></a> Notifications
-### <a id="recurring-downtimes"></a> Recurring Downtimes
+Notifications for service and host problems are an integral part of your
+monitoring setup.
-[ScheduledDowntime objects](#objecttype-scheduleddowntime) can be used to set up
-recurring downtimes for services.
+When a host or service is in a downtime, a problem has been acknowledged or
+the dependency logic determined that the host/service is unreachable, no
+notifications are sent. You can configure additional type and state filters
+refining the notifications being actually sent.
-Example:
+There are many ways of sending notifications, e.g. by e-mail, XMPP,
+IRC, Twitter, etc. On its own Icinga 2 does not know how to send notifications.
+Instead it relies on external mechanisms such as shell scripts to notify users.
- apply ScheduledDowntime "backup-downtime" to Service {
- author = "icingaadmin"
- comment = "Scheduled downtime for backup"
+A notification specification requires one or more users (and/or user groups)
+who will be notified in case of problems. These users must have all custom
+attributes defined which will be used in the `NotificationCommand` on execution.
- ranges = {
- monday = "02:00-03:00"
- tuesday = "02:00-03:00"
- wednesday = "02:00-03:00"
- thursday = "02:00-03:00"
- friday = "02:00-03:00"
- saturday = "02:00-03:00"
- sunday = "02:00-03:00"
- }
+The user `icingaadmin` in the example below will get notified only on `WARNING` and
+`CRITICAL` states and `problem` and `recovery` notification types.
- assign where "backup" in service.groups
+ object User "icingaadmin" {
+ display_name = "Icinga 2 Admin"
+ enable_notifications = true
+ states = [ OK, Warning, Critical ]
+ types = [ Problem, Recovery ]
+ email = "icinga@localhost"
}
+If you don't set the `states` and `types` configuration attributes for the `User`
+object, notifications for all states and types will be sent.
-## <a id="comments"></a> Comments
-
-Comments can be added at runtime and are persistent over restarts. You can
-add useful information for others on repeating incidents (for example
-"last time syslog at 100% cpu on 17.10.2013 due to stale nfs mount") which
-is primarly accessible using web interfaces.
-
-Adding and deleting comment actions are possible through the external command pipe
-provided with the `ExternalCommandListener` configuration. The caller must
-pass the comment id in case of manipulating an existing comment.
-
-
-## <a id="acknowledgements"></a> Acknowledgements
-
-If a problem is alerted and notified you may signal the other notification
-recipients that you are aware of the problem and will handle it.
-
-By sending an acknowledgement to Icinga 2 (using the external command pipe
-provided with `ExternalCommandListener` configuration) all future notifications
-are suppressed, a new comment is added with the provided description and
-a notification with the type `NotificationFilterAcknowledgement` is sent
-to all notified users.
-
-### <a id="expiring-acknowledgements"></a> Expiring Acknowledgements
-
-Once a problem is acknowledged it may disappear from your `handled problems`
-dashboard and no-one ever looks at it again since it will suppress
-notifications too.
-
-This `fire-and-forget` action is quite common. If you're sure that a
-current problem should be resolved in the future at a defined time,
-you can define an expiration time when acknowledging the problem.
-
-Icinga 2 will clear the acknowledgement when expired and start to
-re-notify if the problem persists.
-
-
-
-## <a id="custom-attributes"></a> Custom Attributes
-
-### <a id="runtime-custom-attributes"></a> Using Custom Attributes at Runtime
-
-Custom attributes may be used in command definitions to dynamically change how the command
-is executed.
-
-Additionally there are Icinga 2 features such as the `PerfDataWriter` type
-which use custom attributes to format their output.
+Details on troubleshooting notification problems can be found [here](16-troubleshooting.md#troubleshooting).
-> **Tip**
+> **Note**
>
-> Custom attributes are identified by the 'vars' dictionary attribute as short name.
-> Accessing the different attribute keys is possible using the '.' accessor.
+> Make sure that the [notification](8-cli-commands.md#features) feature is enabled
+> in order to execute notification commands.
-Custom attributes in command definitions or performance data templates are evaluated at
-runtime when executing a command. These custom attributes cannot be used elsewhere
-(e.g. in other configuration attributes).
+You should choose which information you (and your notified users) are interested in
+case of emergency, and also which information does not provide any value to you and
+your environment.
-Custom attribute values must be either a string, a number or a boolean value. Arrays
-and dictionaries cannot be used.
+An example notification command is explained [here](3-monitoring-basics.md#notification-commands).
-Here is an example of a command definition which uses user-defined custom attributes:
+You can add all shared attributes to a `Notification` template which is inherited
+to the defined notifications. That way you'll save duplicated attributes in each
+`Notification` object. Attributes can be overridden locally.
- object CheckCommand "my-ping" {
- import "plugin-check-command"
+ template Notification "generic-notification" {
+ interval = 15m
- command = [
- PluginDir + "/check_ping", "-4"
- ]
+ command = "mail-service-notification"
- arguments = {
- "-H" = "$ping_address$"
- "-w" = "$ping_wrta$,$ping_wpl$%"
- "-c" = "$ping_crta$,$ping_cpl$%"
- "-p" = "$ping_packets$"
- "-t" = "$ping_timeout$"
- }
+ states = [ Warning, Critical, Unknown ]
+ types = [ Problem, Acknowledgement, Recovery, Custom, FlappingStart,
+ FlappingEnd, DowntimeStart, DowntimeEnd, DowntimeRemoved ]
- vars.ping_address = "$address$"
- vars.ping_wrta = 100
- vars.ping_wpl = 5
- vars.ping_crta = 200
- vars.ping_cpl = 15
- vars.ping_packets = 5
- vars.ping_timeout = 0
+ period = "24x7"
}
-Custom attribute names used at runtime must be enclosed in two `$` signs, e.g.
-`$address$`. When using the `$` sign as single character, you need to escape
-it with an additional dollar sign (`$$`). This example also makes use of the
-[command arguments](#command-arguments) passed to the command line. `-4` must
-be added as additional array key.
-
-### <a id="runtime-custom-attributes-evaluation-order"></a> Runtime Custom Attributes Evaluation Order
+The time period `24x7` is included as example configuration with Icinga 2.
-When executing commands Icinga 2 checks the following objects in this order to look
-up custom attributes and their respective values:
-
-1. User object (only for notifications)
-2. Service object
-3. Host object
-4. Command object
-5. Global custom attributes in the `vars` constant
+Use the `apply` keyword to create `Notification` objects for your services:
-This execution order allows you to define default values for custom attributes
-in your command objects. The `my-ping` command shown above uses this to set
-default values for some of the latency thresholds and timeouts.
+ apply Notification "notify-cust-xy-mysql" to Service {
+ import "generic-notification"
-When using the `my-ping` command you can override some or all of the custom
-attributes in the service definition like this:
+ users = [ "noc-xy", "mgmt-xy" ]
- object Service "ping" {
- host_name = "localhost"
- check_command = "my-ping"
-
- vars.ping_packets = 10 // Overrides the default value of 5 given in the command
+ assign where match("*has gold support 24x7*", service.notes) && (host.vars.customer == "customer-xy" || host.vars.always_notify == true
+ ignore where match("*internal", host.name) || (service.vars.priority < 2 && host.vars.is_clustered == true)
}
-If a custom attribute isn't defined anywhere an empty value is used and a warning is
-emitted to the Icinga 2 log.
-> **Best Practice**
+Instead of assigning users to notifications, you can also add the `user_groups`
+attribute with a list of user groups to the `Notification` object. Icinga 2 will
+send notifications to all group members.
+
+> **Note**
>
-> By convention every host should have an `address` attribute. Hosts
-> which have an IPv6 address should also have an `address6` attribute.
+> Only users who have been notified of a problem before (`Warning`, `Critical`, `Unknown`
+> states for services, `Down` for hosts) will receive `Recovery` notifications.
-### <a id="runtime-custom-attribute-env-vars"></a> Runtime Custom Attributes as Environment Variables
+### <a id="notification-escalations"></a> Notification Escalations
-The `env` command object attribute specifies a list of environment variables with values calculated
-from either runtime macros or custom attributes which should be exported as environment variables
-prior to executing the command.
+When a problem notification is sent and a problem still exists at the time of re-notification
+you may want to escalate the problem to the next support level. A different approach
+is to configure the default notification by email, and escalate the problem via SMS
+if not already solved.
-This is useful for example for hiding sensitive information on the command line output
-when passing credentials to database checks:
+You can define notification start and end times as additional configuration
+attributes making the `Notification` object a so-called `notification escalation`.
+Using templates you can share the basic notification attributes such as users or the
+`interval` (and override them for the escalation then).
- object CheckCommand "mysql-health" {
- import "plugin-check-command"
+Using the example from above, you can define additional users being escalated for SMS
+notifications between start and end time.
- command = [
- PluginDir + "/check_mysql"
- ]
+ object User "icinga-oncall-2nd-level" {
+ display_name = "Icinga 2nd Level"
- arguments = {
- "-H" = "$mysql_address$"
- "-d" = "$mysql_database$"
- }
+ vars.mobile = "+1 555 424642"
+ }
- vars.mysql_address = "$address$"
- vars.mysql_database = "icinga"
- vars.mysql_user = "icinga_check"
- vars.mysql_pass = "password"
+ object User "icinga-oncall-1st-level" {
+ display_name = "Icinga 1st Level"
- env.MYSQLUSER = "$mysql_user$"
- env.MYSQLPASS = "$mysql_pass$"
+ vars.mobile = "+1 555 424642"
}
-### <a id="multiple-host-addresses-custom-attributes"></a> Multiple Host Addresses using Custom Attributes
+Define an additional [NotificationCommand](#notification) for SMS notifications.
-The following example defines a `Host` with three different interface addresses defined as
-custom attributes in the `vars` dictionary. The `if-eth0` and `if-eth1` services will import
-these values into the `address` custom attribute. This attribute is available through the
-generic `$address$` runtime macro.
+> **Note**
+>
+> The example is not complete as there are many different SMS providers.
+> Please note that sending SMS notifications will require an SMS provider
+> or local hardware with a SIM card active.
- object Host "multi-ip" {
- check_command = "dummy"
- vars.address_lo = "127.0.0.1"
- vars.address_eth0 = "10.0.0.10"
- vars.address_eth1 = "192.168.1.10"
+ object NotificationCommand "sms-notification" {
+ command = [
+ PluginDir + "/send_sms_notification",
+ "$mobile$",
+ "..."
}
- apply Service "if-eth0" {
- import "generic-service"
+The two new notification escalations are added onto the local host
+and its service `ping4` using the `generic-notification` template.
+The user `icinga-oncall-2nd-level` will get notified by SMS (`sms-notification`
+command) after `30m` until `1h`.
- vars.address = "$host.vars.address_eth0$"
- check_command = "my-generic-interface-check"
+> **Note**
+>
+> The `interval` was set to 15m in the `generic-notification`
+> template example. Lower that value in your escalations by using a secondary
+> template or by overriding the attribute directly in the `notifications` array
+> position for `escalation-sms-2nd-level`.
- assign where host.vars.address_eth0 != ""
- }
+If the problem does not get resolved nor acknowledged preventing further notifications
+the `escalation-sms-1st-level` user will be escalated `1h` after the initial problem was
+notified, but only for one hour (`2h` as `end` key for the `times` dictionary).
- apply Service "if-eth1" {
- import "generic-service"
+ apply Notification "mail" to Service {
+ import "generic-notification"
- vars.address = "$host.vars.address_eth1$"
- check_command = "my-generic-interface-check"
+ command = "mail-notification"
+ users = [ "icingaadmin" ]
- assign where host.vars.address_eth1 != ""
+ assign where service.name == "ping4"
}
- object CheckCommand "my-generic-interface-check" {
- import "plugin-check-command"
-
- command = "echo \"This would be the service $service.description$ using the address value: $address$\""
- }
+ apply Notification "escalation-sms-2nd-level" to Service {
+ import "generic-notification"
-The `CheckCommand` object is just an example to help you with testing and
-understanding the different custom attributes and runtime macros.
+ command = "sms-notification"
+ users = [ "icinga-oncall-2nd-level" ]
-### <a id="modified-attributes"></a> Modified Attributes
+ times = {
+ begin = 30m
+ end = 1h
+ }
-Icinga 2 allows you to modify defined object attributes at runtime different to
-the local configuration object attributes. These modified attributes are
-stored as bit-shifted-value and made available in backends. Icinga 2 stores
-modified attributes in its state file and restores them on restart.
+ assign where service.name == "ping4"
+ }
-Modified Attributes can be reset using external commands.
+ apply Notification "escalation-sms-1st-level" to Service {
+ import "generic-notification"
+ command = "sms-notification"
+ users = [ "icinga-oncall-1st-level" ]
-## <a id="runtime-macros"></a> Runtime Macros
+ times = {
+ begin = 1h
+ end = 2h
+ }
-Next to custom attributes there are additional runtime macros made available by Icinga 2.
-These runtime macros reflect the current object state and may change over time while
-custom attributes are configured statically (but can be modified at runtime using
-external commands).
+ assign where service.name == "ping4"
+ }
-### <a id="runtime-macro-evaluation-order"></a> Runtime Macro Evaluation Order
+### <a id="notification-delay"></a> Notification Delay
-Custom attributes can be accessed at [runtime](#runtime-custom-attributes) using their
-identifier omitting the `vars.` prefix.
-There are special cases when those custom attributes are not set and Icinga 2 provides
-a fallback to existing object attributes for example `host.address`.
+Sometimes the problem in question should not be notified when the notification is due
+(the object reaching the `HARD` state) but a defined time duration afterwards. In Icinga 2
+you can use the `times` dictionary and set `begin = 15m` as key and value if you want to
+postpone the notification window for 15 minutes. Leave out the `end` key - if not set,
+Icinga 2 will not check against any end time for this notification. Make sure to
+specify a relatively low notification `interval` to get notified soon enough again.
-In the following example the `$address$` macro will be resolved with the value of `vars.address`.
+ apply Notification "mail" to Service {
+ import "generic-notification"
- object Host "localhost" {
- import "generic-host"
- check_command = "my-host-macro-test"
- address = "127.0.0.1"
- vars.address = "127.2.2.2"
- }
+ command = "mail-notification"
+ users = [ "icingaadmin" ]
- object CheckCommand "my-host-macro-test" {
- command = "echo \"address: $address$ host.address: $host.address$ host.vars.address: $host.vars.address$\""
- }
+ interval = 5m
-The check command output will look like
+ times.begin = 15m // delay notification window
- "address: 127.2.2.2 host.address: 127.0.0.1 host.vars.address: 127.2.2.2"
+ assign where service.name == "ping4"
+ }
-If you alter the host object and remove the `vars.address` line, Icinga 2 will fail to look up `$address$` in the
-custom attributes dictionary and then look for the host object's attribute.
+### <a id="disable-renotification"></a> Disable Re-notifications
-The check command output will change to
+If you prefer to be notified only once, you can disable re-notifications by setting the
+`interval` attribute to `0`.
- "address: 127.0.0.1 host.address: 127.0.0.1 host.vars.address: "
+ apply Notification "notify-once" to Service {
+ import "generic-notification"
+ command = "mail-notification"
+ users = [ "icingaadmin" ]
-The same example can be defined for services overriding the `address` field based on a specific host custom attribute.
+ interval = 0 // disable re-notification
- object Host "localhost" {
- import "generic-host"
- address = "127.0.0.1"
- vars.macro_address = "127.3.3.3"
+ assign where service.name == "ping4"
}
- apply Service "my-macro-test" to Host {
- import "generic-service"
- check_command = "my-service-macro-test"
- vars.address = "$host.vars.macro_address$"
+### <a id="notification-filters-state-type"></a> Notification Filters by State and Type
- assign where host.address
- }
+If there are no notification state and type filter attributes defined at the `Notification`
+or `User` object Icinga 2 assumes that all states and types are being notified.
- object CheckCommand "my-service-macro-test" {
- command = "echo \"address: $address$ host.address: $host.address$ host.vars.macro_address: $host.vars.macro_address$ service.vars.address: $service.vars.address$\""
- }
+Available state and type filters for notifications are:
-When the service check is executed the output looks like
+ template Notification "generic-notification" {
- "address: 127.3.3.3 host.address: 127.0.0.1 host.vars.macro_address: 127.3.3.3 service.vars.address: 127.3.3.3"
+ states = [ Warning, Critical, Unknown ]
+ types = [ Problem, Acknowledgement, Recovery, Custom, FlappingStart,
+ FlappingEnd, DowntimeStart, DowntimeEnd, DowntimeRemoved ]
+ }
-That way you can easily override existing macros being accessed by their short name like `$address$` and refrain
-from defining multiple check commands (one for `$address$` and one for `$host.vars.macro_address$`).
+If you are familiar with Icinga 1.x `notification_options` please note that they have been split
+into type and state to allow more fine granular filtering for example on downtimes and flapping.
+You can filter for acknowledgements and custom notifications too.
-### <a id="host-runtime-macros"></a> Host Runtime Macros
+## <a id="commands"></a> Commands
-The following host custom attributes are available in all commands that are executed for
-hosts or services:
+Icinga 2 uses three different command object types to specify how
+checks should be performed, notifications should be sent, and
+events should be handled.
- Name | Description
- -----------------------------|--------------
- host.name | The name of the host object.
- host.display_name | The value of the `display_name` attribute.
- host.state | The host's current state. Can be one of `UNREACHABLE`, `UP` and `DOWN`.
- host.state_id | The host's current state. Can be one of `0` (up), `1` (down) and `2` (unreachable).
- host.state_type | The host's current state type. Can be one of `SOFT` and `HARD`.
- host.check_attempt | The current check attempt number.
- host.max_check_attempts | The maximum number of checks which are executed before changing to a hard state.
- host.last_state | The host's previous state. Can be one of `UNREACHABLE`, `UP` and `DOWN`.
- host.last_state_id | The host's previous state. Can be one of `0` (up), `1` (down) and `2` (unreachable).
- host.last_state_type | The host's previous state type. Can be one of `SOFT` and `HARD`.
- host.last_state_change | The last state change's timestamp.
- host.duration_sec | The time since the last state change.
- host.latency | The host's check latency.
- host.execution_time | The host's check execution time.
- host.output | The last check's output.
- host.perfdata | The last check's performance data.
- host.last_check | The timestamp when the last check was executed.
- host.num_services | Number of services associated with the host.
- host.num_services_ok | Number of services associated with the host which are in an `OK` state.
- host.num_services_warning | Number of services associated with the host which are in a `WARNING` state.
- host.num_services_unknown | Number of services associated with the host which are in an `UNKNOWN` state.
- host.num_services_critical | Number of services associated with the host which are in a `CRITICAL` state.
+### <a id="check-commands"></a> Check Commands
-### <a id="service-runtime-macros"></a> Service Runtime Macros
+[CheckCommand](6-object-types.md#objecttype-checkcommand) objects define the command line how
+a check is called.
-The following service macros are available in all commands that are executed for
-services:
+[CheckCommand](6-object-types.md#objecttype-checkcommand) objects are referenced by
+[Host](6-object-types.md#objecttype-host) and [Service](6-object-types.md#objecttype-service) objects
+using the `check_command` attribute.
- Name | Description
- ---------------------------|--------------
- service.name | The short name of the service object.
- service.display_name | The value of the `display_name` attribute.
- service.check_command | The short name of the command along with any arguments to be used for the check.
- service.state | The service's current state. Can be one of `OK`, `WARNING`, `CRITICAL` and `UNKNOWN`.
- service.state_id | The service's current state. Can be one of `0` (ok), `1` (warning), `2` (critical) and `3` (unknown).
- service.state_type | The service's current state type. Can be one of `SOFT` and `HARD`.
- service.check_attempt | The current check attempt number.
- service.max_check_attempts | The maximum number of checks which are executed before changing to a hard state.
- service.last_state | The service's previous state. Can be one of `OK`, `WARNING`, `CRITICAL` and `UNKNOWN`.
- service.last_state_id | The service's previous state. Can be one of `0` (ok), `1` (warning), `2` (critical) and `3` (unknown).
- service.last_state_type | The service's previous state type. Can be one of `SOFT` and `HARD`.
- service.last_state_change | The last state change's timestamp.
- service.duration_sec | The time since the last state change.
- service.latency | The service's check latency.
- service.execution_time | The service's check execution time.
- service.output | The last check's output.
- service.perfdata | The last check's performance data.
- service.last_check | The timestamp when the last check was executed.
+> **Note**
+>
+> Make sure that the [checker](8-cli-commands.md#features) feature is enabled in order to
+> execute checks.
-### <a id="command-runtime-macros"></a> Command Runtime Macros
+#### <a id="command-plugin-integration"></a> Integrate the Plugin with a CheckCommand Definition
-The following custom attributes are available in all commands:
+[CheckCommand](6-object-types.md#objecttype-checkcommand) objects require the [ITL template](7-icinga-template-library.md#itl-plugin-check-command)
+`plugin-check-command` to support native plugin based check methods.
- Name | Description
- -----------------------|--------------
- command.name | The name of the command object.
+Unless you have done so already, download your check plugin and put it
+into the [PluginDir](5-configuring-icinga-2.md#constants-conf) directory. The following example uses the
+`check_disk` plugin contained in the Monitoring Plugins package.
-### <a id="user-runtime-macros"></a> User Runtime Macros
+The plugin path and all command arguments are made a list of
+double-quoted string arguments for proper shell escaping.
-The following custom attributes are available in all commands that are executed for
-users:
+Call the `check_disk` plugin with the `--help` parameter to see
+all available options. Our example defines warning (`-w`) and
+critical (`-c`) thresholds for the disk usage. Without any
+partition defined (`-p`) it will check all local partitions.
- Name | Description
- -----------------------|--------------
- user.name | The name of the user object.
- user.display_name | The value of the display_name attribute.
+ icinga@icinga2 $ /usr/lib/nagios/plugins/check_disk --help
+ ...
+ This plugin checks the amount of used disk space on a mounted file system
+ and generates an alert if free space is less than one of the threshold values
-### <a id="notification-runtime-macros"></a> Notification Runtime Macros
- Name | Description
- -----------------------|--------------
- notification.type | The type of the notification.
- notification.author | The author of the notification comment, if existing.
- notification.comment | The comment of the notification, if existing.
+ Usage:
+ check_disk -w limit -c limit [-W limit] [-K limit] {-p path | -x device}
+ [-C] [-E] [-e] [-f] [-g group ] [-k] [-l] [-M] [-m] [-R path ] [-r path ]
+ [-t timeout] [-u unit] [-v] [-X type] [-N type]
+ ...
-### <a id="global-runtime-macros"></a> Global Runtime Macros
+> **Note**
+>
+> Don't execute plugins as `root` and always use the absolute path to the plugin! Trust us.
-The following macros are available in all executed commands:
+Next step is to understand how command parameters are being passed from
+a host or service object, and add a [CheckCommand](6-object-types.md#objecttype-checkcommand)
+definition based on these required parameters and/or default values.
- Name | Description
- -----------------------|--------------
- icinga.timet | Current UNIX timestamp.
- icinga.long_date_time | Current date and time including timezone information. Example: `2014-01-03 11:23:08 +0000`
- icinga.short_date_time | Current date and time. Example: `2014-01-03 11:23:08`
- icinga.date | Current date. Example: `2014-01-03`
- icinga.time | Current time including timezone information. Example: `11:23:08 +0000`
- icinga.uptime | Current uptime of the Icinga 2 process.
+#### <a id="command-passing-parameters"></a> Passing Check Command Parameters from Host or Service
-The following macros provide global statistics:
+Check command parameters are defined as custom attributes which can be accessed as runtime macros
+by the executed check command.
- Name | Description
- ----------------------------------|--------------
- icinga.num_services_ok | Current number of services in state 'OK'.
- icinga.num_services_warning | Current number of services in state 'Warning'.
- icinga.num_services_critical | Current number of services in state 'Critical'.
- icinga.num_services_unknown | Current number of services in state 'Unknown'.
- icinga.num_services_pending | Current number of pending services.
- icinga.num_services_unreachable | Current number of unreachable services.
- icinga.num_services_flapping | Current number of flapping services.
- icinga.num_services_in_downtime | Current number of services in downtime.
- icinga.num_services_acknowledged | Current number of acknowledged service problems.
- icinga.num_hosts_up | Current number of hosts in state 'Up'.
- icinga.num_hosts_down | Current number of hosts in state 'Down'.
- icinga.num_hosts_unreachable | Current number of unreachable hosts.
- icinga.num_hosts_flapping | Current number of flapping hosts.
- icinga.num_hosts_in_downtime | Current number of hosts in downtime.
- icinga.num_hosts_acknowledged | Current number of acknowledged host problems.
+Define the default check command custom attribute `disk_wfree` and `disk_cfree`
+(freely definable naming schema) and their default threshold values. You can
+then use these custom attributes as runtime macros for [command arguments](3-monitoring-basics.md#command-arguments)
+on the command line.
+> **Tip**
+>
+> Use a common command type as prefix for your command arguments to increase
+> readability. `disk_wfree` helps understanding the context better than just
+> `wfree` as argument.
-## <a id="check-result-freshness"></a> Check Result Freshness
+The default custom attributes can be overridden by the custom attributes
+defined in the service using the check command `my-disk`. The custom attributes
+can also be inherited from a parent template using additive inheritance (`+=`).
-In Icinga 2 active check freshness is enabled by default. It is determined by the
-`check_interval` attribute and no incoming check results in that period of time.
+ object CheckCommand "my-disk" {
+ import "plugin-check-command"
- threshold = last check execution time + check interval
+ command = [ PluginDir + "/check_disk" ]
-Passive check freshness is calculated from the `check_interval` attribute if set.
+ arguments = {
+ "-w" = "$disk_wfree$%"
+ "-c" = "$disk_cfree$%"
+ "-W" = "$disk_inode_wfree$%"
+ "-K" = "$disk_inode_cfree$%"
+ "-p" = "$disk_partitions$"
+ "-x" = "$disk_partitions_excluded$"
+ }
- threshold = last check result time + check interval
+ vars.disk_wfree = 20
+ vars.disk_cfree = 10
+ }
-If the freshness checks are invalid, a new check is executed defined by the
-`check_command` attribute.
+> **Note**
+>
+> A proper example for the `check_disk` plugin is already shipped with Icinga 2
+> ready to use with the [plugin check commands](7-icinga-template-library.md#plugin-check-command-disk).
+The host `localhost` with the applied service `basic-partitions` checks a basic set of disk partitions
+with modified custom attributes (warning thresholds at `10%`, critical thresholds at `5%`
+free disk space).
-## <a id="check-flapping"></a> Check Flapping
+The custom attribute `disk_partition` can either hold a single string or an array of
+string values for passing multiple partitions to the `check_disk` check plugin.
-The flapping algorithm used in Icinga 2 does not store the past states but
-calculcates the flapping threshold from a single value based on counters and
-half-life values. Icinga 2 compares the value with a single flapping threshold
-configuration attribute named `flapping_threshold`.
+ object Host "my-server" {
+ import "generic-host"
+ address = "127.0.0.1"
+ address6 = "::1"
-Flapping detection can be enabled or disabled using the `enable_flapping` attribute.
+ vars.local_disks["basic-partitions"] = {
+ disk_partitions = [ "/", "/tmp", "/var", "/home" ]
+ }
+ }
+ apply Service for (disk => config in host.vars.local_disks) {
+ import "generic-service"
+ check_command = "my-disk"
-## <a id="volatile-services"></a> Volatile Services
+ vars += config
-By default all services remain in a non-volatile state. When a problem
-occurs, the `SOFT` state applies and once `max_check_attempts` attribute
-is reached with the check counter, a `HARD` state transition happens.
-Notifications are only triggered by `HARD` state changes and are then
-re-sent defined by the `interval` attribute.
+ vars.disk_wfree = 10
+ vars.disk_cfree = 5
+ }
-It may be reasonable to have a volatile service which stays in a `HARD`
-state type if the service stays in a `NOT-OK` state. That way each
-service recheck will automatically trigger a notification unless the
-service is acknowledged or in a scheduled downtime.
+More details on using arrays in custom attributes can be found in
+[this chapter](#runtime-custom-attributes).
-## <a id="external-commands"></a> External Commands
-Icinga 2 provides an external command pipe for processing commands
-triggering specific actions (for example rescheduling a service check
-through the web interface).
+#### <a id="command-arguments"></a> Command Arguments
-In order to enable the `ExternalCommandListener` configuration use the
-following command and restart Icinga 2 afterwards:
+By defining a check command line using the `command` attribute Icinga 2
+will resolve all macros in the static string or array. Sometimes it is
+required to extend the arguments list based on a met condition evaluated
+at command execution. Or making arguments optional - only set if the
+macro value can be resolved by Icinga 2.
- # icinga2-enable-feature command
+ object CheckCommand "check_http" {
+ import "plugin-check-command"
-Icinga 2 creates the command pipe file as `/var/run/icinga2/cmd/icinga2.cmd`
-using the default configuration.
+ command = [ PluginDir + "/check_http" ]
-Web interfaces and other Icinga addons are able to send commands to
-Icinga 2 through the external command pipe, for example for rescheduling
-a forced service check:
+ arguments = {
+ "-H" = "$http_vhost$"
+ "-I" = "$http_address$"
+ "-u" = "$http_uri$"
+ "-p" = "$http_port$"
+ "-S" = {
+ set_if = "$http_ssl$"
+ }
+ "--sni" = {
+ set_if = "$http_sni$"
+ }
+ "-a" = {
+ value = "$http_auth_pair$"
+ description = "Username:password on sites with basic authentication"
+ }
+ "--no-body" = {
+ set_if = "$http_ignore_body$"
+ }
+ "-r" = "$http_expect_body_regex$"
+ "-w" = "$http_warn_time$"
+ "-c" = "$http_critical_time$"
+ "-e" = "$http_expect$"
+ }
- # /bin/echo "[`date +%s`] SCHEDULE_FORCED_SVC_CHECK;localhost;ping4;`date +%s`" >> /var/run/icinga2/cmd/icinga2.cmd
+ vars.http_address = "$address$"
+ vars.http_ssl = false
+ vars.http_sni = false
+ }
- # tail -f /var/log/messages
+The example shows the `check_http` check command defining the most common
+arguments. Each of them is optional by default and will be omitted if
+the value is not set. For example if the service calling the check command
+does not have `vars.http_port` set, it won't get added to the command
+line.
- Oct 17 15:01:25 icinga-server icinga2: Executing external command: [1382014885] SCHEDULE_FORCED_SVC_CHECK;localhost;ping4;1382014885
- Oct 17 15:01:25 icinga-server icinga2: Rescheduling next check for service 'ping4'
+If the `vars.http_ssl` custom attribute is set in the service, host or command
+object definition, Icinga 2 will add the `-S` argument based on the `set_if`
+numeric value to the command line. String values are not supported.
-By default the command pipe file is owned by the group `icingacmd` with read/write
-permissions. Add your webserver's user to the group `icingacmd` to
-enable sending commands to Icinga 2 through your web interface:
+If the macro value cannot be resolved, Icinga 2 will not add the defined argument
+to the final command argument array. Empty strings for macro values won't omit
+the argument.
- # usermod -G -a icingacmd www-data
+That way you can use the `check_http` command definition for both, with and
+without SSL enabled checks saving you duplicated command definitions.
-Debian packages use `nagios` as the default user and group name. Therefore change `icingacmd` to
-`nagios`. The webserver's user is different between distributions as well.
+Details on all available options can be found in the
+[CheckCommand object definition](6-object-types.md#objecttype-checkcommand).
-### <a id="external-command-list"></a> External Command List
-A list of currently supported external commands can be found [here](#external-commands-list-detail).
+### <a id="notification-commands"></a> Notification Commands
-Detailed information on the commands and their required parameters can be found
-on the [Icinga 1.x documentation](http://docs.icinga.org/latest/en/extcommands2.html).
+[NotificationCommand](6-object-types.md#objecttype-notificationcommand) objects define how notifications are delivered to external
+interfaces (E-Mail, XMPP, IRC, Twitter, etc).
+[NotificationCommand](6-object-types.md#objecttype-notificationcommand) objects are referenced by
+[Notification](6-object-types.md#objecttype-notification) objects using the `command` attribute.
-## <a id="logging"></a> Logging
+`NotificationCommand` objects require the [ITL template](7-icinga-template-library.md#itl-plugin-notification-command)
+`plugin-notification-command` to support native plugin-based notifications.
-Icinga 2 supports three different types of logging:
+> **Note**
+>
+> Make sure that the [notification](8-cli-commands.md#features) feature is enabled
+> in order to execute notification commands.
-* File logging
-* Syslog (on *NIX-based operating systems)
-* Console logging (`STDOUT` on tty)
+Below is an example using runtime macros from Icinga 2 (such as `$service.output$` for
+the current check output) sending an email to the user(s) associated with the
+notification itself (`$user.email$`).
-You can enable additional loggers using the `icinga2-enable-feature`
-and `icinga2-disable-feature` commands to configure loggers:
+If you want to specify default values for some of the custom attribute definitions,
+you can add a `vars` dictionary as shown for the `CheckCommand` object.
-Feature | Description
----------|------------
-debuglog | Debug log (path: `/var/log/icinga2/debug.log`, severity: `debug` or higher)
-mainlog | Main log (path: `/var/log/icinga2/icinga2.log`, severity: `information` or higher)
-syslog | Syslog (severity: `warning` or higher)
+ object NotificationCommand "mail-service-notification" {
+ import "plugin-notification-command"
-By default file the `mainlog` feature is enabled. When running Icinga 2
-on a terminal log messages with severity `information` or higher are
-written to the console.
+ command = [ SysconfDir + "/icinga2/scripts/mail-notification.sh" ]
+ env = {
+ NOTIFICATIONTYPE = "$notification.type$"
+ SERVICEDESC = "$service.name$"
+ HOSTALIAS = "$host.display_name$"
+ HOSTADDRESS = "$address$"
+ SERVICESTATE = "$service.state$"
+ LONGDATETIME = "$icinga.long_date_time$"
+ SERVICEOUTPUT = "$service.output$"
+ NOTIFICATIONAUTHORNAME = "$notification.author$"
+ NOTIFICATIONCOMMENT = "$notification.comment$"
+ HOSTDISPLAYNAME = "$host.display_name$"
+ SERVICEDISPLAYNAME = "$service.display_name$"
+ USEREMAIL = "$user.email$"
+ }
+ }
-## <a id="performance-data"></a> Performance Data
+The command attribute in the `mail-service-notification` command refers to the following
+shell script. The macros specified in the `env` array are exported
+as environment variables and can be used in the notification script:
-When a host or service check is executed plugins should provide so-called
-`performance data`. Next to that additional check performance data
-can be fetched using Icinga 2 runtime macros such as the check latency
-or the current service state (or additional custom attributes).
+ #!/usr/bin/env bash
+ template=$(cat <<TEMPLATE
+ ***** Icinga *****
-The performance data can be passed to external applications which aggregate and
-store them in their backends. These tools usually generate graphs for historical
-reporting and trending.
+ Notification Type: $NOTIFICATIONTYPE
-Well-known addons processing Icinga performance data are PNP4Nagios,
-inGraph and Graphite.
+ Service: $SERVICEDESC
+ Host: $HOSTALIAS
+ Address: $HOSTADDRESS
+ State: $SERVICESTATE
-### <a id="writing-performance-data-files"></a> Writing Performance Data Files
+ Date/Time: $LONGDATETIME
-PNP4Nagios, inGraph and Graphios use performance data collector daemons to fetch
-the current performance files for their backend updates.
+ Additional Info: $SERVICEOUTPUT
-Therefore the Icinga 2 `PerfdataWriter` object allows you to define
-the output template format for host and services backed with Icinga 2
-runtime vars.
+ Comment: [$NOTIFICATIONAUTHORNAME] $NOTIFICATIONCOMMENT
+ TEMPLATE
+ )
- host_format_template = "DATATYPE::HOSTPERFDATA\tTIMET::$icinga.timet$\tHOSTNAME::$host.name$\tHOSTPERFDATA::$host.perfdata$\tHOSTCHECKCOMMAND::$host.checkcommand$\tHOSTSTATE::$host.state$\tHOSTSTATETYPE::$host.statetype$"
- service_format_template = "DATATYPE::SERVICEPERFDATA\tTIMET::$icinga.timet$\tHOSTNAME::$host.name$\tSERVICEDESC::$service.name$\tSERVICEPERFDATA::$service.perfdata$\tSERVICECHECKCOMMAND::$service.checkcommand$\tHOSTSTATE::$host.state$\tHOSTSTATETYPE::$host.statetype$\tSERVICESTATE::$service.state$\tSERVICESTATETYPE::$service.statetype$"
+ /usr/bin/printf "%b" $template | mail -s "$NOTIFICATIONTYPE - $HOSTDISPLAYNAME - $SERVICEDISPLAYNAME is $SERVICESTATE" $USEREMAIL
-The default templates are already provided with the Icinga 2 feature configuration
-which can be enabled using
+> **Note**
+>
+> This example is for `exim` only. Requires changes for `sendmail` and
+> other MTAs.
- # icinga2-enable-feature perfdata
+While it's possible to specify the entire notification command right
+in the NotificationCommand object it is generally advisable to create a
+shell script in the `/etc/icinga2/scripts` directory and have the
+NotificationCommand object refer to that.
-By default all performance data files are rotated in a 15 seconds interval into
-the `/var/spool/icinga2/perfdata/` directory as `host-perfdata.<timestamp>` and
-`service-perfdata.<timestamp>`.
-External collectors need to parse the rotated performance data files and then
-remove the processed files.
+### <a id="event-commands"></a> Event Commands
-### <a id="graphite-carbon-cache-writer"></a> Graphite Carbon Cache Writer
+Unlike notifications event commands for hosts/services are called on every
+check execution if one of these conditions match:
-While there are some Graphite collector scripts and daemons like Graphios available for
-Icinga 1.x it's more reasonable to directly process the check and plugin performance
-in memory in Icinga 2. Once there are new metrics available, Icinga 2 will directly
-write them to the defined Graphite Carbon daemon tcp socket.
+* The host/service is in a [soft state](3-monitoring-basics.md#hard-soft-states)
+* The host/service state changes into a [hard state](3-monitoring-basics.md#hard-soft-states)
+* The host/service state recovers from a [soft or hard state](3-monitoring-basics.md#hard-soft-states) to [OK](3-monitoring-basics.md#service-states)/[Up](3-monitoring-basics.md#host-states)
-You can enable the feature using
+[EventCommand](6-object-types.md#objecttype-eventcommand) objects are referenced by
+[Host](6-object-types.md#objecttype-host) and [Service](6-object-types.md#objecttype-service) objects
+using the `event_command` attribute.
- # icinga2-enable-feature graphite
+Therefore the `EventCommand` object should define a command line
+evaluating the current service state and other service runtime attributes
+available through runtime vars. Runtime macros such as `$service.state_type$`
+and `$service.state$` will be processed by Icinga 2 helping on fine-granular
+events being triggered.
-By default the `GraphiteWriter` object expects the Graphite Carbon Cache to listen at
-`127.0.0.1` on port `2003`.
+Common use case scenarios are a failing HTTP check requiring an immediate
+restart via event command, or if an application is locked and requires
+a restart upon detection.
-The current naming schema is
+`EventCommand` objects require the ITL template `plugin-event-command`
+to support native plugin based checks.
- icinga.<hostname>.<metricname>
- icinga.<hostname>.<servicename>.<metricname>
+#### <a id="event-command-restart-service-daemon"></a> Use Event Commands to Restart Service Daemon
-To make sure Icinga 2 writes a valid label into Graphite some characters are replaced
-with `_` in the target name:
+The following example will triggert a restart of the `httpd` daemon
+via ssh when the `http` service check fails. If the service state is
+`OK`, it will not trigger any event action.
- \/.- (and space)
+Requirements:
-The resulting name in Graphite might look like:
+* ssh connection
+* icinga user with public key authentication
+* icinga user with sudo permissions for restarting the httpd daemon.
- www-01 / http-cert / response time
- icinga.www_01.http_cert.response_time
+Example on Debian:
-In addition to the performance data retrieved from the check plugin, Icinga 2 sends
-some internal statistics data of the check to Graphite:
+ # ls /home/icinga/.ssh/
+ authorized_keys
- metric | description
- -------------------|------------------------------------------
- current_attempt | attempt of the state evaluation
- max_check_attempts | max. attempts until hard state
- reachable | designated cluster component is reachable
- execution_time | check execution time
- latency | latency between check end and processing
- state | current state of the check
- state_type | 0=SOFT, 1=HARD state
+ # visudo
+ icinga ALL=(ALL) NOPASSWD: /etc/init.d/apache2 restart
-Here is an example how one could configure the storage-schemas for Carbon Cache.
-Please make sure your ordering is correct, the first match wins!
- [icinga_internals]
- pattern = ^icinga\..*\.(max_check_attempts|reachable|current_attempt|execution_time|latency|state|state_type)
- retentions = 5m:7d
+Define a generic [EventCommand](6-object-types.md#objecttype-eventcommand) object `event_by_ssh`
+which can be used for all event commands triggered using ssh:
- [icinga_default]
- # intervals like PNP4Nagios uses them per default
- pattern = ^icinga\.
- retentions = 1m:2d,5m:10d,30m:90d,360m:4y
+ /* pass event commands through ssh */
+ object EventCommand "event_by_ssh" {
+ import "plugin-event-command"
-## <a id="status-data"></a> Status Data
+ command = [ PluginDir + "/check_by_ssh" ]
-Icinga 1.x writes object configuration data and status data in a cyclic
-interval to its `objects.cache` and `status.dat` files. Icinga 2 provides
-the `StatusDataWriter` object which dumps all configuration objects and
-status updates in a regular interval.
+ arguments = {
+ "-H" = "$event_by_ssh_address$"
+ "-p" = "$event_by_ssh_port$"
+ "-C" = "$event_by_ssh_command$"
+ "-l" = "$event_by_ssh_logname$"
+ "-i" = "$event_by_ssh_identity$"
+ "-q" = {
+ set_if = "$event_by_ssh_quiet$"
+ }
+ "-w" = "$event_by_ssh_warn$"
+ "-c" = "$event_by_ssh_crit$"
+ "-t" = "$event_by_ssh_timeout$"
+ }
- # icinga2-enable-feature statusdata
+ vars.event_by_ssh_address = "$address$"
+ vars.event_by_ssh_quiet = false
+ }
-Icinga 1.x Classic UI requires this data set as part of its backend.
+The actual event command only passes the `event_by_ssh_command` attribute.
+The `event_by_ssh_service` custom attribute takes care of passing the correct
+daemon name, while `test $service.state_id$ -gt 0` makes sure that the daemon
+is only restarted when the service is an a not `OK` state.
-> **Note**
->
-> If you are not using any web interface or addon which uses these files
-> you can safely disable this feature.
+ object EventCommand "event_by_ssh_restart_service" {
+ import "event_by_ssh"
-## <a id="compat-logging"></a> Compat Logging
+ //only restart the daemon if state > 0 (not-ok)
+ //requires sudo permissions for the icinga user
+ vars.event_by_ssh_command = "test $service.state_id$ -gt 0 && sudo /etc/init.d/$event_by_ssh_service$ restart"
+ }
-The Icinga 1.x log format is considered being the `Compat Log`
-in Icinga 2 provided with the `CompatLogger` object.
-These logs are not only used for informational representation in
-external web interfaces parsing the logs, but also to generate
-SLA reports and trends in Icinga 1.x Classic UI. Furthermore the
-[Livestatus](#livestatus) feature uses these logs for answering queries to
-historical tables.
+Now set the `event_command` attribute to `event_by_ssh_restart_service` and tell it
+which service should be restarted using the `event_by_ssh_service` attribute.
-The `CompatLogger` object can be enabled with
+ object Service "http" {
+ import "generic-service"
+ host_name = "remote-http-host"
+ check_command = "http"
- # icinga2-enable-feature compatlog
+ event_command = "event_by_ssh_restart_service"
+ vars.event_by_ssh_service = "$host.vars.httpd_name$"
-By default, the Icinga 1.x log file called `icinga.log` is located
-in `/var/log/icinga2/compat`. Rotated log files are moved into
-`var/log/icinga2/compat/archives`.
+ //vars.event_by_ssh_logname = "icinga"
+ //vars.event_by_ssh_identity = "/home/icinga/.ssh/id_rsa.pub"
+ }
-The format cannot be changed without breaking compatibility to
-existing log parsers.
- # tail -f /var/log/icinga2/compat/icinga.log
+Each host with this service then must define the `httpd_name` custom attribute
+(for example generated from your cmdb):
- [1382115688] LOG ROTATION: HOURLY
- [1382115688] LOG VERSION: 2.0
- [1382115688] HOST STATE: CURRENT;localhost;UP;HARD;1;
- [1382115688] SERVICE STATE: CURRENT;localhost;disk;WARNING;HARD;1;
- [1382115688] SERVICE STATE: CURRENT;localhost;http;OK;HARD;1;
- [1382115688] SERVICE STATE: CURRENT;localhost;load;OK;HARD;1;
- [1382115688] SERVICE STATE: CURRENT;localhost;ping4;OK;HARD;1;
- [1382115688] SERVICE STATE: CURRENT;localhost;ping6;OK;HARD;1;
- [1382115688] SERVICE STATE: CURRENT;localhost;processes;WARNING;HARD;1;
- [1382115688] SERVICE STATE: CURRENT;localhost;ssh;OK;HARD;1;
- [1382115688] SERVICE STATE: CURRENT;localhost;users;OK;HARD;1;
- [1382115706] EXTERNAL COMMAND: SCHEDULE_FORCED_SVC_CHECK;localhost;disk;1382115705
- [1382115706] EXTERNAL COMMAND: SCHEDULE_FORCED_SVC_CHECK;localhost;http;1382115705
- [1382115706] EXTERNAL COMMAND: SCHEDULE_FORCED_SVC_CHECK;localhost;load;1382115705
- [1382115706] EXTERNAL COMMAND: SCHEDULE_FORCED_SVC_CHECK;localhost;ping4;1382115705
- [1382115706] EXTERNAL COMMAND: SCHEDULE_FORCED_SVC_CHECK;localhost;ping6;1382115705
- [1382115706] EXTERNAL COMMAND: SCHEDULE_FORCED_SVC_CHECK;localhost;processes;1382115705
- [1382115706] EXTERNAL COMMAND: SCHEDULE_FORCED_SVC_CHECK;localhost;ssh;1382115705
- [1382115706] EXTERNAL COMMAND: SCHEDULE_FORCED_SVC_CHECK;localhost;users;1382115705
- [1382115731] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;localhost;ping6;2;critical test|
- [1382115731] SERVICE ALERT: localhost;ping6;CRITICAL;SOFT;2;critical test
+ object Host "remote-http-host" {
+ import "generic-host"
+ address = "192.168.1.100"
+ vars.httpd_name = "apache2"
+ }
+You can testdrive this example by manually stopping the `httpd` daemon
+on your `remote-http-host`. Enable the `debuglog` feature and tail the
+`/var/log/icinga2/debug.log` file.
+Remote Host Terminal:
-## <a id="db-ido"></a> DB IDO
+ # date; service apache2 status
+ Mon Sep 15 18:57:39 CEST 2014
+ Apache2 is running (pid 23651).
+ # date; service apache2 stop
+ Mon Sep 15 18:57:47 CEST 2014
+ [ ok ] Stopping web server: apache2 ... waiting .
-The IDO (Icinga Data Output) modules for Icinga 2 take care of exporting all
-configuration and status information into a database. The IDO database is used
-by a number of projects including Icinga Web 1.x and 2.
+Icinga 2 Host Terminal:
-Details on the installation can be found in the [Getting Started](#configuring-ido)
-chapter. Details on the configuration can be found in the
-[IdoMysqlConnection](#objecttype-idomysqlconnection) and
-[IdoPgsqlConnection](#objecttype-idoPgsqlconnection)
-object configuration documentation.
-The DB IDO feature supports [High Availability](##high-availability-db-ido) in
-the Icinga 2 cluster.
+ [2014-09-15 18:58:32 +0200] notice/Process: Running command '/usr/lib64/nagios/plugins/check_http' '-I' '192.168.1.100': PID 32622
+ [2014-09-15 18:58:32 +0200] notice/Process: PID 32622 ('/usr/lib64/nagios/plugins/check_http' '-I' '192.168.1.100') terminated with exit code 2
+ [2014-09-15 18:58:32 +0200] notice/Checkable: State Change: Checkable remote-http-host!http soft state change from OK to CRITICAL detected.
+ [2014-09-15 18:58:32 +0200] notice/Checkable: Executing event handler 'event_by_ssh_restart_service' for service 'remote-http-host!http'
+ [2014-09-15 18:58:32 +0200] notice/Process: Running command '/usr/lib64/nagios/plugins/check_by_ssh' '-C' 'test 2 -gt 0 && sudo /etc/init.d/apache2 restart' '-H' '192.168.1.100': PID 32623
+ [2014-09-15 18:58:33 +0200] notice/Process: PID 32623 ('/usr/lib64/nagios/plugins/check_by_ssh' '-C' 'test 2 -gt 0 && sudo /etc/init.d/apache2 restart' '-H' '192.168.1.100') terminated with exit code 0
-The following example query checks the health of the current Icinga 2 instance
-writing its current status to the DB IDO backend table `icinga_programstatus`
-every 10 seconds. By default it checks 60 seconds into the past which is a reasonable
-amount of time - adjust it for your requirements. If the condition is not met,
-the query returns an empty result.
+Remote Host Terminal:
-> **Tip**
->
-> Use [check plugins](#plugins) to monitor the backend.
+ # date; service apache2 status
+ Mon Sep 15 18:58:44 CEST 2014
+ Apache2 is running (pid 24908).
-Replace the `default` string with your instance name, if different.
-Example for MySQL:
+## <a id="dependencies"></a> Dependencies
- # mysql -u root -p icinga -e "SELECT status_update_time FROM icinga_programstatus ps
- JOIN icinga_instances i ON ps.instance_id=i.instance_id
- WHERE (UNIX_TIMESTAMP(ps.status_update_time) > UNIX_TIMESTAMP(NOW())-60)
- AND i.instance_name='default';"
+Icinga 2 uses host and service [Dependency](6-object-types.md#objecttype-dependency) objects
+for determing their network reachability.
- +---------------------+
- | status_update_time |
- +---------------------+
- | 2014-05-29 14:29:56 |
- +---------------------+
+A service can depend on a host, and vice versa. A service has an implicit
+dependency (parent) to its host. A host to host dependency acts implicitly
+as host parent relation.
+When dependencies are calculated, not only the immediate parent is taken into
+account but all parents are inherited.
+The `parent_host_name` and `parent_service_name` attributes are mandatory for
+service dependencies, `parent_host_name` is required for host dependencies.
+[Apply rules](3-monitoring-basics.md#using-apply) will allow you to
+[determine these attributes](3-monitoring-basics.md#dependencies-apply-custom-attributes) in a more
+dynamic fashion if required.
-Example for PostgreSQL:
+ parent_host_name = "core-router"
+ parent_service_name = "uplink-port"
- # export PGPASSWORD=icinga; psql -U icinga -d icinga -c "SELECT ps.status_update_time FROM icinga_programstatus AS ps
- JOIN icinga_instances AS i ON ps.instance_id=i.instance_id
- WHERE ((SELECT extract(epoch from status_update_time) FROM icinga_programstatus) > (SELECT extract(epoch from now())-60))
- AND i.instance_name='default'";
+Notifications are suppressed by default if a host or service becomes unreachable.
+You can control that option by defining the `disable_notifications` attribute.
- status_update_time
- ------------------------
- 2014-05-29 15:11:38+02
- (1 Zeile)
+ disable_notifications = false
+If the dependency should be triggered in the parent object's soft state, you
+need to set `ignore_soft_states` to `false`.
-A detailed list on the available table attributes can be found in the [DB IDO Schema documentation](#schema-db-ido).
+The dependency state filter must be defined based on the parent object being
+either a host (`Up`, `Down`) or a service (`OK`, `Warning`, `Critical`, `Unknown`).
+The following example will make the dependency fail and trigger it if the parent
+object is **not** in one of these states:
-## <a id="livestatus"></a> Livestatus
+ states = [ OK, Critical, Unknown ]
-The [MK Livestatus](http://mathias-kettner.de/checkmk_livestatus.html) project
-implements a query protocol that lets users query their Icinga instance for
-status information. It can also be used to send commands.
+Rephrased: If the parent service object changes into the `Warning` state, this
+dependency will fail and render all child objects (hosts or services) unreachable.
-Details on the installation can be found in the [Getting Started](#setting-up-livestatus)
-chapter.
+You can determine the child's reachability by querying the `is_reachable` attribute
+in for example [DB IDO](22-appendix.md#schema-db-ido-extensions).
-### <a id="livestatus-sockets"></a> Livestatus Sockets
+### <a id="dependencies-implicit-host-service"></a> Implicit Dependencies for Services on Host
-Other to the Icinga 1.x Addon, Icinga 2 supports two socket types
+Icinga 2 automatically adds an implicit dependency for services on their host. That way
+service notifications are suppressed when a host is `DOWN` or `UNREACHABLE`. This dependency
+does not overwrite other dependencies and implicitely sets `disable_notifications = true` and
+`states = [ Up ]` for all service objects.
-* Unix socket (default)
-* TCP socket
+Service checks are still executed. If you want to prevent them from happening, you can
+apply the following dependency to all services setting their host as `parent_host_name`
+and disabling the checks. `assign where true` matches on all `Service` objects.
-Details on the configuration can be found in the [LivestatusListener](#objecttype-livestatuslistener)
-object configuration.
+ apply Dependency "disable-host-service-checks" to Service {
+ disable_checks = true
+ assign where true
+ }
-### <a id="livestatus-get-queries"></a> Livestatus GET Queries
+### <a id="dependencies-network-reachability"></a> Dependencies for Network Reachability
-> **Note**
->
-> All Livestatus queries require an additional empty line as query end identifier.
-> The `unixcat` tool is either available by the MK Livestatus project or as separate
-> binary.
+A common scenario is the Icinga 2 server behind a router. Checking internet
+access by pinging the Google DNS server `google-dns` is a common method, but
+will fail in case the `dsl-router` host is down. Therefore the example below
+defines a host dependency which acts implicitly as parent relation too.
-There also is a Perl module available in CPAN for accessing the Livestatus socket
-programmatically: [Monitoring::Livestatus](http://search.cpan.org/~nierlein/Monitoring-Livestatus-0.74/)
+Furthermore the host may be reachable but ping probes are dropped by the
+router's firewall. In case the `dsl-router``ping4` service check fails, all
+further checks for the `ping4` service on host `google-dns` service should
+be suppressed. This is achieved by setting the `disable_checks` attribute to `true`.
+ object Host "dsl-router" {
+ import "generic-host"
+ address = "192.168.1.1"
+ }
-Example using the unix socket:
+ object Host "google-dns" {
+ import "generic-host"
+ address = "8.8.8.8"
+ }
- # echo -e "GET services\n" | unixcat /var/run/icinga2/cmd/livestatus
+ apply Service "ping4" {
+ import "generic-service"
-Example using the tcp socket listening on port `6558`:
+ check_command = "ping4"
- # echo -e 'GET services\n' | netcat 127.0.0.1 6558
+ assign where host.address
+ }
- # cat servicegroups <<EOF
- GET servicegroups
+ apply Dependency "internet" to Host {
+ parent_host_name = "dsl-router"
+ disable_checks = true
+ disable_notifications = true
- EOF
+ assign where host.name != "dsl-router"
+ }
- (cat servicegroups; sleep 1) | netcat 127.0.0.1 6558
+ apply Dependency "internet" to Service {
+ parent_host_name = "dsl-router"
+ parent_service_name = "ping4"
+ disable_checks = true
+ assign where host.name != "dsl-router"
+ }
-### <a id="livestatus-command-queries"></a> Livestatus COMMAND Queries
+### <a id="dependencies-apply-custom-attributes"></a> Apply Dependencies based on Custom Attributes
-A list of available external commands and their parameters can be found [here](#external-commands-list-detail)
+You can use [apply rules](3-monitoring-basics.md#using-apply) to set parent or
+child attributes e.g. `parent_host_name`to other object's
+attributes.
- $ echo -e 'COMMAND <externalcommandstring>' | netcat 127.0.0.1 6558
+A common example are virtual machines hosted on a master. The object
+name of that master is auto-generated from your CMDB or VMWare inventory
+into the host's custom attributes (or a generic template for your
+cloud).
+Define your master host object:
-### <a id="livestatus-filters"></a> Livestatus Filters
+ /* your master */
+ object Host "master.example.com" {
+ import "generic-host"
+ }
-and, or, negate
+Add a generic template defining all common host attributes:
- Operator | Negate | Description
- ----------|------------------------
- = | != | Equality
- ~ | !~ | Regex match
- =~ | !=~ | Equality ignoring case
- ~~ | !~~ | Regex ignoring case
- < | | Less than
- > | | Greater than
- <= | | Less than or equal
- >= | | Greater than or equal
+ /* generic template for your virtual machines */
+ template Host "generic-vm" {
+ import "generic-host"
+ }
+Add a template for all hosts on your example.com cloud setting
+custom attribute `vm_parent` to `master.example.com`:
-### <a id="livestatus-stats"></a> Livestatus Stats
+ template Host "generic-vm-example.com" {
+ import "generic-vm"
+ vars.vm_parent = "master.example.com"
+ }
-Schema: "Stats: aggregatefunction aggregateattribute"
+Define your guest hosts:
- Aggregate Function | Description
- -------------------|--------------
- sum |
- min |
- max |
- avg | sum / count
- std | standard deviation
- suminv | sum (1 / value)
- avginv | suminv / count
- count | ordinary default for any stats query if not aggregate function defined
+ object Host "www.example1.com" {
+ import "generic-vm-master.example.com"
+ }
-Example:
+ object Host "www.example2.com" {
+ import "generic-vm-master.example.com"
+ }
- GET hosts
- Filter: has_been_checked = 1
- Filter: check_type = 0
- Stats: sum execution_time
- Stats: sum latency
- Stats: sum percent_state_change
- Stats: min execution_time
- Stats: min latency
- Stats: min percent_state_change
- Stats: max execution_time
- Stats: max latency
- Stats: max percent_state_change
- OutputFormat: json
- ResponseHeader: fixed16
+Apply the host dependency to all child hosts importing the
+`generic-vm` template and set the `parent_host_name`
+to the previously defined custom attribute `host.vars.vm_parent`.
-### <a id="livestatus-output"></a> Livestatus Output
+ apply Dependency "vm-host-to-parent-master" to Host {
+ parent_host_name = host.vars.vm_parent
+ assign where "generic-vm" in host.templates
+ }
-* CSV
+You can extend this example, and make your services depend on the
+`master.example.com` host too. Their local scope allows you to use
+`host.vars.vm_parent` similar to the example above.
-CSV Output uses two levels of array separators: The members array separator
-is a comma (1st level) while extra info and host|service relation separator
-is a pipe (2nd level).
+ apply Dependency "vm-service-to-parent-master" to Service {
+ parent_host_name = host.vars.vm_parent
+ assign where "generic-vm" in host.templates
+ }
-Separators can be set using ASCII codes like:
+That way you don't need to wait for your guest hosts becoming
+unreachable when the master host goes down. Instead the services
+will detect their reachability immediately when executing checks.
- Separators: 10 59 44 124
+> **Note**
+>
+> This method with setting locally scoped variables only works in
+> apply rules, but not in object definitions.
-* JSON
-Default separators.
+### <a id="dependencies-agent-checks"></a> Dependencies for Agent Checks
-### <a id="livestatus-error-codes"></a> Livestatus Error Codes
+Another classic example are agent based checks. You would define a health check
+for the agent daemon responding to your requests, and make all other services
+querying that daemon depend on that health check.
- Code | Description
- ----------|--------------
- 200 | OK
- 404 | Table does not exist
- 452 | Exception on query
+The following configuration defines two nrpe based service checks `nrpe-load`
+and `nrpe-disk` applied to the `nrpe-server`. The health check is defined as
+`nrpe-health` service.
-### <a id="livestatus-tables"></a> Livestatus Tables
+ apply Service "nrpe-health" {
+ import "generic-service"
+ check_command = "nrpe"
+ assign where match("nrpe-*", host.name)
+ }
- Table | Join |Description
- --------------|-----------|----------------------------
- hosts | | host config and status attributes, services counter
- hostgroups | | hostgroup config, status attributes and host/service counters
- services | hosts | service config and status attributes
- servicegroups | | servicegroup config, status attributes and service counters
- contacts | | contact config and status attributes
- contactgroups | | contact config, members
- commands | | command name and line
- status | | programstatus, config and stats
- comments | services | status attributes
- downtimes | services | status attributes
- timeperiods | | name and is inside flag
- endpoints | | config and status attributes
- log | services, hosts, contacts, commands | parses [compatlog](#objecttype-compatlogger) and shows log attributes
- statehist | hosts, services | parses [compatlog](#objecttype-compatlogger) and aggregates state change attributes
+ apply Service "nrpe-load" {
+ import "generic-service"
+ check_command = "nrpe"
+ vars.nrpe_command = "check_load"
+ assign where match("nrpe-*", host.name)
+ }
-The `commands` table is populated with `CheckCommand`, `EventCommand` and `NotificationCommand` objects.
+ apply Service "nrpe-disk" {
+ import "generic-service"
+ check_command = "nrpe"
+ vars.nrpe_command = "check_disk"
+ assign where match("nrpe-*", host.name)
+ }
-A detailed list on the available table attributes can be found in the [Livestatus Schema documentation](#schema-livestatus).
+ object Host "nrpe-server" {
+ import "generic-host"
+ address = "192.168.1.5"
+ }
+ apply Dependency "disable-nrpe-checks" to Service {
+ parent_service_name = "nrpe-health"
-## <a id="check-result-files"></a> Check Result Files
+ states = [ OK ]
+ disable_checks = true
+ disable_notifications = true
+ assign where service.check_command == "nrpe"
+ ignore where service.name == "nrpe-health"
+ }
-Icinga 1.x writes its check result files to a temporary spool directory
-where they are processed in a regular interval.
-While this is extremely inefficient in performance regards it has been
-rendered useful for passing passive check results directly into Icinga 1.x
-skipping the external command pipe.
+The `disable-nrpe-checks` dependency is applied to all services
+on the `nrpe-service` host using the `nrpe` check_command attribute
+but not the `nrpe-health` service itself.
-Several clustered/distributed environments and check-aggregation addons
-use that method. In order to support step-by-step migration of these
-environments, Icinga 2 ships the `CheckResultReader` object.
-There is no feature configuration available, but it must be defined
-on-demand in your Icinga 2 objects configuration.
- object CheckResultReader "reader" {
- spool_dir = "/data/check-results"
- }
projects / icinga2 / blobdiff