1 # Advanced Topics <a id="advanced-topics"></a>
3 This chapter covers a number of advanced topics. If you're new to Icinga, you
4 can safely skip over things you're not interested in.
6 ## Downtimes <a id="downtimes"></a>
8 Downtimes can be scheduled for planned server maintenance or
9 any other targeted service outage you are aware of in advance.
11 Downtimes suppress notifications and can trigger other
12 downtimes too. If the downtime was set by accident, or the duration
13 exceeds the maintenance windows, you can manually cancel the downtime.
15 ### Scheduling a downtime <a id="scheduling-downtime"></a>
17 The most convenient way to schedule planned downtimes is to create
18 them in Icinga Web 2 inside the host/service detail view. Select
19 multiple hosts/services from the listing with the shift key to
20 schedule multiple downtimes.
22 ![Downtime in Icinga Web 2](images/advanced-topics/icingaweb2_downtime_handled.png)
24 In addition to that you can schedule a downtime by using the Icinga 2 API action
25 [schedule-downtime](12-icinga2-api.md#icinga2-api-actions-schedule-downtime).
26 This is especially useful to schedule a downtime on-demand inside a (remote) backup
27 script, or create maintenance downtimes from a cron job for specific dates and intervals.
29 Multiple downtimes for a single object may overlap. This is useful
30 when you want to extend your maintenance window taking longer than expected.
31 If there are multiple downtimes triggered for one object, the overall downtime depth
32 will be greater than `1`.
34 If the downtime was scheduled after the problem changed to a critical hard
35 state triggering a problem notification, and the service recovers during
36 the downtime window, the recovery notification won't be suppressed.
38 Planned downtimes are also taken into account for SLA reporting
39 tools calculating the SLAs based on the state and downtime history.
41 ### Fixed and Flexible Downtimes <a id="fixed-flexible-downtimes"></a>
43 A `fixed` downtime will be activated at the defined start time, and
44 removed at the end time. During this time window the service state
45 will change to `NOT-OK` and then actually trigger the downtime.
46 Notifications are suppressed and the downtime depth is incremented.
48 Common scenarios are a planned distribution upgrade on your linux
49 servers, or database updates in your warehouse. The customer knows
50 about a fixed downtime window between 23:00 and 24:00. After 24:00
51 all problems should be alerted again. Solution is simple -
52 schedule a `fixed` downtime starting at 23:00 and ending at 24:00.
54 Unlike a `fixed` downtime, a `flexible` downtime will be triggered
55 by the state change in the time span defined by start and end time,
56 and then last for the specified duration in minutes.
58 Imagine the following scenario: Your service is frequently polled
59 by users trying to grab free deleted domains for immediate registration.
60 Between 07:30 and 08:00 the impact will hit for 15 minutes and generate
61 a network outage visible to the monitoring. The service is still alive,
62 but answering too slow to Icinga 2 service checks.
63 For that reason, you may want to schedule a downtime between 07:30 and
64 08:00 with a duration of 15 minutes. The downtime will then last from
65 its trigger time until the duration is over. After that, the downtime
66 is removed (may happen before or after the actual end time!).
68 #### Fixed Downtime <a id="fixed-downtime"></a>
70 If the host/service changes into a NOT-OK state between the start and
71 end time window, the downtime will be marked as `in effect` and
72 increases the downtime depth counter.
80 #### Flexible Downtime <a id="flexible-downtime"></a>
82 A flexible downtime defines a time window where the downtime may be
83 triggered from a host/service NOT-OK state change. It will then last
84 until the specified time duration is reached. That way it can happen
85 that the downtime end time is already gone, but the downtime ends
86 at `trigger time + duration`.
91 start | end actual end time
92 |--------------duration--------|
97 ### Triggered Downtimes <a id="triggered-downtimes"></a>
99 This is optional when scheduling a downtime. If there is already a downtime
100 scheduled for a future maintenance, the current downtime can be triggered by
101 that downtime. This renders useful if you have scheduled a host downtime and
102 are now scheduling a child host's downtime getting triggered by the parent
103 downtime on `NOT-OK` state change.
105 ### Recurring Downtimes <a id="recurring-downtimes"></a>
107 [ScheduledDowntime objects](09-object-types.md#objecttype-scheduleddowntime) can be used to set up
108 recurring downtimes for services.
113 apply ScheduledDowntime "backup-downtime" to Service {
114 author = "icingaadmin"
115 comment = "Scheduled downtime for backup"
118 monday = "02:00-03:00"
119 tuesday = "02:00-03:00"
120 wednesday = "02:00-03:00"
121 thursday = "02:00-03:00"
122 friday = "02:00-03:00"
123 saturday = "02:00-03:00"
124 sunday = "02:00-03:00"
127 assign where "backup" in service.groups
131 Icinga 2 attempts to find the next possible segment from a ScheduledDowntime object's
132 `ranges` attribute, and wont create multiple downtimes in the future. In case you need
133 all these downtimes planned and visible for the next days, weeks or months, schedule them
134 manually via the [REST API](12-icinga2-api.md#icinga2-api-actions-schedule-downtime) using
135 a script or cron job.
139 > If ScheduledDowntime objects are synced in a distributed high-availability setup,
140 > both will create the next possible downtime on their own. These runtime generated
141 > downtimes are synced among both zone instances, and you may see sort-of duplicate downtimes
145 ## Comments <a id="comments-intro"></a>
147 Comments can be added at runtime and are persistent over restarts. You can
148 add useful information for others on repeating incidents (for example
149 "last time syslog at 100% cpu on 17.10.2013 due to stale nfs mount") which
150 is primarily accessible using web interfaces.
152 You can add a comment either by using the Icinga 2 API action
153 [add-comment](12-icinga2-api.md#icinga2-api-actions-add-comment) or
154 by sending an [external command](14-features.md#external-commands).
156 ## Acknowledgements <a id="acknowledgements"></a>
158 If a problem persists and notifications have been sent, you can
159 acknowledge the problem. That way other users will get
160 a notification that you're aware of the issue and probably are
161 already working on a fix.
163 Note: Acknowledgements also add a new [comment](08-advanced-topics.md#comments-intro)
164 which contains the author and text fields.
166 You can send an acknowledgement either by using the Icinga 2 API action
167 [acknowledge-problem](12-icinga2-api.md#icinga2-api-actions-acknowledge-problem) or
168 by sending an [external command](14-features.md#external-commands).
171 ### Sticky Acknowledgements <a id="sticky-acknowledgements"></a>
173 The acknowledgement is removed if a state change occurs or if the host/service
174 recovers (OK/Up state).
176 If you acknowledge a problem once you've received a `Critical` notification,
177 the acknowledgement will be removed if there is a state transition to `Warning`.
179 OK -> WARNING -> CRITICAL -> WARNING -> OK
182 If you prefer to keep the acknowledgement until the problem is resolved (`OK`
183 recovery) you need to enable the `sticky` parameter.
186 ### Expiring Acknowledgements <a id="expiring-acknowledgements"></a>
188 Once a problem is acknowledged it may disappear from your `handled problems`
189 dashboard and no-one ever looks at it again since it will suppress
192 This `fire-and-forget` action is quite common. If you're sure that a
193 current problem should be resolved in the future at a defined time,
194 you can define an expiration time when acknowledging the problem.
196 Icinga 2 will clear the acknowledgement when expired and start to
197 re-notify, if the problem persists.
200 ## Time Periods <a id="timeperiods"></a>
202 [Time Periods](09-object-types.md#objecttype-timeperiod) define
203 time ranges in Icinga where event actions are triggered, for
204 example whether a service check is executed or not within
205 the `check_period` attribute. Or a notification should be sent to
206 users or not, filtered by the `period` and `notification_period`
207 configuration attributes for `Notification` and `User` objects.
209 The `TimePeriod` attribute `ranges` may contain multiple directives,
210 including weekdays, days of the month, and calendar dates.
211 These types may overlap/override other types in your ranges dictionary.
213 The descending order of precedence is as follows:
215 * Calendar date (2008-01-01)
216 * Specific month date (January 1st)
217 * Generic month date (Day 15)
218 * Offset weekday of specific month (2nd Tuesday in December)
219 * Offset weekday (3rd Monday)
220 * Normal weekday (Tuesday)
222 If you don't set any `check_period` or `notification_period` attribute
223 on your configuration objects, Icinga 2 assumes `24x7` as time period
227 object TimePeriod "24x7" {
228 display_name = "Icinga 2 24x7 TimePeriod"
230 "monday" = "00:00-24:00"
231 "tuesday" = "00:00-24:00"
232 "wednesday" = "00:00-24:00"
233 "thursday" = "00:00-24:00"
234 "friday" = "00:00-24:00"
235 "saturday" = "00:00-24:00"
236 "sunday" = "00:00-24:00"
241 If your operation staff should only be notified during workhours,
242 create a new timeperiod named `workhours` defining a work day from
246 object TimePeriod "workhours" {
247 display_name = "Icinga 2 8x5 TimePeriod"
249 "monday" = "09:00-17:00"
250 "tuesday" = "09:00-17:00"
251 "wednesday" = "09:00-17:00"
252 "thursday" = "09:00-17:00"
253 "friday" = "09:00-17:00"
258 If you want to specify a notification period across midnight,
259 you can define it the following way:
262 object Timeperiod "across-midnight" {
263 display_name = "Nightly Notification"
265 "saturday" = "22:00-24:00"
266 "sunday" = "00:00-03:00"
271 Below you can see another example for configuring timeperiods across several
272 days, weeks or months. This can be useful when taking components offline
273 for a distinct period of time.
276 object Timeperiod "standby" {
277 display_name = "Standby"
279 "2016-09-30 - 2016-10-30" = "00:00-24:00"
284 Please note that the spaces before and after the dash are mandatory.
286 Once your time period is configured you can Use the `period` attribute
287 to assign time periods to `Notification` and `Dependency` objects:
290 apply Notification "mail-icingaadmin" to Service {
291 import "mail-service-notification"
292 user_groups = host.vars.notification.mail.groups
293 users = host.vars.notification.mail.users
297 assign where host.vars.notification.mail
301 ### Time Periods Inclusion and Exclusion <a id="timeperiods-includes-excludes"></a>
303 Sometimes it is necessary to exclude certain time ranges from
304 your default time period definitions, for example, if you don't
305 want to send out any notification during the holiday season,
306 or if you only want to allow small time windows for executed checks.
308 The [TimePeriod object](09-object-types.md#objecttype-timeperiod)
309 provides the `includes` and `excludes` attributes to solve this issue.
310 `prefer_includes` defines whether included or excluded time periods are
313 The following example defines a time period called `holidays` where
314 notifications should be suppressed:
317 object TimePeriod "holidays" {
319 "january 1" = "00:00-24:00" //new year's day
320 "july 4" = "00:00-24:00" //independence day
321 "december 25" = "00:00-24:00" //christmas
322 "december 31" = "18:00-24:00" //new year's eve (6pm+)
323 "2017-04-16" = "00:00-24:00" //easter 2017
324 "monday -1 may" = "00:00-24:00" //memorial day (last monday in may)
325 "monday 1 september" = "00:00-24:00" //labor day (1st monday in september)
326 "thursday 4 november" = "00:00-24:00" //thanksgiving (4th thursday in november)
331 In addition to that the time period `weekends` defines an additional
332 time window which should be excluded from notifications:
335 object TimePeriod "weekends-excluded" {
337 "saturday" = "00:00-09:00,18:00-24:00"
338 "sunday" = "00:00-09:00,18:00-24:00"
343 The time period `prod-notification` defines the default time ranges
344 and adds the excluded time period names as an array.
347 object TimePeriod "prod-notification" {
348 excludes = [ "holidays", "weekends-excluded" ]
351 "monday" = "00:00-24:00"
352 "tuesday" = "00:00-24:00"
353 "wednesday" = "00:00-24:00"
354 "thursday" = "00:00-24:00"
355 "friday" = "00:00-24:00"
356 "saturday" = "00:00-24:00"
357 "sunday" = "00:00-24:00"
362 ## External Check Results <a id="external-check-results"></a>
364 Hosts or services which do not actively execute a check plugin to receive
365 the state and output are called "passive checks" or "external check results".
366 In this scenario an external client or script is sending in check results.
368 You can feed check results into Icinga 2 with the following transport methods:
370 * [process-check-result action](12-icinga2-api.md#icinga2-api-actions-process-check-result) available with the [REST API](12-icinga2-api.md#icinga2-api) (remote and local)
371 * External command sent via command pipe (local only)
373 Each time a new check result is received, the next expected check time
374 is updated. This means that if there are no check result received from
375 the external source, Icinga 2 will execute [freshness checks](08-advanced-topics.md#check-result-freshness).
379 > The REST API action allows to specify the `check_source` attribute
380 > which helps identifying the external sender. This is also visible
381 > in Icinga Web 2 and the REST API queries.
383 ## Check Result Freshness <a id="check-result-freshness"></a>
385 In Icinga 2 active check freshness is enabled by default. It is determined by the
386 `check_interval` attribute and no incoming check results in that period of time.
388 The threshold is calculated based on the last check execution time for actively executed checks:
391 (last check execution time + check interval) > current time
394 If this host/service receives check results from an [external source](08-advanced-topics.md#external-check-results),
395 the threshold is based on the last time a check result was received:
398 (last check result time + check interval) > current time
403 > The [process-check-result](12-icinga2-api.md#icinga2-api-actions-process-check-result) REST API
404 > action allows to overrule the pre-defined check interval with a specified TTL in Icinga 2 v2.9+.
406 If the freshness checks fail, Icinga 2 will execute the defined check command.
408 Best practice is to define a [dummy](10-icinga-template-library.md#itl-dummy) `check_command` which gets
409 executed when freshness checks fail.
412 apply Service "external-check" {
413 check_command = "dummy"
416 /* Set the state to UNKNOWN (3) if freshness checks fail. */
419 /* Use a runtime function to retrieve the last check time and more details. */
421 var service = get_service(macro("$host.name$"), macro("$service.name$"))
422 var lastCheck = DateTime(service.last_check).to_string()
424 return "No check results received. Last result time: " + lastCheck
427 assign where "external" in host.vars.services
431 References: [get_service](18-library-reference.md#objref-get_service), [macro](18-library-reference.md#scoped-functions-macro), [DateTime](18-library-reference.md#datetime-type).
433 Example output in Icinga Web 2:
435 ![Icinga 2 Freshness Checks](images/advanced-topics/icinga2_external_checks_freshness_icingaweb2.png)
438 ## Check Flapping <a id="check-flapping"></a>
440 Icinga 2 supports optional detection of hosts and services that are "flapping".
442 Flapping occurs when a service or host changes state too frequently, which would result in a storm of problem and
443 recovery notifications. With flapping detection enabled a flapping notification will be sent while other notifications are
444 suppressed until it calms down after receiving the same status from checks a few times. Flapping detection can help detect
445 configuration problems (wrong thresholds), troublesome services or network problems.
447 Flapping detection can be enabled or disabled using the `enable_flapping` attribute.
448 The `flapping_threshold_high` and `flapping_threshold_low` attributes allows to specify the thresholds that control
449 when a [host](09-object-types.md#objecttype-host) or [service](09-object-types.md#objecttype-service) is considered to be flapping.
451 The default thresholds are 30% for high and 25% for low. If the computed flapping value exceeds the high threshold a
452 host or service is considered flapping until it drops below the low flapping threshold.
454 `FlappingStart` and `FlappingEnd` notifications will be sent out accordingly, if configured. See the chapter on
455 [notifications](alert-notifications) for details
457 > Note: There is no distinctions between hard and soft states with flapping. All state changes count and notifications
458 > will be sent out regardless of the objects state.
460 ### How it works <a id="check-flapping-how-it-works"></a>
462 Icinga 2 saves the last 20 state changes for every host and service. See the graphic below:
464 ![Icinga 2 Flapping State Timeline](images/advanced-topics/flapping-state-graph.png)
466 All the states are weighted, with the most recent one being worth the most (1.15) and the 20th the least (0.8). The
467 states in between are fairly distributed. The final flapping value are the weighted state changes divided by the total
470 In the example above, the added states would have a total value of 7.82 (`0.84 + 0.86 + 0.88 + 0.9 + 0.98 + 1.06 + 1.12 + 1.18`).
471 This yields a flapping percentage of 39.1% (`7.82 / 20 * 100`). As the default upper flapping threshold is 30%, it would be
474 If the next seven check results then would not be state changes, the flapping percentage would fall below the lower threshold
475 of 25% and therefore the host or service would recover from flapping.
477 ## Volatile Services and Hosts <a id="volatile-services-hosts"></a>
479 The `volatile` option, if enabled for a host or service, makes it treat every [state change](03-monitoring-basics.md#hard-soft-states)
480 as a `HARD` state change. It is comparable to `max_check_attempts = 1`. With this any `NOT-OK` result will
481 ignore `max_check_attempts` and trigger notifications etc. It will further cause any additional `NOT-OK`
482 result to re-send notifications.
484 It may be reasonable to have a volatile service which stays in a `HARD` state if the service stays in a `NOT-OK`
485 state. That way each service recheck will automatically trigger a notification unless the service is acknowledged or
486 in a scheduled downtime.
488 A common example are security checks where each `NOT-OK` check result should immediately trigger a notification.
490 The default for this option is `false` and should only be enabled when required.
493 ## Monitoring Icinga 2 <a id="monitoring-icinga"></a>
495 Why should you do that? Icinga and its components run like any other
496 service application on your server. There are predictable issues
497 such as "disk space is running low" and your monitoring suffers from just
500 You would also like to ensure that features and backends are running
501 and storing required data. Be it the database backend where Icinga Web 2
502 presents fancy dashboards, forwarded metrics to Graphite or InfluxDB or
503 the entire distributed setup.
505 This list isn't complete but should help with your own setup.
506 Windows client specific checks are highlighted.
508 Type | Description | Plugins and CheckCommands
509 ----------------|-------------------------------|-----------------------------------------------------
510 System | Filesystem | [disk](10-icinga-template-library.md#plugin-check-command-disk), [disk-windows](10-icinga-template-library.md#windows-plugins) (Windows Client)
511 System | Memory, Swap | [mem](10-icinga-template-library.md#plugin-contrib-command-mem), [swap](10-icinga-template-library.md#plugin-check-command-swap), [memory](10-icinga-template-library.md#windows-plugins) (Windows Client)
512 System | Hardware | [hpasm](10-icinga-template-library.md#plugin-contrib-command-hpasm), [ipmi-sensor](10-icinga-template-library.md#plugin-contrib-command-ipmi-sensor)
513 System | Virtualization | [VMware](10-icinga-template-library.md#plugin-contrib-vmware), [esxi_hardware](10-icinga-template-library.md#plugin-contrib-command-esxi-hardware)
514 System | Processes | [procs](10-icinga-template-library.md#plugin-check-command-processes), [service-windows](10-icinga-template-library.md#windows-plugins) (Windows Client)
515 System | System Activity Reports | [check_sar_perf](https://github.com/dnsmichi/icinga-plugins/blob/master/scripts/check_sar_perf.py)
516 System | I/O | [iostat](10-icinga-template-library.md#plugin-contrib-command-iostat)
517 System | Network interfaces | [nwc_health](10-icinga-template-library.md#plugin-contrib-command-nwc_health), [interfaces](10-icinga-template-library.md#plugin-contrib-command-interfaces)
518 System | Users | [users](10-icinga-template-library.md#plugin-check-command-users), [users-windows](10-icinga-template-library.md#windows-plugins) (Windows Client)
519 System | Logs | Forward them to [Elastic Stack](14-features.md#elastic-stack-integration) or [Graylog](14-features.md#graylog-integration) and add your own alerts.
520 System | NTP | [ntp_time](10-icinga-template-library.md#plugin-check-command-ntp-time)
521 System | Updates | [apt](10-icinga-template-library.md#plugin-check-command-apt), [yum](10-icinga-template-library.md#plugin-contrib-command-yum)
522 Icinga | Status & Stats | [icinga](10-icinga-template-library.md#itl-icinga) (more below)
523 Icinga | Cluster & Clients | [health checks](06-distributed-monitoring.md#distributed-monitoring-health-checks)
524 Database | MySQL | [mysql_health](10-icinga-template-library.md#plugin-contrib-command-mysql_health)
525 Database | PostgreSQL | [postgres](10-icinga-template-library.md#plugin-contrib-command-postgres)
526 Database | Housekeeping | Check the database size and growth and analyse metrics to examine trends.
527 Database | DB IDO | [ido](10-icinga-template-library.md#itl-icinga-ido) (more below)
528 Webserver | Apache2, Nginx, etc. | [http](10-icinga-template-library.md#plugin-check-command-http), [apache-status](10-icinga-template-library.md#plugin-contrib-command-apache-status), [nginx_status](10-icinga-template-library.md#plugin-contrib-command-nginx_status)
529 Webserver | Certificates | [http](10-icinga-template-library.md#plugin-check-command-http)
530 Webserver | Authorization | [http](10-icinga-template-library.md#plugin-check-command-http)
531 Notifications | Mail (queue) | [smtp](10-icinga-template-library.md#plugin-check-command-smtp), [mailq](10-icinga-template-library.md#plugin-check-command-mailq)
532 Notifications | SMS (GSM modem) | [check_sms3_status](https://exchange.icinga.com/netways/check_sms3status)
533 Notifications | Messengers, Cloud services | XMPP, Twitter, IRC, Telegram, PagerDuty, VictorOps, etc.
534 Metrics | PNP, RRDTool | [check_pnp_rrds](https://github.com/lingej/pnp4nagios/tree/master/scripts) checks for stale RRD files.
535 Metrics | Graphite | [graphite](10-icinga-template-library.md#plugin-contrib-command-graphite)
536 Metrics | InfluxDB | [check_influxdb](https://exchange.icinga.com/Mikanoshi/InfluxDB+data+monitoring+plugin)
537 Metrics | Elastic Stack | [elasticsearch](10-icinga-template-library.md#plugin-contrib-command-elasticsearch), [Elastic Stack integration](14-features.md#elastic-stack-integration)
538 Metrics | Graylog | [Graylog integration](14-features.md#graylog-integration)
541 The [icinga](10-icinga-template-library.md#itl-icinga) CheckCommand provides metrics for the runtime stats of
542 Icinga 2. You can forward them to your preferred graphing solution.
543 If you require more metrics you can also query the [REST API](12-icinga2-api.md#icinga2-api) and write
544 your own custom check plugin. Or you keep using the built-in [object accessor functions](08-advanced-topics.md#access-object-attributes-at-runtime)
545 to calculate stats in-memory.
547 There is a built-in [ido](10-icinga-template-library.md#itl-icinga-ido) check available for DB IDO MySQL/PostgreSQL
548 which provides additional metrics for the IDO database.
551 apply Service "ido-mysql" {
552 check_command = "ido"
554 vars.ido_type = "IdoMysqlConnection"
555 vars.ido_name = "ido-mysql" //the name defined in /etc/icinga2/features-enabled/ido-mysql.conf
557 assign where match("master*.localdomain", host.name)
561 More specific database queries can be found in the [DB IDO](14-features.md#db-ido) chapter.
563 Distributed setups should include specific [health checks](06-distributed-monitoring.md#distributed-monitoring-health-checks).
564 You might also want to add additional checks for SSL certificate expiration.
567 ## Advanced Configuration Hints <a id="advanced-configuration-hints"></a>
569 ### Advanced Use of Apply Rules <a id="advanced-use-of-apply-rules"></a>
571 [Apply rules](03-monitoring-basics.md#using-apply) can be used to create a rule set which is
572 entirely based on host objects and their attributes.
573 In addition to that [apply for and custom attribute override](03-monitoring-basics.md#using-apply-for)
574 extend the possibilities.
576 The following example defines a dictionary on the host object which contains
577 configuration attributes for multiple web servers. This then used to add three checks:
579 * A `ping4` check using the local IP `address` of the web server.
580 * A `tcp` check querying the TCP port where the HTTP service is running on.
581 * If the `url` key is defined, the third apply for rule will create service objects using the `http` CheckCommand.
582 In addition to that you can optionally define the `ssl` attribute which enables HTTPS checks.
587 object Host "webserver01" {
588 import "generic-host"
589 address = "192.168.56.200"
593 instance["status"] = {
594 address = "192.168.56.201"
598 instance["tomcat"] = {
599 address = "192.168.56.202"
602 instance["icingaweb2"] = {
603 address = "192.168.56.210"
612 Service apply for definitions:
615 apply Service "webserver_ping" for (instance => config in host.vars.webserver.instance) {
616 display_name = "webserver_" + instance
617 check_command = "ping4"
619 vars.ping_address = config.address
621 assign where host.vars.webserver.instance
624 apply Service "webserver_port" for (instance => config in host.vars.webserver.instance) {
625 display_name = "webserver_" + instance + "_" + config.port
626 check_command = "tcp"
628 vars.tcp_address = config.address
629 vars.tcp_port = config.port
631 assign where host.vars.webserver.instance
634 apply Service "webserver_url" for (instance => config in host.vars.webserver.instance) {
635 display_name = "webserver_" + instance + "_" + config.url
636 check_command = "http"
638 vars.http_address = config.address
639 vars.http_port = config.port
640 vars.http_uri = config.url
643 vars.http_ssl = config.ssl
646 assign where config.url != ""
650 The variables defined in the host dictionary are not using the typical custom attribute
651 prefix recommended for CheckCommand parameters. Instead they are re-used for multiple
652 service checks in this example.
653 In addition to defining check parameters this way, you can also enrich the `display_name`
654 attribute with more details. This will be shown in in Icinga Web 2 for example.
656 ### Use Functions in Object Configuration <a id="use-functions-object-config"></a>
658 There is a limited scope where functions can be used as object attributes such as:
660 * As value for [Custom Attributes](03-monitoring-basics.md#custom-attributes-functions)
661 * Returning boolean expressions for [set_if](08-advanced-topics.md#use-functions-command-arguments-setif) inside command arguments
662 * Returning a [command](08-advanced-topics.md#use-functions-command-attribute) array inside command objects
664 The other way around you can create objects dynamically using your own global functions.
668 > Functions called inside command objects share the same global scope as runtime macros.
669 > Therefore you can access host custom attributes like `host.vars.os`, or any other
670 > object attribute from inside the function definition used for [set_if](08-advanced-topics.md#use-functions-command-arguments-setif) or [command](08-advanced-topics.md#use-functions-command-attribute).
672 Tips when implementing functions:
674 * Use [log()](18-library-reference.md#global-functions-log) to dump variables. You can see the output
675 inside the `icinga2.log` file depending in your log severity
676 * Use the `icinga2 console` to test basic functionality (e.g. iterating over a dictionary)
677 * Build them step-by-step. You can always refactor your code later on.
679 #### Register and Use Global Functions <a id="use-functions-global-register"></a>
681 [Functions](17-language-reference.md#functions) can be registered into the global scope. This allows custom functions being available
682 in objects and other functions. Keep in mind that these functions are not marked
683 as side-effect-free and as such are not available via the REST API.
685 Add a new configuration file `functions.conf` and include it into the [icinga2.conf](04-configuring-icinga-2.md#icinga2-conf)
686 configuration file in the very beginning, e.g. after `constants.conf`. You can also manage global
687 functions inside `constants.conf` if you prefer.
689 The following function converts a given state parameter into a returned string value. The important
690 bits for registering it into the global scope are:
692 * `globals.<unique_function_name>` adds a new globals entry.
693 * `function()` specifies that a call to `state_to_string()` executes a function.
694 * Function parameters are defined inside the `function()` definition.
697 globals.state_to_string = function(state) {
700 } else if (state == 1) {
702 } else if (state == 0) {
704 } else if (state == 3) {
707 log(LogWarning, "state_to_string", "Unknown state " + state + " provided.")
712 The else-condition allows for better error handling. This warning will be shown in the Icinga 2
713 log file once the function is called.
717 > If these functions are used in a distributed environment, you must ensure to deploy them
720 In order to test-drive the newly created function, restart Icinga 2 and use the [debug console](11-cli-commands.md#cli-command-console)
721 to connect to the REST API.
724 $ ICINGA2_API_PASSWORD=icinga icinga2 console --connect 'https://root@localhost:5665/'
725 Icinga 2 (version: v2.8.1-373-g4bea6d25c)
726 <1> => globals.state_to_string(1)
728 <2> => state_to_string(2)
732 You can see that this function is now registered into the [global scope](17-language-reference.md#variable-scopes). The function call
733 `state_to_string()` can be used in any object at static config compile time or inside runtime
736 The following service object example uses the service state and converts it to string output.
737 The function definition is not optimized and is enrolled for better readability including a log message.
740 object Service "state-test" {
741 check_command = "dummy"
747 var h = macro("$host.name$")
748 var s = macro("$service.name$")
750 var state = get_service(h, s).state
752 log(LogInformation, "dummy_state", "Host: " + h + " Service: " + s + " State: " + state)
754 return state_to_string(state)
760 #### Use Custom Functions as Attribute <a id="custom-functions-as-attribute"></a>
762 To use custom functions as attributes, the function must be defined in a
763 slightly unexpected way. The following example shows how to assign values
764 depending on group membership. All hosts in the `slow-lan` host group use 300
765 as value for `ping_wrta`, all other hosts use 100.
768 globals.group_specific_value = function(group, group_value, non_group_value) {
769 return function() use (group, group_value, non_group_value) {
770 if (group in host.groups) {
773 return non_group_value
778 apply Service "ping4" {
779 import "generic-service"
780 check_command = "ping4"
782 vars.ping_wrta = group_specific_value("slow-lan", 300, 100)
783 vars.ping_crta = group_specific_value("slow-lan", 500, 200)
789 #### Use Functions in Assign Where Expressions <a id="use-functions-assign-where"></a>
791 If a simple expression for matching a name or checking if an item
792 exists in an array or dictionary does not fit, you should consider
793 writing your own global [functions](17-language-reference.md#functions).
794 You can call them inside `assign where` and `ignore where` expressions
795 for [apply rules](03-monitoring-basics.md#using-apply-expressions) or
796 [group assignments](03-monitoring-basics.md#group-assign-intro) just like
797 any other global functions for example [match](18-library-reference.md#global-functions-match).
799 The following example requires the host `myprinter` being added
800 to the host group `printers-lexmark` but only if the host uses
801 a template matching the name `lexmark*`.
804 template Host "lexmark-printer-host" {
805 vars.printer_type = "Lexmark"
808 object Host "myprinter" {
809 import "generic-host"
810 import "lexmark-printer-host"
812 address = "192.168.1.1"
815 /* register a global function for the assign where call */
816 globals.check_host_templates = function(host, search) {
817 /* iterate over all host templates and check if the search matches */
818 for (tmpl in host.templates) {
819 if (match(search, tmpl)) {
824 /* nothing matched */
828 object HostGroup "printers-lexmark" {
829 display_name = "Lexmark Printers"
830 /* call the global function and pass the arguments */
831 assign where check_host_templates(host, "lexmark*")
835 Take a different more complex example: All hosts with the
836 custom attribute `vars_app` as nested dictionary should be
837 added to the host group `ABAP-app-server`. But only if the
838 `app_type` for all entries is set to `ABAP`.
840 It could read as wildcard match for nested dictionaries:
843 where host.vars.vars_app["*"].app_type == "ABAP"
846 The solution for this problem is to register a global
847 function which checks the `app_type` for all hosts
848 with the `vars_app` dictionary.
851 object Host "appserver01" {
852 check_command = "dummy"
853 vars.vars_app["ABC"] = { app_type = "ABAP" }
855 object Host "appserver02" {
856 check_command = "dummy"
857 vars.vars_app["DEF"] = { app_type = "ABAP" }
860 globals.check_app_type = function(host, type) {
861 /* ensure that other hosts without the custom attribute do not match */
862 if (typeof(host.vars.vars_app) != Dictionary) {
866 /* iterate over the vars_app dictionary */
867 for (key => val in host.vars.vars_app) {
868 /* if the value is a dictionary and if contains the app_type being the requested type */
869 if (typeof(val) == Dictionary && val.app_type == type) {
874 /* nothing matched */
878 object HostGroup "ABAP-app-server" {
879 assign where check_app_type(host, "ABAP")
883 #### Use Functions in Command Arguments set_if <a id="use-functions-command-arguments-setif"></a>
885 The `set_if` attribute inside the command arguments definition in the
886 [CheckCommand object definition](09-object-types.md#objecttype-checkcommand) is primarily used to
887 evaluate whether the command parameter should be set or not.
889 By default you can evaluate runtime macros for their existence. If the result is not an empty
890 string, the command parameter is passed. This becomes fairly complicated when want to evaluate
891 multiple conditions and attributes.
893 The following example was found on the community support channels. The user had defined a host
894 dictionary named `compellent` with the key `disks`. This was then used inside service apply for rules.
897 object Host "dict-host" {
898 check_command = "check_compellent"
899 vars.compellent["disks"] = {
900 file = "/var/lib/check_compellent/san_disks.0.json",
906 The more significant problem was to only add the command parameter `--disk` to the plugin call
907 when the dictionary `compellent` contains the key `disks`, and omit it if not found.
909 By defining `set_if` as [abbreviated lambda function](17-language-reference.md#nullary-lambdas)
910 and evaluating the host custom attribute `compellent` containing the `disks` this problem was
914 object CheckCommand "check_compellent" {
915 command = [ "/usr/bin/check_compellent" ]
919 var host_vars = host.vars
921 var compel = host_vars.compellent
923 compel.contains("disks")
930 This implementation uses the dictionary type method [contains](18-library-reference.md#dictionary-contains)
931 and will fail if `host.vars.compellent` is not of the type `Dictionary`.
932 Therefore you can extend the checks using the [typeof](17-language-reference.md#types) function.
934 You can test the types using the `icinga2 console`:
938 Icinga (version: v2.3.0-193-g3eb55ad)
939 <1> => srv_vars.compellent["check_a"] = { file="outfile_a.json", checks = [ "disks", "fans" ] }
941 <2> => srv_vars.compellent["check_b"] = { file="outfile_b.json", checks = [ "power", "voltages" ] }
943 <3> => typeof(srv_vars.compellent)
948 The more programmatic approach for `set_if` could look like this:
953 var srv_vars = service.vars
954 if(len(srv_vars) > 0) {
955 if (typeof(srv_vars.compellent) == Dictionary) {
956 return srv_vars.compellent.contains("disks")
958 log(LogInformationen, "checkcommand set_if", "custom attribute compellent_checks is not a dictionary, ignoring it.")
962 log(LogWarning, "checkcommand set_if", "empty custom attributes")
969 #### Use Functions as Command Attribute <a id="use-functions-command-attribute"></a>
971 This comes in handy for [NotificationCommands](09-object-types.md#objecttype-notificationcommand)
972 or [EventCommands](09-object-types.md#objecttype-eventcommand) which does not require
973 a returned checkresult including state/output.
975 The following example was taken from the community support channels. The requirement was to
976 specify a custom attribute inside the notification apply rule and decide which notification
977 script to call based on that.
980 object User "short-dummy" {
983 object UserGroup "short-dummy-group" {
984 assign where user.name == "short-dummy"
987 apply Notification "mail-admins-short" to Host {
988 import "mail-host-notification"
989 command = "mail-host-notification-test"
990 user_groups = [ "short-dummy-group" ]
992 assign where host.vars.notification.mail
996 The solution is fairly simple: The `command` attribute is implemented as function returning
997 an array required by the caller Icinga 2.
998 The local variable `mailscript` sets the default value for the notification scrip location.
999 If the notification custom attribute `short` is set, it will override the local variable `mailscript`
1001 The `mailscript` variable is then used to compute the final notification command array being
1004 You can omit the `log()` calls, they only help debugging.
1007 object NotificationCommand "mail-host-notification-test" {
1009 log("command as function")
1010 var mailscript = "mail-host-notification-long.sh"
1011 if (notification.vars.short) {
1012 mailscript = "mail-host-notification-short.sh"
1014 log("Running command")
1017 var cmd = [ ConfigDir + "/scripts/" + mailscript ]
1018 log(LogCritical, "me", cmd)
1027 ### Access Object Attributes at Runtime <a id="access-object-attributes-at-runtime"></a>
1029 The [Object Accessor Functions](18-library-reference.md#object-accessor-functions)
1030 can be used to retrieve references to other objects by name.
1032 This allows you to access configuration and runtime object attributes. A detailed
1033 list can be found [here](09-object-types.md#object-types).
1035 #### Access Object Attributes at Runtime: Cluster Check <a id="access-object-attributes-at-runtime-cluster-check"></a>
1037 This is a simple cluster example for accessing two host object states and calculating a virtual
1038 cluster state and output:
1041 object Host "cluster-host-01" {
1042 check_command = "dummy"
1043 vars.dummy_state = 2
1044 vars.dummy_text = "This host is down."
1047 object Host "cluster-host-02" {
1048 check_command = "dummy"
1049 vars.dummy_state = 0
1050 vars.dummy_text = "This host is up."
1053 object Host "cluster" {
1054 check_command = "dummy"
1055 vars.cluster_nodes = [ "cluster-host-01", "cluster-host-02" ]
1057 vars.dummy_state = {{
1060 var cluster_nodes = macro("$cluster_nodes$")
1062 for (node in cluster_nodes) {
1063 if (get_host(node).state > 0) {
1070 if (up_count >= down_count) {
1071 return 0 //same up as down -> UP
1073 return 2 //something is broken
1077 vars.dummy_text = {{
1078 var output = "Cluster hosts:\n"
1079 var cluster_nodes = macro("$cluster_nodes$")
1081 for (node in cluster_nodes) {
1082 output += node + ": " + get_host(node).last_check_result.output + "\n"
1090 #### Time Dependent Thresholds <a id="access-object-attributes-at-runtime-time-dependent-thresholds"></a>
1092 The following example sets time dependent thresholds for the load check based on the current
1093 time of the day compared to the defined time period.
1096 object TimePeriod "backup" {
1098 monday = "02:00-03:00"
1099 tuesday = "02:00-03:00"
1100 wednesday = "02:00-03:00"
1101 thursday = "02:00-03:00"
1102 friday = "02:00-03:00"
1103 saturday = "02:00-03:00"
1104 sunday = "02:00-03:00"
1108 object Host "webserver-with-backup" {
1109 check_command = "hostalive"
1110 address = "127.0.0.1"
1113 object Service "webserver-backup-load" {
1114 check_command = "load"
1115 host_name = "webserver-with-backup"
1117 vars.load_wload1 = {{
1118 if (get_time_period("backup").is_inside) {
1124 vars.load_cload1 = {{
1125 if (get_time_period("backup").is_inside) {
1135 ## Advanced Value Types <a id="advanced-value-types"></a>
1137 In addition to the default value types Icinga 2 also uses a few other types
1138 to represent its internal state. The following types are exposed via the [API](12-icinga2-api.md#icinga2-api).
1140 ### CheckResult <a id="advanced-value-types-checkresult"></a>
1142 Name | Type | Description
1143 --------------------------|-----------------------|----------------------------------
1144 exit\_status | Number | The exit status returned by the check execution.
1145 output | String | The check output.
1146 performance\_data | Array | Array of [performance data values](08-advanced-topics.md#advanced-value-types-perfdatavalue).
1147 check\_source | String | Name of the node executing the check.
1148 state | Number | The current state (0 = OK, 1 = WARNING, 2 = CRITICAL, 3 = UNKNOWN).
1149 command | Value | Array of command with shell-escaped arguments or command line string.
1150 execution\_start | Timestamp | Check execution start time (as a UNIX timestamp).
1151 execution\_end | Timestamp | Check execution end time (as a UNIX timestamp).
1152 schedule\_start | Timestamp | Scheduled check execution start time (as a UNIX timestamp).
1153 schedule\_end | Timestamp | Scheduled check execution end time (as a UNIX timestamp).
1154 active | Boolean | Whether the result is from an active or passive check.
1155 vars\_before | Dictionary | Internal attribute used for calculations.
1156 vars\_after | Dictionary | Internal attribute used for calculations.
1157 ttl | Number | Time-to-live duration in seconds for this check result. The next expected check result is `now + ttl` where freshness checks are executed.
1159 ### PerfdataValue <a id="advanced-value-types-perfdatavalue"></a>
1161 Icinga 2 parses performance data strings returned by check plugins and makes the information available to external interfaces (e.g. [GraphiteWriter](09-object-types.md#objecttype-graphitewriter) or the [Icinga 2 API](12-icinga2-api.md#icinga2-api)).
1163 Name | Type | Description
1164 --------------------------|-----------------------|----------------------------------
1165 label | String | Performance data label.
1166 value | Number | Normalized performance data value without unit.
1167 counter | Boolean | Enabled if the original value contains `c` as unit. Defaults to `false`.
1168 unit | String | Unit of measurement (`seconds`, `bytes`. `percent`) according to the [plugin API](05-service-monitoring.md#service-monitoring-plugin-api).
1169 crit | Value | Critical threshold value.
1170 warn | Value | Warning threshold value.
1171 min | Value | Minimum value returned by the check.
1172 max | Value | Maximum value returned by the check.
1174 ### NotificationResult <a id="advanced-value-types-notificationresult"></a>
1176 Name | Type | Description
1177 --------------------------|-----------------------|----------------------------------
1178 exit\_status | Number | The exit status returned by the check execution.
1179 output | String | The notification command output.
1180 execution\_endpoint | String | Name of the node executing the check.
1181 command | Value | Array of command with shell-escaped arguments or command line string.
1182 execution\_start | Timestamp | Check execution start time (as a UNIX timestamp).
1183 execution\_end | Timestamp | Check execution end time (as a UNIX timestamp).
1184 active | Boolean | Whether the result is from an active or passive check.