1 # <a id="advanced-topics"></a> Advanced Topics
3 This chapter covers a number of advanced topics. If you're new to Icinga, you
4 can safely skip over things you're not interested in.
6 ## <a id="downtimes"></a> Downtimes
8 Downtimes can be scheduled for planned server maintenance or
9 any other targeted service outage you are aware of in advance.
11 Downtimes will suppress any notifications, and may trigger other
12 downtimes too. If the downtime was set by accident, or the duration
13 exceeds the maintenance, you can manually cancel the downtime.
14 Planned downtimes will also be taken into account for SLA reporting
15 tools calculating the SLAs based on the state and downtime history.
17 Multiple downtimes for a single object may overlap. This is useful
18 when you want to extend your maintenance window taking longer than expected.
19 If there are multiple downtimes triggered for one object, the overall downtime depth
20 will be greater than `1`.
23 If the downtime was scheduled after the problem changed to a critical hard
24 state triggering a problem notification, and the service recovers during
25 the downtime window, the recovery notification won't be suppressed.
27 ### <a id="fixed-flexible-downtimes"></a> Fixed and Flexible Downtimes
29 A `fixed` downtime will be activated at the defined start time, and
30 removed at the end time. During this time window the service state
31 will change to `NOT-OK` and then actually trigger the downtime.
32 Notifications are suppressed and the downtime depth is incremented.
34 Common scenarios are a planned distribution upgrade on your linux
35 servers, or database updates in your warehouse. The customer knows
36 about a fixed downtime window between 23:00 and 24:00. After 24:00
37 all problems should be alerted again. Solution is simple -
38 schedule a `fixed` downtime starting at 23:00 and ending at 24:00.
40 Unlike a `fixed` downtime, a `flexible` downtime will be triggered
41 by the state change in the time span defined by start and end time,
42 and then last for the specified duration in minutes.
44 Imagine the following scenario: Your service is frequently polled
45 by users trying to grab free deleted domains for immediate registration.
46 Between 07:30 and 08:00 the impact will hit for 15 minutes and generate
47 a network outage visible to the monitoring. The service is still alive,
48 but answering too slow to Icinga 2 service checks.
49 For that reason, you may want to schedule a downtime between 07:30 and
50 08:00 with a duration of 15 minutes. The downtime will then last from
51 its trigger time until the duration is over. After that, the downtime
52 is removed (may happen before or after the actual end time!).
54 ### <a id="scheduling-downtime"></a> Scheduling a downtime
56 This can either happen through a web interface or by sending an [external command](15-features.md#external-commands)
57 to the external command pipe provided by the `ExternalCommandListener` configuration.
59 Fixed downtimes require a start and end time (a duration will be ignored).
60 Flexible downtimes need a start and end time for the time span, and a duration
61 independent from that time span.
63 ### <a id="triggered-downtimes"></a> Triggered Downtimes
65 This is optional when scheduling a downtime. If there is already a downtime
66 scheduled for a future maintenance, the current downtime can be triggered by
67 that downtime. This renders useful if you have scheduled a host downtime and
68 are now scheduling a child host's downtime getting triggered by the parent
69 downtime on `NOT-OK` state change.
71 ### <a id="recurring-downtimes"></a> Recurring Downtimes
73 [ScheduledDowntime objects](6-object-types.md#objecttype-scheduleddowntime) can be used to set up
74 recurring downtimes for services.
78 apply ScheduledDowntime "backup-downtime" to Service {
79 author = "icingaadmin"
80 comment = "Scheduled downtime for backup"
83 monday = "02:00-03:00"
84 tuesday = "02:00-03:00"
85 wednesday = "02:00-03:00"
86 thursday = "02:00-03:00"
87 friday = "02:00-03:00"
88 saturday = "02:00-03:00"
89 sunday = "02:00-03:00"
92 assign where "backup" in service.groups
96 ## <a id="comments-intro"></a> Comments
98 Comments can be added at runtime and are persistent over restarts. You can
99 add useful information for others on repeating incidents (for example
100 "last time syslog at 100% cpu on 17.10.2013 due to stale nfs mount") which
101 is primarily accessible using web interfaces.
103 Adding and deleting comment actions are possible through the external command pipe
104 provided with the `ExternalCommandListener` configuration. The caller must
105 pass the comment id in case of manipulating an existing comment.
108 ## <a id="acknowledgements"></a> Acknowledgements
110 If a problem is alerted and notified, you may signal the other notification
111 recipients that you are aware of the problem and will handle it.
113 By sending an acknowledgement to Icinga 2 (using the external command pipe
114 provided with `ExternalCommandListener` configuration) all future notifications
115 are suppressed, a new comment is added with the provided description and
116 a notification with the type `NotificationFilterAcknowledgement` is sent
117 to all notified users.
119 ### <a id="expiring-acknowledgements"></a> Expiring Acknowledgements
121 Once a problem is acknowledged it may disappear from your `handled problems`
122 dashboard and no-one ever looks at it again since it will suppress
125 This `fire-and-forget` action is quite common. If you're sure that a
126 current problem should be resolved in the future at a defined time,
127 you can define an expiration time when acknowledging the problem.
129 Icinga 2 will clear the acknowledgement when expired and start to
130 re-notify, if the problem persists.
133 ## <a id="timeperiods"></a> Time Periods
135 [Time Periods](6-object-types.md#objecttype-timeperiod) define
136 time ranges in Icinga where event actions are triggered, for
137 example whether a service check is executed or not within
138 the `check_period` attribute. Or a notification should be sent to
139 users or not, filtered by the `period` and `notification_period`
140 configuration attributes for `Notification` and `User` objects.
144 > If you are familiar with Icinga 1.x, these time period definitions
145 > are called `legacy timeperiods` in Icinga 2.
147 > An Icinga 2 legacy timeperiod requires the `ITL` provided template
148 >`legacy-timeperiod`.
150 The `TimePeriod` attribute `ranges` may contain multiple directives,
151 including weekdays, days of the month, and calendar dates.
152 These types may overlap/override other types in your ranges dictionary.
154 The descending order of precedence is as follows:
156 * Calendar date (2008-01-01)
157 * Specific month date (January 1st)
158 * Generic month date (Day 15)
159 * Offset weekday of specific month (2nd Tuesday in December)
160 * Offset weekday (3rd Monday)
161 * Normal weekday (Tuesday)
163 If you don't set any `check_period` or `notification_period` attribute
164 on your configuration objects, Icinga 2 assumes `24x7` as time period
167 object TimePeriod "24x7" {
168 import "legacy-timeperiod"
170 display_name = "Icinga 2 24x7 TimePeriod"
172 "monday" = "00:00-24:00"
173 "tuesday" = "00:00-24:00"
174 "wednesday" = "00:00-24:00"
175 "thursday" = "00:00-24:00"
176 "friday" = "00:00-24:00"
177 "saturday" = "00:00-24:00"
178 "sunday" = "00:00-24:00"
182 If your operation staff should only be notified during workhours,
183 create a new timeperiod named `workhours` defining a work day from
186 object TimePeriod "workhours" {
187 import "legacy-timeperiod"
189 display_name = "Icinga 2 8x5 TimePeriod"
191 "monday" = "09:00-17:00"
192 "tuesday" = "09:00-17:00"
193 "wednesday" = "09:00-17:00"
194 "thursday" = "09:00-17:00"
195 "friday" = "09:00-17:00"
199 Use the `period` attribute to assign time periods to
200 `Notification` and `Dependency` objects:
202 object Notification "mail" {
203 import "generic-notification"
205 host_name = "localhost"
207 command = "mail-notification"
208 users = [ "icingaadmin" ]
212 ### <a id="timeperiods-includes-excludes"></a> Time Periods Inclusion and Exclusion
214 Sometimes it is necessary to exclude certain time ranges from
215 your default time period definitions, for example, if you don't
216 want to send out any notification during the holiday season,
217 or if you only want to allow small time windows for executed checks.
219 The [TimePeriod object](6-object-types.md#objecttype-timeperiod)
220 provides the `includes` and `excludes` attributes to solve this issue.
221 `prefer_includes` defines whether included or excluded time periods are
224 The following example defines a time period called `holidays` where
225 notifications should be supressed:
227 object TimePeriod "holidays" {
228 import "legacy-timeperiod"
231 "january 1" = "00:00-24:00" //new year's day
232 "july 4" = "00:00-24:00" //independence day
233 "december 25" = "00:00-24:00" //christmas
234 "december 31" = "18:00-24:00" //new year's eve (6pm+)
235 "2017-04-16" = "00:00-24:00" //easter 2017
236 "monday -1 may" = "00:00-24:00" //memorial day (last monday in may)
237 "monday 1 september" = "00:00-24:00" //labor day (1st monday in september)
238 "thursday 4 november" = "00:00-24:00" //thanksgiving (4th thursday in november)
242 In addition to that the time period `weekends` defines an additional
243 time window which should be excluded from notifications:
245 object TimePeriod "weekends-excluded" {
246 import "legacy-timeperiod"
249 "saturday" = "00:00-09:00,18:00-24:00"
250 "sunday" = "00:00-09:00,18:00-24:00"
254 The time period `prod-notification` defines the default time ranges
255 and adds the excluded time period names as an array.
257 object TimePeriod "prod-notification" {
258 import "legacy-timeperiod"
260 excludes = [ "holidays", "weekends-excluded" ]
263 "monday" = "00:00-24:00"
264 "tuesday" = "00:00-24:00"
265 "wednesday" = "00:00-24:00"
266 "thursday" = "00:00-24:00"
267 "friday" = "00:00-24:00"
268 "saturday" = "00:00-24:00"
269 "sunday" = "00:00-24:00"
274 ## <a id="use-functions-object-config"></a> Use Functions in Object Configuration
276 There is a limited scope where functions can be used as object attributes such as:
278 * As value for [Custom Attributes](3-monitoring-basics.md#custom-attributes-functions)
279 * Returning boolean expressions for [set_if](5-advanced-topics.md#use-functions-command-arguments-setif) inside command arguments
280 * Returning a [command](5-advanced-topics.md#use-functions-command-attribute) array inside command objects
282 The other way around you can create objects dynamically using your own global functions.
286 > Functions called inside command objects share the same global scope as runtime macros.
287 > Therefore you can access host custom attributes like `host.vars.os`, or any other
288 > object attribute from inside the function definition used for [set_if](5-advanced-topics.md#use-functions-command-arguments-setif) or [command](5-advanced-topics.md#use-functions-command-attribute).
290 Tips when implementing functions:
292 * Use [log()](19-library-reference.md#global-functions) to dump variables. You can see the output
293 inside the `icinga2.log` file depending in your log severity
294 * Use the `icinga2 console` to test basic functionality (e.g. iterating over a dictionary)
295 * Build them step-by-step. You can always refactor your code later on.
297 ### <a id="use-functions-command-arguments-setif"></a> Use Functions in Command Arguments set_if
299 The `set_if` attribute inside the command arguments definition in the
300 [CheckCommand object definition](6-object-types.md#objecttype-checkcommand) is primarily used to
301 evaluate whether the command parameter should be set or not.
303 By default you can evaluate runtime macros for their existence. If the result is not an empty
304 string, the command parameter is passed. This becomes fairly complicated when want to evaluate
305 multiple conditions and attributes.
307 The following example was found on the community support channels. The user had defined a host
308 dictionary named `compellent` with the key `disks`. This was then used inside service apply for rules.
310 object Host "dict-host" {
311 check_command = "check_compellent"
312 vars.compellent["disks"] = {
313 file = "/var/lib/check_compellent/san_disks.0.json",
318 The more significant problem was to only add the command parameter `--disk` to the plugin call
319 when the dictionary `compellent` contains the key `disks`, and omit it if not found.
321 By defining `set_if` as [abbreviated lambda function](18-language-reference.md#nullary-lambdas)
322 and evaluating the host custom attribute `compellent` containing the `disks` this problem was
325 object CheckCommand "check_compellent" {
326 import "plugin-check-command"
327 command = [ "/usr/bin/check_compellent" ]
331 var host_vars = host.vars
333 var compel = host_vars.compellent
335 compel.contains("disks")
341 This implementation uses the dictionary type method [contains](19-library-reference.md#dictionary-contains)
342 and will fail if `host.vars.compellent` is not of the type `Dictionary`.
343 Therefore you can extend the checks using the [typeof](18-language-reference.md#types) function.
345 You can test the types using the `icinga2 console`:
348 Icinga (version: v2.3.0-193-g3eb55ad)
349 <1> => srv_vars.compellent["check_a"] = { file="outfile_a.json", checks = [ "disks", "fans" ] }
351 <2> => srv_vars.compellent["check_b"] = { file="outfile_b.json", checks = [ "power", "voltages" ] }
353 <3> => typeof(srv_vars.compellent)
357 The more programmatic approach for `set_if` could look like this:
361 var srv_vars = service.vars
362 if(len(srv_vars) > 0) {
363 if (typeof(srv_vars.compellent) == Dictionary) {
364 return srv_vars.compellent.contains("disks")
366 log(LogInformationen, "checkcommand set_if", "custom attribute compellent_checks is not a dictionary, ignoring it.")
370 log(LogWarning, "checkcommand set_if", "empty custom attributes")
377 ### <a id="use-functions-command-attribute"></a> Use Functions as Command Attribute
379 This comes in handy for [NotificationCommands](6-object-types.md#objecttype-notificationcommand)
380 or [EventCommands](6-object-types.md#objecttype-eventcommand) which does not require
381 a returned checkresult including state/output.
383 The following example was taken from the community support channels. The requirement was to
384 specify a custom attribute inside the notification apply rule and decide which notification
385 script to call based on that.
387 object User "short-dummy" {
390 object UserGroup "short-dummy-group" {
391 assign where user.name == "short-dummy"
394 apply Notification "mail-admins-short" to Host {
395 import "mail-host-notification"
396 command = "mail-host-notification-test"
397 user_groups = [ "short-dummy-group" ]
399 assign where host.vars.notification.mail
402 The solution is fairly simple: The `command` attribute is implemented as function returning
403 an array required by the caller Icinga 2.
404 The local variable `mailscript` sets the default value for the notification scrip location.
405 If the notification custom attribute `short` is set, it will override the local variable `mailscript`
407 The `mailscript` variable is then used to compute the final notification command array being
410 You can omit the `log()` calls, they only help debugging.
412 object NotificationCommand "mail-host-notification-test" {
413 import "plugin-notification-command"
415 log("command as function")
416 var mailscript = "mail-host-notification-long.sh"
417 if (notification.vars.short) {
418 mailscript = "mail-host-notification-short.sh"
420 log("Running command")
423 var cmd = [ SysconfDir + "/icinga2/scripts/" + mailscript ]
424 log(LogCritical, "me", cmd)
432 ### <a id="custom-functions-as-attribute"></a> Use Custom Functions as Attribute
434 To use custom functions as attributes, the function must be defined in a
435 slightly unexpected way. The following example shows how to assign values
436 depending on group membership. All hosts in the `slow-lan` host group use 300
437 as value for `ping_wrta`, all other hosts use 100.
439 globals.group_specific_value = function(group, group_value, non_group_value) {
440 return function() use (group, group_value, non_group_value) {
441 if (group in host.groups) {
444 return non_group_value
449 apply Service "ping4" {
450 import "generic-service"
451 check_command = "ping4"
453 vars.ping_wrta = group_specific_value("slow-lan", 300, 100)
454 vars.ping_crta = group_specific_value("slow-lan", 500, 200)
459 ### <a id="use-functions-assign-where"></a> Use Functions in Assign Where Expressions
461 If a simple expression for matching a name or checking if an item
462 exists in an array or dictionary does not fit, you should consider
463 writing your own global [functions](18-language-reference.md#functions).
464 You can call them inside `assign where` and `ignore where` expressions
465 for [apply rules](3-monitoring-basics.md#using-apply-expressions) or
466 [group assignments](3-monitoring-basics.md#group-assign-intro) just like
467 any other global functions for example [match](19-library-reference.md#global-functions).
469 The following example requires the host `myprinter` being added
470 to the host group `printers-lexmark` but only if the host uses
471 a template matching the name `lexmark*`.
473 template Host "lexmark-printer-host" {
474 vars.printer_type = "Lexmark"
477 object Host "myprinter" {
478 import "generic-host"
479 import "lexmark-printer-host"
481 address = "192.168.1.1"
484 /* register a global function for the assign where call */
485 globals.check_host_templates = function(host, search) {
486 /* iterate over all host templates and check if the search matches */
487 for (tmpl in host.templates) {
488 if (match(search, tmpl)) {
493 /* nothing matched */
497 object HostGroup "printers-lexmark" {
498 display_name = "Lexmark Printers"
499 /* call the global function and pass the arguments */
500 assign where check_host_templates(host, "lexmark*")
504 Take a different more complex example: All hosts with the
505 custom attribute `vars_app` as nested dictionary should be
506 added to the host group `ABAP-app-server`. But only if the
507 `app_type` for all entries is set to `ABAP`.
509 It could read as wildcard match for nested dictionaries:
511 where host.vars.vars_app["*"].app_type == "ABAP"
513 The solution for this problem is to register a global
514 function which checks the `app_type` for all hosts
515 with the `vars_app` dictionary.
517 object Host "appserver01" {
518 check_command = "dummy"
519 vars.vars_app["ABC"] = { app_type = "ABAP" }
521 object Host "appserver02" {
522 check_command = "dummy"
523 vars.vars_app["DEF"] = { app_type = "ABAP" }
526 globals.check_app_type = function(host, type) {
527 /* ensure that other hosts without the custom attribute do not match */
528 if (typeof(host.vars.vars_app) != Dictionary) {
532 /* iterate over the vars_app dictionary */
533 for (key => val in host.vars.vars_app) {
534 /* if the value is a dictionary and if contains the app_type being the requested type */
535 if (typeof(val) == Dictionary && val.app_type == type) {
540 /* nothing matched */
544 object HostGroup "ABAP-app-server" {
545 assign where check_app_type(host, "ABAP")
548 ## <a id="access-object-attributes-at-runtime"></a> Access Object Attributes at Runtime
550 The [Object Accessor Functions](19-library-reference.md#object-accessor-functions)
551 can be used to retrieve references to other objects by name.
553 This allows you to access configuration and runtime object attributes. A detailed
554 list can be found [here](6-object-types.md#object-types).
556 Simple cluster example for accessing two host object states and calculating a virtual
557 cluster state and output:
559 object Host "cluster-host-01" {
560 check_command = "dummy"
562 vars.dummy_text = "This host is down."
565 object Host "cluster-host-02" {
566 check_command = "dummy"
568 vars.dummy_text = "This host is up."
571 object Host "cluster" {
572 check_command = "dummy"
573 vars.cluster_nodes = [ "cluster-host-01", "cluster-host-02" ]
575 vars.dummy_state = {{
578 var cluster_nodes = macro("$cluster_nodes$")
580 for (node in cluster_nodes) {
581 if (get_host(node).state > 0) {
588 if (up_count >= down_count) {
589 return 0 //same up as down -> UP
591 return 2 //something is broken
596 var output = "Cluster hosts:\n"
597 var cluster_nodes = macro("$cluster_nodes$")
599 for (node in cluster_nodes) {
600 output += node + ": " + get_host(node).last_check_result.output + "\n"
608 The following example sets time dependent thresholds for the load check based on the current
609 time of the day compared to the defined time period.
611 object TimePeriod "backup" {
612 import "legacy-timeperiod"
615 monday = "02:00-03:00"
616 tuesday = "02:00-03:00"
617 wednesday = "02:00-03:00"
618 thursday = "02:00-03:00"
619 friday = "02:00-03:00"
620 saturday = "02:00-03:00"
621 sunday = "02:00-03:00"
625 object Host "webserver-with-backup" {
626 check_command = "hostalive"
627 address = "127.0.0.1"
630 object Service "webserver-backup-load" {
631 check_command = "load"
632 host_name = "webserver-with-backup"
634 vars.load_wload1 = {{
635 if (get_time_period("backup").is_inside) {
641 vars.load_cload1 = {{
642 if (get_time_period("backup").is_inside) {
651 ## <a id="check-result-freshness"></a> Check Result Freshness
653 In Icinga 2 active check freshness is enabled by default. It is determined by the
654 `check_interval` attribute and no incoming check results in that period of time.
656 threshold = last check execution time + check interval
658 Passive check freshness is calculated from the `check_interval` attribute if set.
660 threshold = last check result time + check interval
662 If the freshness checks are invalid, a new check is executed defined by the
663 `check_command` attribute.
666 ## <a id="check-flapping"></a> Check Flapping
668 The flapping algorithm used in Icinga 2 does not store the past states but
669 calculates the flapping threshold from a single value based on counters and
670 half-life values. Icinga 2 compares the value with a single flapping threshold
671 configuration attribute named `flapping_threshold`.
673 Flapping detection can be enabled or disabled using the `enable_flapping` attribute.
676 ## <a id="volatile-services"></a> Volatile Services
678 By default all services remain in a non-volatile state. When a problem
679 occurs, the `SOFT` state applies and once `max_check_attempts` attribute
680 is reached with the check counter, a `HARD` state transition happens.
681 Notifications are only triggered by `HARD` state changes and are then
682 re-sent defined by the `interval` attribute.
684 It may be reasonable to have a volatile service which stays in a `HARD`
685 state type if the service stays in a `NOT-OK` state. That way each
686 service recheck will automatically trigger a notification unless the
687 service is acknowledged or in a scheduled downtime.