1 # Icinga 2 Troubleshooting <a id="troubleshooting"></a>
3 ## Required Information <a id="troubleshooting-information-required"></a>
5 Please ensure to provide any detail which may help reproduce and understand your issue.
6 Whether you ask on the community channels or you create an issue at [GitHub](https://github.com/Icinga), make sure
7 that others can follow your explanations. If necessary, draw a picture and attach it for
8 better illustration. This is especially helpful if you are troubleshooting a distributed
11 We've come around many community questions and compiled this list. Add your own
12 findings and details please.
14 * Describe the expected behavior in your own words.
15 * Describe the actual behavior in one or two sentences.
16 * Ensure to provide general information such as:
17 * How was Icinga 2 installed (and which repository in case) and which distribution are you using
19 * `icinga2 feature list`
21 * [Icinga Web 2](https://icinga.com/products/icinga-web-2/) version (screenshot from System - About)
22 * [Icinga Web 2 modules](https://icinga.com/products/icinga-web-2-modules/) e.g. the Icinga Director (optional)
23 * Configuration insights:
24 * Provide complete configuration snippets explaining your problem in detail
25 * Your [icinga2.conf](04-configuring-icinga-2.md#icinga2-conf) file
26 * If you run multiple Icinga 2 instances, the [zones.conf](04-configuring-icinga-2.md#zones-conf) file (or `icinga2 object list --type Endpoint` and `icinga2 object list --type Zone`) from all affected nodes.
28 * Relevant output from your main and [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) in `/var/log/icinga2`. Please add step-by-step explanations with timestamps if required.
29 * The newest Icinga 2 crash log if relevant, located in `/var/log/icinga2/crash`
31 * If the check command failed, what's the output of your manual plugin tests?
32 * In case of [debugging](21-development.md#development) Icinga 2, the full back traces and outputs
34 ## Analyze your Environment <a id="troubleshooting-analyze-environment"></a>
36 There are many components involved on a server running Icinga 2. When you
37 analyze a problem, keep in mind that basic system administration knowledge
38 is also key to identify bottlenecks and issues.
42 > [Monitor Icinga 2](08-advanced-topics.md#monitoring-icinga) and use the hints for further analysis.
44 * Analyze the system's performance and dentify bottlenecks and issues.
45 * Collect details about all applications (e.g. Icinga 2, MySQL, Apache, Graphite, Elastic, etc.).
46 * If data is exchanged via network (e.g. central MySQL cluster) ensure to monitor the bandwidth capabilities too.
47 * Add graphs and screenshots to your issue description
49 Install tools which help you to do so. Opinions differ, let us know if you have any additions here!
51 ### Analyse your Linux/Unix Environment <a id="troubleshooting-analyze-environment-linux"></a>
53 [htop](https://hisham.hm/htop/) is a better replacement for `top` and helps to analyze processes
61 If you are for example experiencing performance issues, open `htop` and take a screenshot.
62 Add it to your question and/or bug report.
64 Analyse disk I/O performance in Grafana, take a screenshot and obfuscate any sensitive details.
65 Attach it when posting a question to the community channels.
67 The [sysstat](https://github.com/sysstat/sysstat) package provides a number of tools to
68 analyze the performance on Linux. On FreeBSD you could use `systat` for example.
72 apt-get install sysstat
75 Example for `vmstat` (summary of memory, processes, etc.):
80 // print timestamps, format in MB, stats every 1 second, 5 times
98 `sysstat` also provides the `iostat` binary. On FreeBSD you could use `systat` for example.
100 If you are missing checks and metrics found in your analysis, add them to your monitoring!
102 ### Analyze your Windows Environment <a id="troubleshooting-analyze-environment-windows"></a>
104 A good tip for Windows are the tools found inside the [Sysinternals Suite](https://technet.microsoft.com/en-us/sysinternals/bb842062.aspx).
106 You can also start `perfmon` and analyze specific performance counters.
107 Keep notes which could be important for your monitoring, and add service
110 ## Enable Debug Output <a id="troubleshooting-enable-debug-output"></a>
112 ### Enable Debug Output on Linux/Unix <a id="troubleshooting-enable-debug-output-linux"></a>
114 Enable the `debuglog` feature:
117 # icinga2 feature enable debuglog
118 # service icinga2 restart
121 The debug log file can be found in `/var/log/icinga2/debug.log`.
123 Alternatively you may run Icinga 2 in the foreground with debugging enabled. Specify the console
124 log severity as an additional parameter argument to `-x`.
127 # /usr/sbin/icinga2 daemon -x notice
130 The [log severity](09-object-types.md#objecttype-filelogger) can be one of `critical`, `warning`, `information`, `notice`
133 ### Enable Debug Output on Windows <a id="troubleshooting-enable-debug-output-windows"></a>
135 Open a command prompt with administrative privileges and enable the debug log feature.
138 C:> icinga2.exe feature enable debuglog
141 Ensure that the Icinga 2 service already writes the main log into `C:\ProgramData\icinga2\var\log\icinga2`.
142 Restart the Icinga 2 service and open the newly created `debug.log` file.
146 C:> net start icinga2
149 ## Configuration Troubleshooting <a id="troubleshooting-configuration"></a>
151 ### List Configuration Objects <a id="troubleshooting-list-configuration-objects"></a>
153 The `icinga2 object list` CLI command can be used to list all configuration objects and their
154 attributes. The tool also shows where each of the attributes was modified.
158 > Use the Icinga 2 API to access [config objects at runtime](12-icinga2-api.md#icinga2-api-config-objects) directly.
160 That way you can also identify which objects have been created from your [apply rules](17-language-reference.md#apply).
163 # icinga2 object list
165 Object 'localhost!ssh' of type 'Service':
166 * __name = 'localhost!ssh'
167 * check_command = 'ssh'
168 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 5:3-5:23
169 * check_interval = 60
170 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 24:3-24:21
171 * host_name = 'localhost'
172 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 4:3-4:25
173 * max_check_attempts = 3
174 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 23:3-23:24
176 * retry_interval = 30
177 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 25:3-25:22
178 * templates = [ 'ssh', 'generic-service' ]
179 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 1:0-7:1
180 % += modified in '/etc/icinga2/conf.d/templates.conf', lines 22:1-26:1
183 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
185 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
190 You can also filter by name and type:
193 # icinga2 object list --name *ssh* --type Service
194 Object 'localhost!ssh' of type 'Service':
195 * __name = 'localhost!ssh'
196 * check_command = 'ssh'
197 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 5:3-5:23
198 * check_interval = 60
199 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 24:3-24:21
200 * host_name = 'localhost'
201 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 4:3-4:25
202 * max_check_attempts = 3
203 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 23:3-23:24
205 * retry_interval = 30
206 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 25:3-25:22
207 * templates = [ 'ssh', 'generic-service' ]
208 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 1:0-7:1
209 % += modified in '/etc/icinga2/conf.d/templates.conf', lines 22:1-26:1
212 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
214 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
216 Found 1 Service objects.
218 [2014-10-15 14:27:19 +0200] information/cli: Parsed 175 objects.
221 Runtime modifications via the [REST API](12-icinga2-api.md#icinga2-api-config-objects)
222 are not immediately updated. Furthermore there is a known issue with
223 [group assign expressions](17-language-reference.md#group-assign) which are not reflected in the host object output.
224 You need to restart Icinga 2 in order to update the `icinga2.debug` cache file.
226 ### Apply rules do not match <a id="apply-rules-do-not-match"></a>
228 You can analyze apply rules and matching objects by using the [script debugger](20-script-debugger.md#script-debugger).
230 ### Where are the check command definitions? <a id="check-command-definitions"></a>
232 Icinga 2 features a number of built-in [check command definitions](10-icinga-template-library.md#icinga-template-library) which are
240 in the [icinga2.conf](04-configuring-icinga-2.md#icinga2-conf) configuration file. These files are not considered
241 configuration files and will be overridden on upgrade, so please send modifications as proposed patches upstream.
242 The default include path is set to `/usr/share/icinga2/includes` with the constant `IncludeConfDir`.
244 You should add your own command definitions to a new file in `conf.d/` called `commands.conf`
247 ### Configuration is ignored <a id="configuration-ignored"></a>
249 * Make sure that the line(s) are not [commented out](17-language-reference.md#comments) (starting with `//` or `#`, or
250 encapsulated by `/* ... */`).
251 * Is the configuration file included in [icinga2.conf](04-configuring-icinga-2.md#icinga2-conf)?
253 Run the [configuration validation](11-cli-commands.md#config-validation) and add `notice` as log severity.
254 Search for the file which should be included i.e. using the `grep` CLI command.
257 # icinga2 daemon -C -x notice | grep command
260 ### Configuration attributes are inherited from <a id="configuration-attribute-inheritance"></a>
262 Icinga 2 allows you to import templates using the [import](17-language-reference.md#template-imports) keyword. If these templates
263 contain additional attributes, your objects will automatically inherit them. You can override
264 or modify these attributes in the current object.
266 The [object list](15-troubleshooting.md#troubleshooting-list-configuration-objects) CLI command allows you to verify the attribute origin.
268 ### Configuration Value with Single Dollar Sign <a id="configuration-value-dollar-sign"></a>
270 In case your configuration validation fails with a missing closing dollar sign error message, you
271 did not properly escape the single dollar sign preventing its usage as [runtime macro](03-monitoring-basics.md#runtime-macros).
274 critical/config: Error: Validation failed for Object 'ping4' (Type: 'Service') at /etc/icinga2/zones.d/global-templates/windows.conf:24: Closing $ not found in macro format string 'top-syntax=${list}'.
277 Correct the custom attribute value to
280 "top-syntax=$${list}"
284 ## Checks Troubleshooting <a id="troubleshooting-checks"></a>
286 ### Executed Command for Checks <a id="checks-executed-command"></a>
288 * Use the Icinga 2 API to [query](12-icinga2-api.md#icinga2-api-config-objects-query) host/service objects
289 for their check result containing the executed shell command.
290 * Use the Icinga 2 [console cli command](11-cli-commands.md#cli-command-console)
291 to fetch the checkable object, its check result and the executed shell command.
292 * Alternatively enable the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) and look for the executed command.
294 Example for a service object query using a [regex match](18-library-reference.md#global-functions-regex)
298 $ curl -k -s -u root:icinga -H 'Accept: application/json' -H 'X-HTTP-Method-Override: GET' -X POST 'https://localhost:5665/v1/objects/services' \
299 -d '{ "filter": "regex(pattern, service.name)", "filter_vars": { "pattern": "^http" }, "attrs": [ "__name", "last_check_result" ], "pretty": true }'
304 "__name": "example.localdomain!http",
305 "last_check_result": {
307 "check_source": "example.localdomain",
309 "/usr/local/sbin/check_http",
322 "name": "example.localdomain!http",
329 Example for using the `icinga2 console` CLI command evaluation functionality:
332 $ ICINGA2_API_PASSWORD=icinga icinga2 console --connect 'https://root@localhost:5665/' \
333 --eval 'get_service("example.localdomain", "http").last_check_result.command' | python -m json.tool
335 "/usr/local/sbin/check_http",
343 Example for searching the debug log:
346 # icinga2 feature enable debuglog
347 # systemctl restart icinga2
348 # tail -f /var/log/icinga2/debug.log | grep "notice/Process"
352 ### Checks are not executed <a id="checks-not-executed"></a>
354 * Check the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) to see if the check command gets executed.
355 * Verify that failed depedencies do not prevent command execution.
356 * Make sure that the plugin is executable by the Icinga 2 user (run a manual test).
357 * Make sure the [checker](11-cli-commands.md#enable-features) feature is enabled.
358 * Use the Icinga 2 API [event streams](12-icinga2-api.md#icinga2-api-event-streams) to receive live check result streams.
363 # sudo -u icinga /usr/lib/nagios/plugins/check_ping -4 -H 127.0.0.1 -c 5000,100% -w 3000,80%
365 # icinga2 feature enable checker
366 The feature 'checker' is already enabled.
369 Fetch all check result events matching the `event.service` name `random`:
372 $ curl -k -s -u root:icinga -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/events?queue=debugchecks&types=CheckResult&filter=match%28%22random*%22,event.service%29'
376 ### Analyze Check Source <a id="checks-check-source"></a>
378 Sometimes checks are not executed on the remote host, but on the master and so on.
379 This could lead into unwanted results or NOT-OK states.
381 The `check_source` attribute is the best indication where a check command
382 was actually executed. This could be a satellite with synced configuration
383 or a client as remote command bridge -- both will return the check source
384 as where the plugin is called.
386 Example for retrieving the check source from all `disk` services using a
387 [regex match](18-library-reference.md#global-functions-regex) on the name:
390 $ curl -k -s -u root:icinga -H 'Accept: application/json' -H 'X-HTTP-Method-Override: GET' -X POST 'https://localhost:5665/v1/objects/services' \
391 -d '{ "filter": "regex(pattern, service.name)", "filter_vars": { "pattern": "^disk" }, "attrs": [ "__name", "last_check_result" ], "pretty": true }'
396 "__name": "icinga2-client1.localdomain!disk",
397 "last_check_result": {
399 "check_source": "icinga2-client1.localdomain",
407 "name": "icinga2-client1.localdomain!disk",
414 Example for using the `icinga2 console` CLI command evaluation functionality:
417 $ ICINGA2_API_PASSWORD=icinga icinga2 console --connect 'https://root@localhost:5665/' \
418 --eval 'get_service("icinga2-client1.localdomain", "disk").last_check_result.check_source' | python -m json.tool
420 "icinga2-client1.localdomain"
424 ### NSClient++ Check Errors with nscp-local <a id="nsclient-check-errors-nscp-local"></a>
426 The [nscp-local](10-icinga-template-library.md#nscp-check-local) CheckCommand object definitions call the local `nscp.exe` command.
427 If a Windows client service check fails to find the `nscp.exe` command, the log output would look like this:
430 Command ".\nscp.exe" "client" "-a" "drive=d" "-a" "show-all" "-b" "-q" "check_drivesize" failed to execute: 2, "The system cannot find the file specified."
437 scp.exe" "client" "-a" "drive=d" "-a" "show-all" "-b" "-q" "check_drivesize" failed to execute: 2, "The system cannot find the file specified."
440 The above actually prints `.\\nscp.exe` where the escaped `\n` character gets interpreted as new line.
442 Both errors lead to the assumption that the `NscpPath` constant is empty or set to a `.` character.
443 This could mean the following:
445 * The command is **not executed on the Windows client**. Check the [check_source](15-troubleshooting.md#checks-check-source) attribute from the check result.
446 * You are using an outdated NSClient++ version (0.3.x or 0.4.x) which is not compatible with Icinga 2.
447 * You are using a custom NSClient++ installer which does not register the correct GUID for NSClient++
449 More troubleshooting:
451 Retrieve the `NscpPath` constant on your Windows client:
454 C:\Program Files\ICINGA2\sbin\icinga2.exe variable get NscpPath
457 If the variable is returned empty, manually test how Icinga 2 would resolve
458 its path (this can be found inside the ITL):
461 C:\Program Files\ICINGA2\sbin\icinga2.exe console --eval "dirname(msi_get_component_path(\"{5C45463A-4AE9-4325-96DB-6E239C034F93}\"))"
464 If this command does not return anything, NSClient++ is not properly installed.
465 Verify that inside the `Programs and Features` (`appwiz.cpl`) control panel.
467 You can run the bundled NSClient++ installer from the Icinga 2 Windows package.
468 The msi package is located in `C:\Program Files\ICINGA2\sbin`.
470 The bundled NSClient++ version has properly been tested with Icinga 2. Keep that
471 in mind when using a different package.
474 ### Check Thresholds Not Applied <a id="check-thresholds-not-applied"></a>
476 This could happen with [clients as command endpoint execution](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint).
478 If you have for example a client host `icinga2-client1.localdomain`
479 and a service `disk` check defined on the master, the warning and
480 critical thresholds are sometimes to applied and unwanted notification
483 This happens because the client itself includes a host object with
484 its `NodeName` and a basic set of checks in the [conf.d](04-configuring-icinga-2.md#conf-d)
485 directory, i.e. `disk` with the default thresholds.
487 Clients which have the `checker` feature enabled will attempt
488 to execute checks for local services and send their results
491 If you now have the same host and service objects on the
492 master you will receive wrong check results from the client.
496 * Disable the `checker` feature on clients: `icinga2 feature disable checker`.
497 * Remove the inclusion of [conf.d](04-configuring-icinga-2.md#conf-d) as suggested in the [client setup docs](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint).
499 ### Check Fork Errors <a id="check-fork-errors"></a>
501 Newer versions of systemd on Linux limit spawned processes for
504 * v227 introduces the `TasksMax` setting to units which allows to specify the spawned process limit.
505 * v228 adds `DefaultTasksMax` in the global `systemd-system.conf` with a default setting of 512 processes.
506 * v231 changes the default value to 15%
508 This can cause problems with Icinga 2 in large environments with many
509 commands executed in parallel starting with systemd v228. Some distributions
510 also may have changed the defaults.
512 The error message could look like this:
515 2017-01-12T11:55:40.742685+01:00 icinga2-master1 kernel: [65567.582895] cgroup: fork rejected by pids controller in /system.slice/icinga2.service
518 In order to solve the problem, increase the value for `DefaultTasksMax`
519 or set it to `infinity`.
522 mkdir /etc/systemd/system/icinga2.service.d
523 cat >/etc/systemd/system/icinga2.service.d/limits.conf <<EOF
525 DefaultTasksMax=infinity
528 systemctl daemon-reload
529 systemctl restart icinga2
532 An example is available inside the GitHub repository in [etc/initsystem](https://github.com/Icinga/icinga2/tree/master/etc/initsystem).
536 * [Fork limit for cgroups](https://lwn.net/Articles/663873/)
537 * [systemd changelog](https://github.com/systemd/systemd/blob/master/NEWS)
538 * [Icinga 2 upstream issue](https://github.com/Icinga/icinga2/issues/5611)
539 * [systemd upstream discussion](https://github.com/systemd/systemd/issues/3211)
541 ### Systemd Watchdog <a id="check-systemd-watchdog"></a>
543 Usually Icinga 2 is a mission critical part of infrastructure and should be
544 online at all times. In case of a recoverable crash (e.g. OOM) you may want to
545 restart Icinga 2 automatically. With systemd it is as easy as overriding some
546 settings of the Icinga 2 systemd service by creating
547 `/etc/systemd/system/icinga2.service.d/override.conf` with the following
554 StartLimitInterval=10
558 Using the watchdog can also help with monitoring Icinga 2, to activate and use it add the following to the override:
564 This way systemd will kill Icinga 2 if does not notify for over 30 seconds, a timout of less than 10 seconds is not
565 recommended. When the watchdog is activated, `Restart=` can be set to `watchdog` to restart Icinga 2 in the case of a
568 Run `systemctl daemon-reload && systemctl restart icinga2` to apply the changes.
569 Now systemd will always try to restart Icinga 2 (except if you run
570 `systemctl stop icinga2`). After three failures in ten seconds it will stop
571 trying because you probably have a problem that requires manual intervention.
573 ### Late Check Results <a id="late-check-results"></a>
575 [Icinga Web 2](https://icinga.com/products/icinga-web-2/) provides
576 a dashboard overview for `overdue checks`.
578 The REST API provides the [status](12-icinga2-api.md#icinga2-api-status) URL endpoint with some generic metrics
579 on Icinga and its features.
582 # curl -k -s -u root:icinga 'https://localhost:5665/v1/status?pretty=1' | less
585 You can also calculate late check results via the REST API:
587 * Fetch the `last_check` timestamp from each object
588 * Compare the timestamp with the current time and add `check_interval` multiple times (change it to see which results are really late, like five times check_interval)
590 You can use the [icinga2 console](11-cli-commands.md#cli-command-console) to connect to the instance, fetch all data
591 and calculate the differences. More infos can be found in [this blogpost](https://icinga.com/2016/08/11/analyse-icinga-2-problems-using-the-console-api/).
594 # ICINGA2_API_USERNAME=root ICINGA2_API_PASSWORD=icinga icinga2 console --connect 'https://localhost:5665/'
596 <1> => var res = []; for (s in get_objects(Service).filter(s => s.last_check < get_time() - 2 * s.check_interval)) { res.add([s.__name, DateTime(s.last_check).to_string()]) }; res
598 [ [ "10807-host!10807-service", "2016-06-10 15:54:55 +0200" ], [ "mbmif.int.netways.de!disk /", "2016-01-26 16:32:29 +0100" ] ]
601 Or if you are just interested in numbers, call [len](18-library-reference.md#array-len) on the result array `res`:
604 <2> => var res = []; for (s in get_objects(Service).filter(s => s.last_check < get_time() - 2 * s.check_interval)) { res.add([s.__name, DateTime(s.last_check).to_string()]) }; res.len()
609 If you need to analyze that problem multiple times, just add the current formatted timestamp
610 and repeat the commands.
613 <23> => DateTime(get_time()).to_string()
615 "2017-04-04 16:09:39 +0200"
617 <24> => var res = []; for (s in get_objects(Service).filter(s => s.last_check < get_time() - 2 * s.check_interval)) { res.add([s.__name, DateTime(s.last_check).to_string()]) }; res.len()
622 More details about the Icinga 2 DSL and its possibilities can be
623 found in the [language](17-language-reference.md#language-reference) and [library](18-library-reference.md#library-reference) reference chapters.
625 ### Late Check Results in Distributed Environments <a id="late-check-results-distributed"></a>
627 When it comes to a distributed HA setup, each node is responsible for a load-balanced amount of checks.
628 Host and Service objects provide the attribute `paused`. If this is set to `false`, the current node
629 actively attempts to schedule and execute checks. Otherwise the node does not feel responsible.
632 <3> => var res = {}; for (s in get_objects(Service).filter(s => s.last_check < get_time() - 2 * s.check_interval)) { res[s.paused] += 1 }; res
639 You may ask why this analysis is important? Fair enough - if the numbers are not inverted in a HA zone
640 with two members, this may give a hint that the cluster nodes are in a split-brain scenario, or you've
641 found a bug in the cluster.
644 If you are running a cluster setup where the master/satellite executes checks on the client via
645 [top down command endpoint](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint) mode,
646 you might want to know which zones are affected.
648 This analysis assumes that clients which are not connected, have the string `connected` in their
649 service check result output and their state is `UNKNOWN`.
652 <4> => var res = {}; for (s in get_objects(Service)) { if (s.state==3) { if (match("*connected*", s.last_check_result.output)) { res[s.zone] += [s.host_name] } } }; for (k => v in res) { res[k] = len(v.unique()) }; res
661 The result set shows the configured zones and their affected hosts in a unique list. The output also just prints the numbers
662 but you can adjust this by omitting the `len()` call inside the for loop.
664 ## Notifications Troubleshooting <a id="troubleshooting-notifications"></a>
666 ### Notifications are not sent <a id="notifications-not-sent"></a>
668 * Check the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) to see if a notification is triggered.
669 * If yes, verify that all conditions are satisfied.
670 * Are any errors on the notification command execution logged?
672 Please ensure to add these details with your own description
673 to any question or issue posted to the community channels.
675 Verify the following configuration:
677 * Is the host/service `enable_notifications` attribute set, and if so, to which value?
678 * Do the [notification](09-object-types.md#objecttype-notification) attributes `states`, `types`, `period` match the notification conditions?
679 * Do the [user](09-object-types.md#objecttype-user) attributes `states`, `types`, `period` match the notification conditions?
680 * Are there any notification `begin` and `end` times configured?
681 * Make sure the [notification](11-cli-commands.md#enable-features) feature is enabled.
682 * Does the referenced NotificationCommand work when executed as Icinga user on the shell?
684 If notifications are to be sent via mail, make sure that the mail program specified inside the
685 [NotificationCommand object](09-object-types.md#objecttype-notificationcommand) exists.
686 The name and location depends on the distribution so the preconfigured setting might have to be
687 changed on your system.
693 # icinga2 feature enable notification
694 The feature 'notification' is already enabled.
698 # icinga2 feature enable debuglog
699 # systemctl restart icinga2
701 # grep Notification /var/log/icinga2/debug.log > /root/analyze_notification_problem.log
704 You can use the Icinga 2 API [event streams](12-icinga2-api.md#icinga2-api-event-streams) to receive live notification streams:
707 $ curl -k -s -u root:icinga -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/events?queue=debugnotifications&types=Notification'
710 ## Feature Troubleshooting <a id="troubleshooting-features"></a>
712 ### Feature is not working <a id="feature-not-working"></a>
714 * Make sure that the feature configuration is enabled by symlinking from `features-available/`
715 to `features-enabled` and that the latter is included in [icinga2.conf](04-configuring-icinga-2.md#icinga2-conf).
716 * Are the feature attributes set correctly according to the documentation?
717 * Any errors on the logs?
719 Look up the [object type](09-object-types.md#object-types) for the required feature and verify it is enabled:
722 # icinga2 object list --type <feature object type>
725 Example for the `graphite` feature:
728 # icinga2 object list --type GraphiteWriter
731 Look into the log and check whether the feature logs anything specific for this matter.
734 grep GraphiteWriter /var/log/icinga2/icinga2.log
737 ## REST API Troubleshooting <a id="troubleshooting-api"></a>
739 In order to analyse errors on API requests, you can explicitly enable the [verbose parameter](12-icinga2-api.md#icinga2-api-parameters-global).
742 $ curl -k -s -u root:icinga -H 'Accept: application/json' -X DELETE 'https://localhost:5665/v1/objects/hosts/example-cmdb?pretty=1&verbose=1'
744 "diagnostic_information": "Error: Object does not exist.\n\n ....",
746 "status": "No objects found."
750 ### REST API Troubleshooting: No Objects Found <a id="troubleshooting-api-no-objects-found"></a>
752 Please note that the `404` status with no objects being found can also originate
753 from missing or too strict object permissions for the authenticated user.
755 This is a security feature to disable object name guessing. If this would not be the
756 case, restricted users would be able to get a list of names of your objects just by
757 trying every character combination.
759 In order to analyse and fix the problem, please check the following:
761 - use an administrative account with full permissions to check whether the objects are actually there.
762 - verify the permissions on the affected ApiUser object and fix them.
764 ### Missing Runtime Objects (Hosts, Downtimes, etc.) <a id="troubleshooting-api-missing-runtime-objects"></a>
766 Runtime objects consume the internal config packages shared with
767 the REST API config packages. Each host, downtime, comment, service, etc. created
768 via the REST API is stored in the `_api` package.
770 This includes downtimes and comments, which where sometimes stored in the wrong
771 directory path, because the active-stage file was empty/truncated/unreadable at
777 /var/lib/icinga2/api/packages/_api//conf.d/downtimes/1234-5678-9012-3456.conf
783 /var/lib/icinga2/api/packages/_api/dbe0bef8-c72c-4cc9-9779-da7c4527c5b2/conf.d/downtimes/1234-5678-9012-3456.conf
786 At creation time, the object lives in memory but its storage is broken. Upon restart,
787 it is missing and e.g. a missing downtime will re-enable unwanted notifications.
789 `abcd-ef12-3456-7890` is the active stage name which wasn't correctly
790 read by the Icinga daemon. This information is stored in `/var/lib/icinga2/api/packages/_api/active-stage`.
792 2.11 now limits the direct active-stage file access (this is hidden from the user),
793 and caches active stages for packages in-memory.
795 It also tries to repair the broken package, and lots a new message:
798 systemctl restart icinga2
800 tail -f /var/log/icinga2/icinga2.log
802 [2019-05-10 12:27:15 +0200] information/ConfigObjectUtility: Repairing config package '_api' with stage 'dbe0bef8-c72c-4cc9-9779-da7c4527c5b2'.
805 If this does not happen, you can manually fixthe broken config package, and mark a deployed stage as active
806 again, carefully do the following steps with creating a backup before:
808 Navigate into the API package prefix.
811 cd /var/lib/icinga2/api/packages
814 Change into the broken package directory and list all directories and files
815 ordered by latest changes.
821 drwx------ 4 michi wheel 128B Mar 27 14:39 ..
822 -rw-r--r-- 1 michi wheel 25B Mar 27 14:39 include.conf
823 -rw-r--r-- 1 michi wheel 405B Mar 27 14:39 active.conf
824 drwx------ 7 michi wheel 224B Mar 27 15:01 dbe0bef8-c72c-4cc9-9779-da7c4527c5b2
825 drwx------ 5 michi wheel 160B Apr 26 12:47 .
828 As you can see, the `active-stage` file is missing. When it is there, verify that its content
829 is set to the stage directory as follows.
831 If you have more than one stage directory here, pick the latest modified
832 directory. Copy the directory name `abcd-ef12-3456-7890` and
833 add it into a new file `active-stage`. This can be done like this:
836 echo "dbe0bef8-c72c-4cc9-9779-da7c4527c5b2" > active-stage
839 `active.conf` needs to have the correct active stage too, add it again
840 like this. Note: This is deep down in the code, use with care!
843 sed -i 's/ActiveStages\["_api"\].*/ActiveStages\["_api"\] = "dbe0bef8-c72c-4cc9-9779-da7c4527c5b2"/g' /var/lib/icinga2/api/packages/_api/active.conf
849 systemctl restart icinga2
855 > The internal `_api` config package structure may change in the future. Do not modify
856 > things in there manually or with scripts unless guided here or asked by a developer.
859 ## Certificate Troubleshooting <a id="troubleshooting-certificate"></a>
861 ### Certificate Verification <a id="troubleshooting-certificate-verification"></a>
863 If the TLS handshake fails when a client connects to the cluster or the REST API,
864 ensure to verify the used certificates.
866 Print the CA and client certificate and ensure that the following attributes are set:
869 * Serial number is a hex-encoded string.
870 * Issuer should be your certificate authority (defaults to `Icinga CA` for all CLI commands).
871 * Validity, meaning to say the certificate is not expired.
872 * Subject with the common name (CN) matches the client endpoint name and its FQDN.
873 * v3 extensions must set the basic constraint for `CA:TRUE` (ca.crt) or `CA:FALSE` (client certificate).
874 * Subject Alternative Name is set to a proper DNS name (required for REST API and browsers).
878 # cd /var/lib/icinga2/certs/
884 # openssl x509 -in ca.crt -text
889 Serial Number: 1 (0x1)
890 Signature Algorithm: sha256WithRSAEncryption
893 Not Before: Feb 23 14:45:32 2016 GMT
894 Not After : Feb 19 14:45:32 2031 GMT
895 Subject: CN=Icinga CA
896 Subject Public Key Info:
897 Public Key Algorithm: rsaEncryption
898 Public-Key: (4096 bit)
901 Exponent: 65537 (0x10001)
903 X509v3 Basic Constraints: critical
905 Signature Algorithm: sha256WithRSAEncryption
909 Client public certificate:
912 # openssl x509 -in icinga2-client1.localdomain.crt -text
918 86:47:44:65:49:c6:65:6b:5e:6d:4f:a5:fe:6c:76:05:0b:1a:cf:34
919 Signature Algorithm: sha256WithRSAEncryption
922 Not Before: Aug 20 16:20:05 2016 GMT
923 Not After : Aug 17 16:20:05 2031 GMT
924 Subject: CN=icinga2-client1.localdomain
925 Subject Public Key Info:
926 Public Key Algorithm: rsaEncryption
927 Public-Key: (4096 bit)
930 Exponent: 65537 (0x10001)
932 X509v3 Basic Constraints: critical
934 X509v3 Subject Alternative Name:
935 DNS:icinga2-client1.localdomain
936 Signature Algorithm: sha256WithRSAEncryption
940 Make sure to verify the client's certificate and its received `ca.crt` in `/var/lib/icinga2/certs` and ensure that
941 both instances are signed by the **same CA**.
944 # openssl verify -verbose -CAfile /var/lib/icinga2/certs/ca.crt /var/lib/icinga2/certs/icinga2-master1.localdomain.crt
945 icinga2-master1.localdomain.crt: OK
947 # openssl verify -verbose -CAfile /var/lib/icinga2/certs/ca.crt /var/lib/icinga2/certs/icinga2-client1.localdomain.crt
948 icinga2-client1.localdomain.crt: OK
951 Fetch the `ca.crt` file from the client node and compare it to your master's `ca.crt` file:
954 # scp icinga2-client1:/var/lib/icinga2/certs/ca.crt test-client-ca.crt
955 # diff -ur /var/lib/icinga2/certs/ca.crt test-client-ca.crt
959 ### Certificate Signing <a id="troubleshooting-certificate-signing"></a>
963 ### Certificate Problems with OpenSSL 1.1.0 <a id="troubleshooting-certificate-openssl-1-1-0"></a>
965 Users have reported problems with SSL certificates inside a distributed monitoring setup when they
967 * updated their Icinga 2 package to 2.7.0 on Windows or
968 * upgraded their distribution which included an update to OpenSSL 1.1.0.
970 Example during startup on a Windows client:
973 critical/SSL: Error loading and verifying locations in ca key file 'C:\ProgramData\icinga2\etc/icinga2/pki/ca.crt': 219029726, "error:0D0E20DE:asn1 encoding routines:c2i_ibuf:illegal zero content"
974 critical/config: Error: Cannot make SSL context for cert path: 'C:\ProgramData\icinga2\etc/icinga2/pki/client.crt' key path: 'C:\ProgramData\icinga2\etc/icinga2/pki/client.key' ca path: 'C:\ProgramData\icinga2\etc/icinga2/pki/ca.crt'.
977 A technical analysis and solution for re-creating the public CA certificate is
978 available in [this advisory](https://icinga.com/2017/08/30/advisory-for-ssl-problems-with-leading-zeros-on-openssl-1-1-0/).
981 ## Cluster and Clients Troubleshooting <a id="troubleshooting-cluster"></a>
983 This applies to any Icinga 2 node in a [distributed monitoring setup](06-distributed-monitoring.md#distributed-monitoring-scenarios).
985 You should configure the [cluster health checks](06-distributed-monitoring.md#distributed-monitoring-health-checks) if you haven't
990 > Some problems just exist due to wrong file permissions or applied packet filters. Make
991 > sure to check these in the first place.
993 ### Cluster Troubleshooting Connection Errors <a id="troubleshooting-cluster-connection-errors"></a>
995 General connection errors could be one of the following problems:
997 * Incorrect network configuration
999 * Firewall rules preventing traffic
1001 Use tools like `netstat`, `tcpdump`, `nmap`, etc. to make sure that the cluster communication
1002 works (default port is `5665`).
1005 # tcpdump -n port 5665 -i any
1007 # netstat -tulpen | grep icinga
1009 # nmap icinga2-client1.localdomain
1012 ### Cluster Troubleshooting SSL Errors <a id="troubleshooting-cluster-ssl-errors"></a>
1014 If the cluster communication fails with SSL error messages, make sure to check
1017 * File permissions on the SSL certificate files
1018 * Does the used CA match for all cluster endpoints?
1019 * Verify the `Issuer` being your trusted CA
1020 * Verify the `Subject` containing your endpoint's common name (CN)
1021 * Check the validity of the certificate itself
1023 Try to manually connect from `icinga2-client1.localdomain` to the master node `icinga2-master1.localdomain`:
1026 # openssl s_client -CAfile /var/lib/icinga2/certs/ca.crt -cert /var/lib/icinga2/certs/icinga2-client1.localdomain.crt -key /var/lib/icinga2/certs/icinga2-client1.localdomain.key -connect icinga2-master1.localdomain:5665
1033 If the connection attempt fails or your CA does not match, [verify the certificates](15-troubleshooting.md#troubleshooting-certificate-verification).
1036 #### Cluster Troubleshooting Unauthenticated Clients <a id="troubleshooting-cluster-unauthenticated-clients"></a>
1038 Unauthenticated nodes are able to connect. This is required for client setups.
1043 [2015-07-13 18:29:25 +0200] information/ApiListener: New client connection for identity 'icinga2-client1.localdomain' (unauthenticated)
1046 Client as command execution bridge:
1049 [2015-07-13 18:29:26 +1000] notice/ClusterEvents: Discarding 'execute command' message from 'icinga2-master1.localdomain': Invalid endpoint origin (client not allowed).
1052 If these messages do not go away, make sure to [verify the master and client certificates](15-troubleshooting.md#troubleshooting-certificate-verification).
1054 ### Cluster Troubleshooting Message Errors <a id="troubleshooting-cluster-message-errors"></a>
1056 When the network connection is broken or gone, the Icinga 2 instances will be disconnected.
1057 If the connection can't be re-established between endpoints in the same HA zone,
1058 they remain in a Split-Brain-mode and history may differ.
1060 Although the Icinga 2 cluster protocol stores historical events in a [replay log](15-troubleshooting.md#troubleshooting-cluster-replay-log)
1061 for later synchronisation, you should make sure to check why the network connection failed.
1063 Ensure to setup [cluster health checks](06-distributed-monitoring.md#distributed-monitoring-health-checks)
1064 to monitor all endpoints and zones connectivity.
1066 ### Cluster Troubleshooting Command Endpoint Errors <a id="troubleshooting-cluster-command-endpoint-errors"></a>
1068 Command endpoints can be used [for clients](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint)
1069 as well as inside an [High-Availability cluster](06-distributed-monitoring.md#distributed-monitoring-scenarios).
1071 There is no cli command for manually executing the check, but you can verify
1072 the following (e.g. by invoking a forced check from the web interface):
1074 * `/var/log/icinga2/icinga2.log` contains connection and execution errors.
1075 * The ApiListener is not enabled to [accept commands](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint).
1076 * `CheckCommand` definition not found on the remote client.
1077 * Referenced check plugin not found on the remote client.
1078 * Runtime warnings and errors, e.g. unresolved runtime macros or configuration problems.
1079 * Specific error messages are also populated into `UNKNOWN` check results including a detailed error message in their output.
1080 * Verify the `check_source` object attribute. This is populated by the node executing the check.
1081 * More verbose logs are found inside the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output).
1083 * Use the Icinga 2 API [event streams](12-icinga2-api.md#icinga2-api-event-streams) to receive live check result streams.
1085 Fetch all check result events matching the `event.service` name `remote-client`:
1088 $ curl -k -s -u root:icinga -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/events?queue=debugcommandendpoint&types=CheckResult&filter=match%28%22remote-client*%22,event.service%29'
1093 ### Cluster Troubleshooting Config Sync <a id="troubleshooting-cluster-config-sync"></a>
1095 If the cluster zones do not sync their configuration, make sure to check the following:
1097 * Within a config master zone, only one configuration master is allowed to have its config in `/etc/icinga2/zones.d`.
1098 ** The master syncs the configuration to `/var/lib/icinga2/api/zones/` during startup and only syncs valid configuration to the other nodes.
1099 ** The other nodes receive the configuration into `/var/lib/icinga2/api/zones/`.
1100 * The `icinga2.log` log file in `/var/log/icinga2` will indicate whether this ApiListener
1101 [accepts config](06-distributed-monitoring.md#distributed-monitoring-top-down-config-sync), or not.
1103 Verify the object's [version](09-object-types.md#object-types) attribute on all nodes to
1104 check whether the config update and reload was successful or not.
1107 ### Cluster Troubleshooting Overdue Check Results <a id="troubleshooting-cluster-check-results"></a>
1109 If your master does not receive check results (or any other events) from the child zones
1110 (satellite, clients, etc.), make sure to check whether the client sending in events
1111 is allowed to do so.
1115 > General troubleshooting hints on late check results are documented [here](15-troubleshooting.md#late-check-results).
1117 The [distributed monitoring conventions](06-distributed-monitoring.md#distributed-monitoring-conventions)
1118 apply. So, if there's a mismatch between your client node's endpoint name and its provided
1119 certificate's CN, the master will deny all events.
1123 > [Icinga Web 2](02-getting-started.md#setting-up-icingaweb2) provides a dashboard view
1124 > for overdue check results.
1126 Enable the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) on the master
1127 for more verbose insights.
1129 If the client cannot authenticate, it's a more general [problem](15-troubleshooting.md#troubleshooting-cluster-unauthenticated-clients).
1131 The client's endpoint is not configured on nor trusted by the master node:
1134 Discarding 'check result' message from 'icinga2-client1.localdomain': Invalid endpoint origin (client not allowed).
1137 The check result message sent by the client does not belong to the zone the checkable object is
1141 Discarding 'check result' message from 'icinga2-client1.localdomain': Unauthorized access.
1145 ### Cluster Troubleshooting Replay Log <a id="troubleshooting-cluster-replay-log"></a>
1147 If your `/var/lib/icinga2/api/log` directory grows, it generally means that your cluster
1148 cannot replay the log on connection loss and re-establishment. A master node for example
1149 will store all events for not connected endpoints in the same and child zones.
1151 Check the following:
1153 * All clients are connected? (e.g. [cluster health check](06-distributed-monitoring.md#distributed-monitoring-health-checks)).
1154 * Check your [connection](15-troubleshooting.md#troubleshooting-cluster-connection-errors) in general.
1155 * Does the log replay work, e.g. are all events processed and the directory gets cleared up over time?
1156 * Decrease the `log_duration` attribute value for that specific [endpoint](09-object-types.md#objecttype-endpoint).
1160 ### Cluster Troubleshooting: Windows Agents <a id="troubleshooting-cluster-windows-agents"></a>
1162 #### Windows Agents consuming 100% CPU <a id="troubleshooting-cluster-windows-agents-cpu"></a>
1164 Icinga 2 requires the `NodeName` [constant](17-language-reference.md#constants) in various places to run.
1165 This includes loading the TLS certificates, setting the proper check source,
1168 Typically the Windows setup wizard and also the CLI commands populate the [constants.conf](04-configuring-icinga-2.md#constants-conf)
1169 file with the auto-detected or user-provided FQDN/Common Name.
1171 If this constant is not set during startup, Icinga will try to resolve the
1172 FQDN, if that fails, fetch the hostname. If everything fails, it logs
1173 an error and sets this to `localhost`. This results in undefined behaviour
1174 if ignored by the admin.
1176 Querying the DNS when not reachable is CPU consuming, and may look like Icinga
1177 is doing lots of checks, etc. but actually really is just starting up.
1179 In order to fix this, edit the `constants.conf` file and populate
1180 the `NodeName` constant with the FQDN. Ensure this is the same value
1181 as the local endpoint object name.
1184 const NodeName = "windows-client1.domain.com"
1189 #### Windows blocking Icinga 2 with ephemeral port range <a id="troubleshooting-cluster-windows-agents-ephemeral-port-range"></a>
1191 When you see a message like this in your Windows agent logs:
1194 critical/TcpSocket: Invalid socket: 10055, "An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full."
1197 Windows is blocking Icinga 2 and as such, no more TCP connection handling is possible.
1199 Depending on the version, patch level and installed applications, Windows is changing its
1200 range of [ephemeral ports](https://en.wikipedia.org/wiki/Ephemeral_port#Range).
1202 In order to solve this, raise the the `MaxUserPort` value in the registry.
1205 HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
1207 Value Name: MaxUserPort Value
1212 More details in [this blogpost](https://www.netways.de/blog/2019/01/24/windows-blocking-icinga-2-with-ephemeral-port-range/)
1213 and this [MS help entry](https://support.microsoft.com/en-us/help/196271/when-you-try-to-connect-from-tcp-ports-greater-than-5000-you-receive-t).