1 # Icinga 2 Troubleshooting <a id="troubleshooting"></a>
3 ## Required Information <a id="troubleshooting-information-required"></a>
5 Please ensure to provide any detail which may help reproduce and understand your issue.
6 Whether you ask on the [community channels](https://community.icinga.com) or you
7 create an issue at [GitHub](https://github.com/Icinga), make sure
8 that others can follow your explanations. If necessary, draw a picture and attach it for
9 better illustration. This is especially helpful if you are troubleshooting a distributed
12 We've come around many community questions and compiled this list. Add your own
13 findings and details please.
15 * Describe the expected behavior in your own words.
16 * Describe the actual behavior in one or two sentences.
17 * Ensure to provide general information such as:
18 * How was Icinga 2 installed (and which repository in case) and which distribution are you using
20 * `icinga2 feature list`
22 * [Icinga Web 2](https://icinga.com/products/icinga-web-2/) version (screenshot from System - About)
23 * [Icinga Web 2 modules](https://icinga.com/products/icinga-web-2-modules/) e.g. the Icinga Director (optional)
24 * Configuration insights:
25 * Provide complete configuration snippets explaining your problem in detail
26 * Your [icinga2.conf](04-configuration.md#icinga2-conf) file
27 * If you run multiple Icinga 2 instances, the [zones.conf](04-configuration.md#zones-conf) file (or `icinga2 object list --type Endpoint` and `icinga2 object list --type Zone`) from all affected nodes.
29 * Relevant output from your main and [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) in `/var/log/icinga2`. Please add step-by-step explanations with timestamps if required.
30 * The newest Icinga 2 crash log if relevant, located in `/var/log/icinga2/crash`
32 * If the check command failed, what's the output of your manual plugin tests?
33 * In case of [debugging](21-development.md#development) Icinga 2, the full back traces and outputs
35 ## Analyze your Environment <a id="troubleshooting-analyze-environment"></a>
37 There are many components involved on a server running Icinga 2. When you
38 analyze a problem, keep in mind that basic system administration knowledge
39 is also key to identify bottlenecks and issues.
43 > [Monitor Icinga 2](08-advanced-topics.md#monitoring-icinga) and use the hints for further analysis.
45 * Analyze the system's performance and dentify bottlenecks and issues.
46 * Collect details about all applications (e.g. Icinga 2, MySQL, Apache, Graphite, Elastic, etc.).
47 * If data is exchanged via network (e.g. central MySQL cluster) ensure to monitor the bandwidth capabilities too.
48 * Add graphs from Grafana or Graphite as screenshots to your issue description
50 Install tools which help you to do so. Opinions differ, let us know if you have any additions here!
52 ### Analyse your Linux/Unix Environment <a id="troubleshooting-analyze-environment-linux"></a>
54 [htop](https://hisham.hm/htop/) is a better replacement for `top` and helps to analyze processes
62 If you are for example experiencing performance issues, open `htop` and take a screenshot.
63 Add it to your question and/or bug report.
65 Analyse disk I/O performance in Grafana, take a screenshot and obfuscate any sensitive details.
66 Attach it when posting a question to the community channels.
68 The [sysstat](https://github.com/sysstat/sysstat) package provides a number of tools to
69 analyze the performance on Linux. On FreeBSD you could use `systat` for example.
73 apt-get install sysstat
76 Example for `vmstat` (summary of memory, processes, etc.):
81 // print timestamps, format in MB, stats every 1 second, 5 times
99 `sysstat` also provides the `iostat` binary. On FreeBSD you could use `systat` for example.
101 If you are missing checks and metrics found in your analysis, add them to your monitoring!
103 ### Analyze your Windows Environment <a id="troubleshooting-analyze-environment-windows"></a>
105 A good tip for Windows are the tools found inside the [Sysinternals Suite](https://technet.microsoft.com/en-us/sysinternals/bb842062.aspx).
107 You can also start `perfmon` and analyze specific performance counters.
108 Keep notes which could be important for your monitoring, and add service
113 > Use an administrative Powershell to gain more insights.
116 cd C:\ProgramData\icinga2\var\log\icinga2
118 Get-Content .\icinga2.log -tail 10 -wait
121 ## Enable Debug Output <a id="troubleshooting-enable-debug-output"></a>
123 ### Enable Debug Output on Linux/Unix <a id="troubleshooting-enable-debug-output-linux"></a>
125 Enable the `debuglog` feature:
128 # icinga2 feature enable debuglog
129 # service icinga2 restart
132 The debug log file can be found in `/var/log/icinga2/debug.log`.
134 You can tail the log files with an administrative shell:
141 Alternatively you may run Icinga 2 in the foreground with debugging enabled. Specify the console
142 log severity as an additional parameter argument to `-x`.
145 # /usr/sbin/icinga2 daemon -x notice
148 The [log severity](09-object-types.md#objecttype-filelogger) can be one of `critical`, `warning`, `information`, `notice`
151 ### Enable Debug Output on Windows <a id="troubleshooting-enable-debug-output-windows"></a>
153 Open a Powershell with administrative privileges and enable the debug log feature.
156 C:\> cd C:\Program Files\ICINGA2\sbin
158 C:\Program Files\ICINGA2\sbin> .\icinga2.exe feature enable debuglog
161 Ensure that the Icinga 2 service already writes the main log into `C:\ProgramData\icinga2\var\log\icinga2`.
162 Restart the Icinga 2 service in an administrative Powershell and open the newly created `debug.log` file.
165 C:\> Restart-Service icinga2
167 C:\> Get-Service icinga2
170 You can tail the log files with an administrative Powershell:
173 C:\> cd C:\ProgramData\icinga2\var\log\icinga2
175 C:\ProgramData\icinga2\var\log\icinga2> Get-Content .\debug.log -tail 10 -wait
178 ## Configuration Troubleshooting <a id="troubleshooting-configuration"></a>
180 ### List Configuration Objects <a id="troubleshooting-list-configuration-objects"></a>
182 The `icinga2 object list` CLI command can be used to list all configuration objects and their
183 attributes. The tool also shows where each of the attributes was modified.
187 > Use the Icinga 2 API to access [config objects at runtime](12-icinga2-api.md#icinga2-api-config-objects) directly.
189 That way you can also identify which objects have been created from your [apply rules](17-language-reference.md#apply).
192 # icinga2 object list
194 Object 'localhost!ssh' of type 'Service':
195 * __name = 'localhost!ssh'
196 * check_command = 'ssh'
197 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 5:3-5:23
198 * check_interval = 60
199 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 24:3-24:21
200 * host_name = 'localhost'
201 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 4:3-4:25
202 * max_check_attempts = 3
203 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 23:3-23:24
205 * retry_interval = 30
206 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 25:3-25:22
207 * templates = [ 'ssh', 'generic-service' ]
208 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 1:0-7:1
209 % += modified in '/etc/icinga2/conf.d/templates.conf', lines 22:1-26:1
212 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
214 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
219 On Windows, use an administrative Powershell:
222 C:\> cd C:\Program Files\ICINGA2\sbin
224 C:\Program Files\ICINGA2\sbin> .\icinga2.exe object list
227 You can also filter by name and type:
230 # icinga2 object list --name *ssh* --type Service
231 Object 'localhost!ssh' of type 'Service':
232 * __name = 'localhost!ssh'
233 * check_command = 'ssh'
234 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 5:3-5:23
235 * check_interval = 60
236 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 24:3-24:21
237 * host_name = 'localhost'
238 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 4:3-4:25
239 * max_check_attempts = 3
240 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 23:3-23:24
242 * retry_interval = 30
243 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 25:3-25:22
244 * templates = [ 'ssh', 'generic-service' ]
245 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 1:0-7:1
246 % += modified in '/etc/icinga2/conf.d/templates.conf', lines 22:1-26:1
249 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
251 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
253 Found 1 Service objects.
255 [2014-10-15 14:27:19 +0200] information/cli: Parsed 175 objects.
258 Runtime modifications via the [REST API](12-icinga2-api.md#icinga2-api-config-objects)
259 are not immediately updated. Furthermore there is a known issue with
260 [group assign expressions](17-language-reference.md#group-assign) which are not reflected in the host object output.
261 You need to restart Icinga 2 in order to update the `icinga2.debug` cache file.
263 ### Apply rules do not match <a id="apply-rules-do-not-match"></a>
265 You can analyze apply rules and matching objects by using the [script debugger](20-script-debugger.md#script-debugger).
267 ### Where are the check command definitions? <a id="check-command-definitions"></a>
269 Icinga 2 features a number of built-in [check command definitions](10-icinga-template-library.md#icinga-template-library) which are
277 in the [icinga2.conf](04-configuration.md#icinga2-conf) configuration file. These files are not considered
278 configuration files and will be overridden on upgrade, so please send modifications as proposed patches upstream.
279 The default include path is set to `/usr/share/icinga2/includes` with the constant `IncludeConfDir`.
281 You should add your own command definitions to a new file in `conf.d/` called `commands.conf`
284 ### Configuration is ignored <a id="configuration-ignored"></a>
286 * Make sure that the line(s) are not [commented out](17-language-reference.md#comments) (starting with `//` or `#`, or
287 encapsulated by `/* ... */`).
288 * Is the configuration file included in [icinga2.conf](04-configuration.md#icinga2-conf)?
290 Run the [configuration validation](11-cli-commands.md#config-validation) and add `notice` as log severity.
291 Search for the file which should be included i.e. using the `grep` CLI command.
294 # icinga2 daemon -C -x notice | grep command
297 ### Configuration attributes are inherited from <a id="configuration-attribute-inheritance"></a>
299 Icinga 2 allows you to import templates using the [import](17-language-reference.md#template-imports) keyword. If these templates
300 contain additional attributes, your objects will automatically inherit them. You can override
301 or modify these attributes in the current object.
303 The [object list](15-troubleshooting.md#troubleshooting-list-configuration-objects) CLI command allows you to verify the attribute origin.
305 ### Configuration Value with Single Dollar Sign <a id="configuration-value-dollar-sign"></a>
307 In case your configuration validation fails with a missing closing dollar sign error message, you
308 did not properly escape the single dollar sign preventing its usage as [runtime macro](03-monitoring-basics.md#runtime-macros).
311 critical/config: Error: Validation failed for Object 'ping4' (Type: 'Service') at /etc/icinga2/zones.d/global-templates/windows.conf:24: Closing $ not found in macro format string 'top-syntax=${list}'.
314 Correct the custom variable value to
317 "top-syntax=$${list}"
321 ## Checks Troubleshooting <a id="troubleshooting-checks"></a>
323 ### Executed Command for Checks <a id="checks-executed-command"></a>
325 * Use the Icinga 2 API to [query](12-icinga2-api.md#icinga2-api-config-objects-query) host/service objects
326 for their check result containing the executed shell command.
327 * Use the Icinga 2 [console cli command](11-cli-commands.md#cli-command-console)
328 to fetch the checkable object, its check result and the executed shell command.
329 * Alternatively enable the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) and look for the executed command.
331 Example for a service object query using a [regex match](18-library-reference.md#global-functions-regex)
335 $ curl -k -s -u root:icinga -H 'Accept: application/json' -H 'X-HTTP-Method-Override: GET' -X POST 'https://localhost:5665/v1/objects/services' \
336 -d '{ "filter": "regex(pattern, service.name)", "filter_vars": { "pattern": "^http" }, "attrs": [ "__name", "last_check_result" ], "pretty": true }'
341 "__name": "example.localdomain!http",
342 "last_check_result": {
344 "check_source": "example.localdomain",
346 "/usr/local/sbin/check_http",
359 "name": "example.localdomain!http",
366 Alternatively when using the Director, navigate into the Service Detail View
367 in Icinga Web and pick `Inspect` to query the details.
369 Example for using the `icinga2 console` CLI command evaluation functionality:
372 $ ICINGA2_API_PASSWORD=icinga icinga2 console --connect 'https://root@localhost:5665/' \
373 --eval 'get_service("example.localdomain", "http").last_check_result.command' | python -m json.tool
375 "/usr/local/sbin/check_http",
383 Example for searching the debug log:
386 # icinga2 feature enable debuglog
387 # systemctl restart icinga2
388 # tail -f /var/log/icinga2/debug.log | grep "notice/Process"
392 ### Checks are not executed <a id="checks-not-executed"></a>
394 * First off, decide whether the checks are executed locally, or remote in a distributed setup.
396 If the master does not receive check results from the satellite, move your analysis to the satellite
397 and verify why the checks are not executed there.
399 * Check the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) to see if the check command gets executed.
400 * Verify that failed dependencies do not prevent command execution.
401 * Make sure that the plugin is executable by the Icinga 2 user (run a manual test).
402 * Make sure the [checker](11-cli-commands.md#enable-features) feature is enabled.
403 * Use the Icinga 2 API [event streams](12-icinga2-api.md#icinga2-api-event-streams) to receive live check result streams.
405 Test a plugin as icinga user.
408 # sudo -u icinga /usr/lib/nagios/plugins/check_ping -4 -H 127.0.0.1 -c 5000,100% -w 3000,80%
413 > **Never test plugins as root, but the icinga daemon user.** The environment and permissions differ.
415 > Also, the daemon user **does not** spawn a terminal shell (Bash, etc.) so it won't read anything from .bashrc
416 > and variants. The Icinga daemon only relies on sysconfig environment variables being set.
419 Enable the checker feature.
422 # icinga2 feature enable checker
423 The feature 'checker' is already enabled.
426 Fetch all check result events matching the `event.service` name `random`:
429 $ curl -k -s -u root:icinga -H 'Accept: application/json' -X POST \
430 'https://localhost:5665/v1/events?queue=debugchecks&types=CheckResult&filter=match%28%22random*%22,event.service%29'
434 ### Analyze Check Source <a id="checks-check-source"></a>
436 Sometimes checks are not executed on the remote host, but on the master and so on.
437 This could lead into unwanted results or NOT-OK states.
439 The `check_source` attribute is the best indication where a check command
440 was actually executed. This could be a satellite with synced configuration
441 or a client as remote command bridge -- both will return the check source
442 as where the plugin is called.
444 Example for retrieving the check source from all `disk` services using a
445 [regex match](18-library-reference.md#global-functions-regex) on the name:
448 $ curl -k -s -u root:icinga -H 'Accept: application/json' -H 'X-HTTP-Method-Override: GET' -X POST 'https://localhost:5665/v1/objects/services' \
449 -d '{ "filter": "regex(pattern, service.name)", "filter_vars": { "pattern": "^disk" }, "attrs": [ "__name", "last_check_result" ], "pretty": true }'
454 "__name": "icinga2-agent1.localdomain!disk",
455 "last_check_result": {
457 "check_source": "icinga2-agent1.localdomain",
465 "name": "icinga2-agent1.localdomain!disk",
472 Alternatively when using the Director, navigate into the Service Detail View
473 in Icinga Web and pick `Inspect` to query the details.
475 Example with the debug console:
478 $ ICINGA2_API_PASSWORD=icinga icinga2 console --connect 'https://root@localhost:5665/' \
479 --eval 'get_service("icinga2-agent1.localdomain", "disk").last_check_result.check_source' | python -m json.tool
481 "icinga2-agent1.localdomain"
485 ### NSClient++ Check Errors with nscp-local <a id="nsclient-check-errors-nscp-local"></a>
487 The [nscp-local](10-icinga-template-library.md#nscp-check-local) CheckCommand object definitions call the local `nscp.exe` command.
488 If a Windows client service check fails to find the `nscp.exe` command, the log output would look like this:
491 Command ".\nscp.exe" "client" "-a" "drive=d" "-a" "show-all" "-b" "-q" "check_drivesize" failed to execute: 2, "The system cannot find the file specified."
498 scp.exe" "client" "-a" "drive=d" "-a" "show-all" "-b" "-q" "check_drivesize" failed to execute: 2, "The system cannot find the file specified."
501 The above actually prints `.\\nscp.exe` where the escaped `\n` character gets interpreted as new line.
503 Both errors lead to the assumption that the `NscpPath` constant is empty or set to a `.` character.
504 This could mean the following:
506 * The command is **not executed on the Windows client**. Check the [check_source](15-troubleshooting.md#checks-check-source) attribute from the check result.
507 * You are using an outdated NSClient++ version (0.3.x or 0.4.x) which is not compatible with Icinga 2.
508 * You are using a custom NSClient++ installer which does not register the correct GUID for NSClient++
510 More troubleshooting:
512 Retrieve the `NscpPath` constant on your Windows client:
515 C:\Program Files\ICINGA2\sbin\icinga2.exe variable get NscpPath
518 If the variable is returned empty, manually test how Icinga 2 would resolve
519 its path (this can be found inside the ITL):
522 C:\Program Files\ICINGA2\sbin\icinga2.exe console --eval "dirname(msi_get_component_path(\"{5C45463A-4AE9-4325-96DB-6E239C034F93}\"))"
525 If this command does not return anything, NSClient++ is not properly installed.
526 Verify that inside the `Programs and Features` (`appwiz.cpl`) control panel.
528 You can run the bundled NSClient++ installer from the Icinga 2 Windows package.
529 The msi package is located in `C:\Program Files\ICINGA2\sbin`.
531 The bundled NSClient++ version has properly been tested with Icinga 2. Keep that
532 in mind when using a different package.
535 ### Check Thresholds Not Applied <a id="check-thresholds-not-applied"></a>
537 This could happen with [clients as command endpoint execution](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint).
539 If you have for example a client host `icinga2-agent1.localdomain`
540 and a service `disk` check defined on the master, the warning and
541 critical thresholds are sometimes to applied and unwanted notification
544 This happens because the client itself includes a host object with
545 its `NodeName` and a basic set of checks in the [conf.d](04-configuration.md#conf-d)
546 directory, i.e. `disk` with the default thresholds.
548 Clients which have the `checker` feature enabled will attempt
549 to execute checks for local services and send their results
552 If you now have the same host and service objects on the
553 master you will receive wrong check results from the client.
557 * Disable the `checker` feature on clients: `icinga2 feature disable checker`.
558 * Remove the inclusion of [conf.d](04-configuration.md#conf-d) as suggested in the [client setup docs](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint).
560 ### Check Fork Errors <a id="check-fork-errors"></a>
562 Newer versions of systemd on Linux limit spawned processes for
565 * v227 introduces the `TasksMax` setting to units which allows to specify the spawned process limit.
566 * v228 adds `DefaultTasksMax` in the global `systemd-system.conf` with a default setting of 512 processes.
567 * v231 changes the default value to 15%
569 This can cause problems with Icinga 2 in large environments with many
570 commands executed in parallel starting with systemd v228. Some distributions
571 also may have changed the defaults.
573 The error message could look like this:
576 2017-01-12T11:55:40.742685+01:00 icinga2-master1 kernel: [65567.582895] cgroup: fork rejected by pids controller in /system.slice/icinga2.service
579 In order to solve the problem, increase the value for `DefaultTasksMax`
580 or set it to `infinity`.
583 mkdir /etc/systemd/system/icinga2.service.d
584 cat >/etc/systemd/system/icinga2.service.d/limits.conf <<EOF
586 DefaultTasksMax=infinity
589 systemctl daemon-reload
590 systemctl restart icinga2
593 An example is available inside the GitHub repository in [etc/initsystem](https://github.com/Icinga/icinga2/tree/master/etc/initsystem).
597 * [Fork limit for cgroups](https://lwn.net/Articles/663873/)
598 * [systemd changelog](https://github.com/systemd/systemd/blob/master/NEWS)
599 * [Icinga 2 upstream issue](https://github.com/Icinga/icinga2/issues/5611)
600 * [systemd upstream discussion](https://github.com/systemd/systemd/issues/3211)
602 ### Systemd Watchdog <a id="check-systemd-watchdog"></a>
604 Usually Icinga 2 is a mission critical part of infrastructure and should be
605 online at all times. In case of a recoverable crash (e.g. OOM) you may want to
606 restart Icinga 2 automatically. With systemd it is as easy as overriding some
607 settings of the Icinga 2 systemd service by creating
608 `/etc/systemd/system/icinga2.service.d/override.conf` with the following
615 StartLimitInterval=10
619 Using the watchdog can also help with monitoring Icinga 2, to activate and use it add the following to the override:
625 This way systemd will kill Icinga 2 if does not notify for over 30 seconds, a timout of less than 10 seconds is not
626 recommended. When the watchdog is activated, `Restart=` can be set to `watchdog` to restart Icinga 2 in the case of a
629 Run `systemctl daemon-reload && systemctl restart icinga2` to apply the changes.
630 Now systemd will always try to restart Icinga 2 (except if you run
631 `systemctl stop icinga2`). After three failures in ten seconds it will stop
632 trying because you probably have a problem that requires manual intervention.
634 ### Late Check Results <a id="late-check-results"></a>
636 [Icinga Web 2](https://icinga.com/products/icinga-web-2/) provides
637 a dashboard overview for `overdue checks`.
639 The REST API provides the [status](12-icinga2-api.md#icinga2-api-status) URL endpoint with some generic metrics
640 on Icinga and its features.
643 # curl -k -s -u root:icinga 'https://localhost:5665/v1/status?pretty=1' | less
646 You can also calculate late check results via the REST API:
648 * Fetch the `last_check` timestamp from each object
649 * Compare the timestamp with the current time and add `check_interval` multiple times (change it to see which results are really late, like five times check_interval)
651 You can use the [icinga2 console](11-cli-commands.md#cli-command-console) to connect to the instance, fetch all data
652 and calculate the differences. More infos can be found in [this blogpost](https://icinga.com/2016/08/11/analyse-icinga-2-problems-using-the-console-api/).
655 # ICINGA2_API_USERNAME=root ICINGA2_API_PASSWORD=icinga icinga2 console --connect 'https://localhost:5665/'
657 <1> => var res = []; for (s in get_objects(Service).filter(s => s.last_check < get_time() - 2 * s.check_interval)) { res.add([s.__name, DateTime(s.last_check).to_string()]) }; res
659 [ [ "10807-host!10807-service", "2016-06-10 15:54:55 +0200" ], [ "mbmif.int.netways.de!disk /", "2016-01-26 16:32:29 +0100" ] ]
662 Or if you are just interested in numbers, call [len](18-library-reference.md#array-len) on the result array `res`:
665 <2> => var res = []; for (s in get_objects(Service).filter(s => s.last_check < get_time() - 2 * s.check_interval)) { res.add([s.__name, DateTime(s.last_check).to_string()]) }; res.len()
670 If you need to analyze that problem multiple times, just add the current formatted timestamp
671 and repeat the commands.
674 <23> => DateTime(get_time()).to_string()
676 "2017-04-04 16:09:39 +0200"
678 <24> => var res = []; for (s in get_objects(Service).filter(s => s.last_check < get_time() - 2 * s.check_interval)) { res.add([s.__name, DateTime(s.last_check).to_string()]) }; res.len()
683 More details about the Icinga 2 DSL and its possibilities can be
684 found in the [language](17-language-reference.md#language-reference) and [library](18-library-reference.md#library-reference) reference chapters.
686 ### Late Check Results in Distributed Environments <a id="late-check-results-distributed"></a>
688 When it comes to a distributed HA setup, each node is responsible for a load-balanced amount of checks.
689 Host and Service objects provide the attribute `paused`. If this is set to `false`, the current node
690 actively attempts to schedule and execute checks. Otherwise the node does not feel responsible.
693 <3> => var res = {}; for (s in get_objects(Service).filter(s => s.last_check < get_time() - 2 * s.check_interval)) { res[s.paused] += 1 }; res
700 You may ask why this analysis is important? Fair enough - if the numbers are not inverted in a HA zone
701 with two members, this may give a hint that the cluster nodes are in a split-brain scenario, or you've
702 found a bug in the cluster.
705 If you are running a cluster setup where the master/satellite executes checks on the client via
706 [top down command endpoint](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint) mode,
707 you might want to know which zones are affected.
709 This analysis assumes that clients which are not connected, have the string `connected` in their
710 service check result output and their state is `UNKNOWN`.
713 <4> => var res = {}; for (s in get_objects(Service)) { if (s.state==3) { if (match("*connected*", s.last_check_result.output)) { res[s.zone] += [s.host_name] } } }; for (k => v in res) { res[k] = len(v.unique()) }; res
722 The result set shows the configured zones and their affected hosts in a unique list. The output also just prints the numbers
723 but you can adjust this by omitting the `len()` call inside the for loop.
725 ## Notifications Troubleshooting <a id="troubleshooting-notifications"></a>
727 ### Notifications are not sent <a id="troubleshooting-notifications-not-sent"></a>
729 * Check the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) to see if a notification is triggered.
730 * If yes, verify that all conditions are satisfied.
731 * Are any errors on the notification command execution logged?
733 Please ensure to add these details with your own description
734 to any question or issue posted to the community channels.
736 Verify the following configuration:
738 * Is the host/service `enable_notifications` attribute set, and if so, to which value?
739 * Do the [notification](09-object-types.md#objecttype-notification) attributes `states`, `types`, `period` match the notification conditions?
740 * Do the [user](09-object-types.md#objecttype-user) attributes `states`, `types`, `period` match the notification conditions?
741 * Are there any notification `begin` and `end` times configured?
742 * Make sure the [notification](11-cli-commands.md#enable-features) feature is enabled.
743 * Does the referenced NotificationCommand work when executed as Icinga user on the shell?
745 If notifications are to be sent via mail, make sure that the mail program specified inside the
746 [NotificationCommand object](09-object-types.md#objecttype-notificationcommand) exists.
747 The name and location depends on the distribution so the preconfigured setting might have to be
748 changed on your system.
754 # icinga2 feature enable notification
755 The feature 'notification' is already enabled.
759 # icinga2 feature enable debuglog
760 # systemctl restart icinga2
762 # grep Notification /var/log/icinga2/debug.log > /root/analyze_notification_problem.log
765 You can use the Icinga 2 API [event streams](12-icinga2-api.md#icinga2-api-event-streams) to receive live notification streams:
768 $ curl -k -s -u root:icinga -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/events?queue=debugnotifications&types=Notification'
772 ### Analyze Notification Result <a id="troubleshooting-notifications-result"></a>
776 > This feature is available since v2.11 and requires all endpoints
779 Notifications inside a HA enabled zone are balanced between the endpoints,
782 Sometimes notifications may fail, and with looking into the (debug) logs
783 for both masters, you cannot correlate this correctly.
785 The `last_notification_result` runtime attribute is stored and synced for Notification
786 objects and can be queried via REST API.
788 Example for retrieving the notification object and result from all `disk` services using a
789 [regex match](18-library-reference.md#global-functions-regex) on the name:
792 $ curl -k -s -u root:icinga -H 'Accept: application/json' -H 'X-HTTP-Method-Override: GET' -X POST 'https://localhost:5665/v1/objects/notifications' \
793 -d '{ "filter": "regex(pattern, service.name)", "filter_vars": { "pattern": "^disk" }, "attrs": [ "__name", "last_notification_result" ], "pretty": true }'
799 "last_notification_result": {
802 "/etc/icinga2/scripts/mail-service-notification.sh",
812 "2019-08-02 10:54:16 +0200",
816 "icinga2-agent1.localdomain",
818 "icinga2-agent1.localdomain",
820 "DISK OK - free space: / 38108 MB (90.84% inode=100%);",
830 "execution_end": 1564736056.186217,
831 "execution_endpoint": "icinga2-master1.localdomain",
832 "execution_start": 1564736056.132323,
835 "type": "NotificationResult"
840 "name": "icinga2-agent1.localdomain!disk!mail-service-notification",
841 "type": "Notification"
850 Example with the debug console:
853 $ ICINGA2_API_PASSWORD=icinga icinga2 console --connect 'https://root@localhost:5665/' --eval 'get_object(Notification, "icinga2-agent1.localdomain!disk!mail-service-notification").last_notification_result.execution_endpoint' | jq
855 "icinga2-agent1.localdomain"
858 Whenever a notification command failed to execute, you can fetch the output as well.
861 ## Feature Troubleshooting <a id="troubleshooting-features"></a>
863 ### Feature is not working <a id="feature-not-working"></a>
865 * Make sure that the feature configuration is enabled by symlinking from `features-available/`
866 to `features-enabled` and that the latter is included in [icinga2.conf](04-configuration.md#icinga2-conf).
867 * Are the feature attributes set correctly according to the documentation?
868 * Any errors on the logs?
870 Look up the [object type](09-object-types.md#object-types) for the required feature and verify it is enabled:
873 # icinga2 object list --type <feature object type>
876 Example for the `graphite` feature:
879 # icinga2 object list --type GraphiteWriter
882 Look into the log and check whether the feature logs anything specific for this matter.
885 grep GraphiteWriter /var/log/icinga2/icinga2.log
888 ## REST API Troubleshooting <a id="troubleshooting-api"></a>
890 In order to analyse errors on API requests, you can explicitly enable the [verbose parameter](12-icinga2-api.md#icinga2-api-parameters-global).
893 $ curl -k -s -u root:icinga -H 'Accept: application/json' -X DELETE 'https://localhost:5665/v1/objects/hosts/example-cmdb?pretty=1&verbose=1'
895 "diagnostic_information": "Error: Object does not exist.\n\n ....",
897 "status": "No objects found."
901 ### REST API Troubleshooting: No Objects Found <a id="troubleshooting-api-no-objects-found"></a>
903 Please note that the `404` status with no objects being found can also originate
904 from missing or too strict object permissions for the authenticated user.
906 This is a security feature to disable object name guessing. If this would not be the
907 case, restricted users would be able to get a list of names of your objects just by
908 trying every character combination.
910 In order to analyse and fix the problem, please check the following:
912 - use an administrative account with full permissions to check whether the objects are actually there.
913 - verify the permissions on the affected ApiUser object and fix them.
915 ### Missing Runtime Objects (Hosts, Downtimes, etc.) <a id="troubleshooting-api-missing-runtime-objects"></a>
917 Runtime objects consume the internal config packages shared with
918 the REST API config packages. Each host, downtime, comment, service, etc. created
919 via the REST API is stored in the `_api` package.
921 This includes downtimes and comments, which where sometimes stored in the wrong
922 directory path, because the active-stage file was empty/truncated/unreadable at
928 /var/lib/icinga2/api/packages/_api//conf.d/downtimes/1234-5678-9012-3456.conf
934 /var/lib/icinga2/api/packages/_api/dbe0bef8-c72c-4cc9-9779-da7c4527c5b2/conf.d/downtimes/1234-5678-9012-3456.conf
937 At creation time, the object lives in memory but its storage is broken. Upon restart,
938 it is missing and e.g. a missing downtime will re-enable unwanted notifications.
940 `abcd-ef12-3456-7890` is the active stage name which wasn't correctly
941 read by the Icinga daemon. This information is stored in `/var/lib/icinga2/api/packages/_api/active-stage`.
943 2.11 now limits the direct active-stage file access (this is hidden from the user),
944 and caches active stages for packages in-memory.
946 It also tries to repair the broken package, and logs a new message:
949 systemctl restart icinga2
951 tail -f /var/log/icinga2/icinga2.log
953 [2019-05-10 12:27:15 +0200] information/ConfigObjectUtility: Repairing config package '_api' with stage 'dbe0bef8-c72c-4cc9-9779-da7c4527c5b2'.
956 If this does not happen, you can manually fix the broken config package, and mark a deployed stage as active
957 again, carefully do the following steps with creating a backup before:
959 Navigate into the API package prefix.
962 cd /var/lib/icinga2/api/packages
965 Change into the broken package directory and list all directories and files
966 ordered by latest changes.
972 drwx------ 4 michi wheel 128B Mar 27 14:39 ..
973 -rw-r--r-- 1 michi wheel 25B Mar 27 14:39 include.conf
974 -rw-r--r-- 1 michi wheel 405B Mar 27 14:39 active.conf
975 drwx------ 7 michi wheel 224B Mar 27 15:01 dbe0bef8-c72c-4cc9-9779-da7c4527c5b2
976 drwx------ 5 michi wheel 160B Apr 26 12:47 .
979 As you can see, the `active-stage` file is missing. When it is there, verify that its content
980 is set to the stage directory as follows.
982 If you have more than one stage directory here, pick the latest modified
983 directory. Copy the directory name `abcd-ef12-3456-7890` and
984 add it into a new file `active-stage`. This can be done like this:
987 echo "dbe0bef8-c72c-4cc9-9779-da7c4527c5b2" > active-stage
990 `active.conf` needs to have the correct active stage too, add it again
991 like this. Note: This is deep down in the code, use with care!
994 sed -i 's/ActiveStages\["_api"\] = .*/ActiveStages\["_api"\] = "dbe0bef8-c72c-4cc9-9779-da7c4527c5b2"/g' /var/lib/icinga2/api/packages/_api/active.conf
1000 systemctl restart icinga2
1006 > The internal `_api` config package structure may change in the future. Do not modify
1007 > things in there manually or with scripts unless guided here or asked by a developer.
1010 ## Certificate Troubleshooting <a id="troubleshooting-certificate"></a>
1012 Tools for analysing certificates and TLS connections:
1014 - `openssl` binary on Linux/Unix, `openssl.exe` on Windows ([download](https://slproweb.com/products/Win32OpenSSL.html))
1015 - `sslscan` tool, available [here](https://github.com/rbsec/sslscan) (Linux/Windows)
1017 Note: You can also execute sslscan on Windows using Powershell.
1020 ### Certificate Verification <a id="troubleshooting-certificate-verification"></a>
1022 Whenever the TLS handshake fails when a client connects to the cluster or the REST API,
1023 ensure to verify the used certificates.
1025 Print the CA and client certificate and ensure that the following attributes are set:
1027 * Version must be 3.
1028 * Serial number is a hex-encoded string.
1029 * Issuer should be your certificate authority (defaults to `Icinga CA` for all certificates generated by CLI commands and automated signing requests).
1030 * Validity: The certificate must not be expired.
1031 * Subject with the common name (CN) matches the client endpoint name and its FQDN.
1032 * v3 extensions must set the basic constraint for `CA:TRUE` (ca.crt) or `CA:FALSE` (client certificate).
1033 * Subject Alternative Name is set to the resolvable DNS name (required for REST API and browsers).
1036 Navigate into the local certificate store:
1039 $ cd /var/lib/icinga2/certs/
1042 Print the CA certificate:
1045 $ openssl x509 -in ca.crt -text
1050 Serial Number: 1 (0x1)
1051 Signature Algorithm: sha256WithRSAEncryption
1052 Issuer: CN=Icinga CA
1054 Not Before: Feb 23 14:45:32 2016 GMT
1055 Not After : Feb 19 14:45:32 2031 GMT
1056 Subject: CN=Icinga CA
1057 Subject Public Key Info:
1058 Public Key Algorithm: rsaEncryption
1059 Public-Key: (4096 bit)
1062 Exponent: 65537 (0x10001)
1064 X509v3 Basic Constraints: critical
1066 Signature Algorithm: sha256WithRSAEncryption
1070 Print the client public certificate:
1073 $ openssl x509 -in icinga2-agent1.localdomain.crt -text
1079 86:47:44:65:49:c6:65:6b:5e:6d:4f:a5:fe:6c:76:05:0b:1a:cf:34
1080 Signature Algorithm: sha256WithRSAEncryption
1081 Issuer: CN=Icinga CA
1083 Not Before: Aug 20 16:20:05 2016 GMT
1084 Not After : Aug 17 16:20:05 2031 GMT
1085 Subject: CN=icinga2-agent1.localdomain
1086 Subject Public Key Info:
1087 Public Key Algorithm: rsaEncryption
1088 Public-Key: (4096 bit)
1091 Exponent: 65537 (0x10001)
1093 X509v3 Basic Constraints: critical
1095 X509v3 Subject Alternative Name:
1096 DNS:icinga2-agent1.localdomain
1097 Signature Algorithm: sha256WithRSAEncryption
1101 Make sure to verify the client's certificate and its received `ca.crt` in `/var/lib/icinga2/certs` and ensure that
1102 both instances are signed by the **same CA**.
1105 $ openssl verify -verbose -CAfile /var/lib/icinga2/certs/ca.crt /var/lib/icinga2/certs/icinga2-master1.localdomain.crt
1107 icinga2-master1.localdomain.crt: OK
1111 $ openssl verify -verbose -CAfile /var/lib/icinga2/certs/ca.crt /var/lib/icinga2/certs/icinga2-agent1.localdomain.crt
1113 icinga2-agent1.localdomain.crt: OK
1116 Fetch the `ca.crt` file from the client node and compare it to your master's `ca.crt` file:
1119 $ scp icinga2-agent1:/var/lib/icinga2/certs/ca.crt test-client-ca.crt
1120 $ diff -ur /var/lib/icinga2/certs/ca.crt test-client-ca.crt
1123 ### Certificate Signing <a id="troubleshooting-certificate-signing"></a>
1125 Icinga offers two methods:
1127 * [CSR Auto-Signing](06-distributed-monitoring.md#distributed-monitoring-setup-csr-auto-signing) which uses a client (an agent or a satellite) ticket generated on the master as trust identifier.
1128 * [On-Demand CSR Signing](06-distributed-monitoring.md#distributed-monitoring-setup-on-demand-csr-signing) which allows to sign pending certificate requests on the master.
1130 Whenever a signed certificate is not received on the requesting clients, ensure to check the following:
1132 * The ticket was valid and the master's log shows nothing different (CSR Auto-Signing only)
1133 * If the agent/satellite is directly connected to the CA master, check whether the master actually has performance problems to process the request. If the connection is closed without certificate response, analyse the master's health. It is also advised to upgrade to v2.11 where network stack problems have been fixed.
1134 * If you're using a 3+ level cluster, check whether the satellite really forwarded the CSR signing request and the master processed it.
1136 Other common errors:
1138 * The generated ticket is invalid. The client receives this error message, as well as the master logs a warning message.
1139 * The [api](09-object-types.md#objecttype-apilistener) feature does not have the `ticket_salt` attribute set to the generated `TicketSalt` constant by the CLI wizards.
1141 In case you are using On-Demand CSR Signing, `icinga2 ca list` on the master only lists
1142 pending requests since v2.11. Add `--all` to also see signed requests. Keep in mind that
1143 old requests are purged after 1 week automatically.
1146 ### TLS Handshake: Ciphers <a id="troubleshooting-certificate-handshake-ciphers"></a>
1148 Starting with v2.11, the default configured ciphers have been hardened to modern
1149 standards. This includes TLS v1.2 as minimum protocol version too.
1151 In case the TLS handshake fails with `no shared cipher`, first analyse whether both
1152 instances support the same ciphers.
1154 #### Client connects to Server <a id="troubleshooting-certificate-handshake-ciphers-client"></a>
1156 Connect using `openssl s_client` and try to reproduce the connection problem.
1160 > The endpoint with the server role **accepting** the connection picks the preferred
1161 > cipher. E.g. when a satellite connects to the master, the master chooses the cipher.
1163 > Keep this in mind where to simulate the client role connecting to a server with
1164 > CLI tools such as `openssl s_client`.
1167 `openssl s_client` tells you about the supported and shared cipher suites
1168 on the remote server. `openssl ciphers` lists locally available ciphers.
1171 $ openssl s_client -connect 192.168.33.5:5665
1175 SSL handshake has read 2899 bytes and written 786 bytes
1177 New, TLSv1/SSLv3, Cipher is AES256-GCM-SHA384
1178 Server public key is 4096 bit
1179 Secure Renegotiation IS supported
1185 Cipher : AES256-GCM-SHA384
1190 You can specifically use one cipher or a list with the `-cipher` parameter:
1193 openssl s_client -connect 192.168.33.5:5665 -cipher 'ECDHE-RSA-AES256-GCM-SHA384'
1196 In order to fully simulate a connecting client, provide the certificates too:
1199 CERTPATH='/var/lib/icinga2/certs'
1200 HOSTNAME='icinga2.vagrant.demo.icinga.com'
1201 openssl s_client -connect 192.168.33.5:5665 -cert "${CERTPATH}/${HOSTNAME}.crt" -key "${CERTPATH}/${HOSTNAME}.key" -CAfile "${CERTPATH}/ca.crt" -cipher 'ECDHE-RSA-AES256-GCM-SHA384'
1204 In case to need to change the default cipher list,
1205 set the [cipher_list](09-object-types.md#objecttype-apilistener) attribute
1206 in the `api` feature configuration accordingly.
1208 Beware of using insecure ciphers, this may become a
1209 security risk in your organisation.
1211 #### Server Accepts Client <a id="troubleshooting-certificate-handshake-ciphers-server"></a>
1213 If the master node does not actively connect to the satellite/agent node(s), but instead
1214 the child node actively connectsm, you can still simulate a TLS handshake.
1216 Use `openssl s_server` instead of `openssl s_client` on the master during the connection
1220 $ openssl s_server -connect 192.168.56.101:5665
1223 Since the server role chooses the preferred cipher suite in Icinga,
1224 you can test-drive the "agent connects to master" mode here, granted that
1225 the TCP connection is not blocked by the firewall.
1228 #### Cipher Scan Tools <a id="troubleshooting-certificate-handshake-ciphers-scantools"></a>
1230 You can also use different tools to test the available cipher suites, this is what SSL Labs, etc.
1231 provide for TLS enabled websites as well. [This post](https://superuser.com/questions/109213/how-do-i-list-the-ssl-tls-cipher-suites-a-particular-website-offers)
1232 highlights some tools and scripts such as [sslscan](https://github.com/rbsec/sslscan) or [testssl.sh](https://github.com/drwetter/testssl.sh/)
1234 Example for sslscan on macOS against a Debian 10 Buster instance
1238 $ brew install sslscan
1240 $ sslscan 192.168.33.22:5665
1241 Version: 1.11.13-static
1242 OpenSSL 1.0.2f 28 Jan 2016
1244 Connected to 192.168.33.22
1246 Testing SSL server 192.168.33.22 on port 5665 using SNI name 192.168.33.22
1249 Server supports TLS Fallback SCSV
1252 Session renegotiation not supported
1255 Compression disabled
1258 TLS 1.2 not vulnerable to heartbleed
1259 TLS 1.1 not vulnerable to heartbleed
1260 TLS 1.0 not vulnerable to heartbleed
1262 Supported Server Cipher(s):
1263 Preferred TLSv1.2 256 bits ECDHE-RSA-AES256-GCM-SHA384 Curve P-256 DHE 256
1264 Accepted TLSv1.2 128 bits ECDHE-RSA-AES128-GCM-SHA256 Curve P-256 DHE 256
1265 Accepted TLSv1.2 256 bits ECDHE-RSA-AES256-SHA384 Curve P-256 DHE 256
1266 Accepted TLSv1.2 128 bits ECDHE-RSA-AES128-SHA256 Curve P-256 DHE 256
1269 Signature Algorithm: sha256WithRSAEncryption
1270 RSA Key Strength: 4096
1272 Subject: icinga2-debian10.vagrant.demo.icinga.com
1273 Altnames: DNS:icinga2-debian10.vagrant.demo.icinga.com
1276 Not valid before: Jul 12 07:39:55 2019 GMT
1277 Not valid after: Jul 8 07:39:55 2034 GMT
1280 ## Distributed Troubleshooting <a id="troubleshooting-cluster"></a>
1282 This applies to any Icinga 2 node in a [distributed monitoring setup](06-distributed-monitoring.md#distributed-monitoring-scenarios).
1284 You should configure the [cluster health checks](06-distributed-monitoring.md#distributed-monitoring-health-checks) if you haven't
1289 > Some problems just exist due to wrong file permissions or applied packet filters. Make
1290 > sure to check these in the first place.
1292 ### Cluster Troubleshooting Connection Errors <a id="troubleshooting-cluster-connection-errors"></a>
1294 General connection errors could be one of the following problems:
1296 * Incorrect network configuration
1298 * Firewall rules preventing traffic
1300 Use tools like `netstat`, `tcpdump`, `nmap`, etc. to make sure that the cluster communication
1301 works (default port is `5665`).
1304 # tcpdump -n port 5665 -i any
1306 # netstat -tulpen | grep icinga
1308 # nmap icinga2-agent1.localdomain
1311 ### Cluster Troubleshooting TLS Errors <a id="troubleshooting-cluster-tls-errors"></a>
1313 If the cluster communication fails with TLS/SSL error messages, make sure to check
1316 * File permissions on the TLS certificate files
1317 * Does the used CA match for all cluster endpoints?
1318 * Verify the `Issuer` being your trusted CA
1319 * Verify the `Subject` containing your endpoint's common name (CN)
1320 * Check the validity of the certificate itself
1322 Try to manually connect from `icinga2-agent1.localdomain` to the master node `icinga2-master1.localdomain`:
1325 $ openssl s_client -CAfile /var/lib/icinga2/certs/ca.crt -cert /var/lib/icinga2/certs/icinga2-agent1.localdomain.crt -key /var/lib/icinga2/certs/icinga2-agent1.localdomain.key -connect icinga2-master1.localdomain:5665
1332 If the connection attempt fails or your CA does not match, [verify the certificates](15-troubleshooting.md#troubleshooting-certificate-verification).
1335 #### Cluster Troubleshooting Unauthenticated Clients <a id="troubleshooting-cluster-unauthenticated-clients"></a>
1337 Unauthenticated nodes are able to connect. This is required for agent/satellite setups.
1342 [2015-07-13 18:29:25 +0200] information/ApiListener: New client connection for identity 'icinga2-agent1.localdomain' (unauthenticated)
1345 Agent as command execution bridge:
1348 [2015-07-13 18:29:26 +1000] notice/ClusterEvents: Discarding 'execute command' message from 'icinga2-master1.localdomain': Invalid endpoint origin (client not allowed).
1351 If these messages do not go away, make sure to [verify the master and agent certificates](15-troubleshooting.md#troubleshooting-certificate-verification).
1354 ### Cluster Troubleshooting Message Errors <a id="troubleshooting-cluster-message-errors"></a>
1356 When the network connection is broken or gone, the Icinga 2 instances will be disconnected.
1357 If the connection can't be re-established between endpoints in the same HA zone,
1358 they remain in a Split-Brain-mode and history may differ.
1360 Although the Icinga 2 cluster protocol stores historical events in a [replay log](15-troubleshooting.md#troubleshooting-cluster-replay-log)
1361 for later synchronisation, you should make sure to check why the network connection failed.
1363 Ensure to setup [cluster health checks](06-distributed-monitoring.md#distributed-monitoring-health-checks)
1364 to monitor all endpoints and zones connectivity.
1367 ### Cluster Troubleshooting Command Endpoint Errors <a id="troubleshooting-cluster-command-endpoint-errors"></a>
1369 Command endpoints can be used [for agents](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint)
1370 as well as inside an [High-Availability cluster](06-distributed-monitoring.md#distributed-monitoring-scenarios).
1372 There is no CLI command for manually executing the check, but you can verify
1373 the following (e.g. by invoking a forced check from the web interface):
1375 * `/var/log/icinga2/icinga2.log` shows connection and execution errors.
1376 * The ApiListener is not enabled to [accept commands](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint). This is visible as `UNKNOWN` check result output.
1377 * `CheckCommand` definition not found on the remote client. This is visible as `UNKNWON` check result output.
1378 * Referenced check plugin not found on the remote agent.
1379 * Runtime warnings and errors, e.g. unresolved runtime macros or configuration problems.
1380 * Specific error messages are also populated into `UNKNOWN` check results including a detailed error message in their output.
1381 * Verify the [check source](15-troubleshooting.md#checks-check-source). This is populated by the node executing the check. You can see that in Icinga Web's detail view or by querying the REST API for this checkable object.
1385 * More verbose logs are found inside the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output).
1387 * Use the Icinga 2 API [event streams](12-icinga2-api.md#icinga2-api-event-streams) to receive live check result streams.
1389 Fetch all check result events matching the `event.service` name `remote-client`:
1392 $ curl -k -s -u root:icinga -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/events?queue=debugcommandendpoint&types=CheckResult&filter=match%28%22remote-client*%22,event.service%29'
1397 ### Cluster Troubleshooting Config Sync <a id="troubleshooting-cluster-config-sync"></a>
1399 In order to troubleshoot this, remember the key things with the config sync:
1401 * Within a config master zone, only one configuration master is allowed to have its config in `/etc/icinga2/zones.d`.
1402 * The config master copies the zone configuration from `/etc/icinga2/zones.d` to `/var/lib/icinga2/api/zones`. This storage is the same for all cluster endpoints, and the source for all config syncs.
1403 * The config master puts the `.authoritative` marker on these zone files locally. This is to ensure that it doesn't receive config updates from other endpoints. If you have copied the content from `/var/lib/icinga2/api/zones` to another node, ensure to remove them.
1404 * During startup, the master validates the entire configuration and only syncs valid configuration to other zone endpoints.
1406 Satellites/Agents < 2.11 store the received configuration directly in `/var/lib/icinga2/api/zones`, validating it and reloading the daemon.
1407 Satellites/Agents >= 2.11 put the received configuration into the staging directory `/var/lib/icinga2/api/zones-stage` first, and will only copy this to the production directory `/var/lib/icinga2/api/zones` once the validation was successful.
1409 The configuration sync logs the operations during startup with the `information` severity level. Received zone configuration is also logged.
1413 * The api feature doesn't [accept config](06-distributed-monitoring.md#distributed-monitoring-top-down-config-sync). This is logged into `/var/lib/icinga2/icinga2.log`.
1414 * The received configuration zone is not configured in [zones.conf](04-configuration.md#zones-conf) and Icinga denies it. This is logged into `/var/lib/icinga2/icinga2.log`.
1415 * The satellite/agent has local configuration in `/etc/icinga2/zones.d` and thinks it is authoritive for this zone. It then denies the received update. Purge the content from `/etc/icinga2/zones.d`, `/var/lib/icinga2/api/zones/*` and restart Icinga to fix this.
1417 #### New configuration does not trigger a reload <a id="troubleshooting-cluster-config-sync-no-reload"></a>
1419 The debug/notice log dumps the calculated checksums for all files and the comparison. Analyse this to troubleshoot further.
1421 A complete sync for the `director-global` global zone can look like this:
1424 [2019-08-01 09:20:25 +0200] notice/JsonRpcConnection: Received 'config::Update' message from 'icinga2-master1.localdomain'
1425 [2019-08-01 09:20:25 +0200] information/ApiListener: Applying config update from endpoint 'icinga2-master1.localdomain' of zone 'master'.
1426 [2019-08-01 09:20:25 +0200] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/director-global/.checksums'.
1427 [2019-08-01 09:20:25 +0200] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/director-global/.timestamp'.
1428 [2019-08-01 09:20:25 +0200] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/director-global/director/001-director-basics.conf'.
1429 [2019-08-01 09:20:25 +0200] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/director-global/director/host_templates.conf'.
1430 [2019-08-01 09:20:25 +0200] information/ApiListener: Received configuration for zone 'director-global' from endpoint 'icinga2-master1.localdomain'. Comparing the checksums.
1431 [2019-08-01 09:20:25 +0200] debug/ApiListener: Checking for config change between stage and production. Old (4): '{"/.checksums":"c4dd1237e36dcad9142f4d9a81324a7cae7d01543a672299
1432 b8c1bb08b629b7d1","/.timestamp":"f21c0e6551328812d9f5176e5e31f390de0d431d09800a85385630727b404d83","/director/001-director-basics.conf":"f86583eec81c9bf3a1823a761991fb53d640bd0dc
1433 6cd12bf8c5e6a275359970f","/director/host_templates.conf":"831e9b7e3ec1e33288e56a51e63c688da1d6316155349382a101f7fce6229ecc"}' vs. new (4): '{"/.checksums":"c4dd1237e36dcad9142f4d
1434 9a81324a7cae7d01543a672299b8c1bb08b629b7d1","/.timestamp":"f21c0e6551328812d9f5176e5e31f390de0d431d09800a85385630727b404d83","/director/001-director-basics.conf":"f86583eec81c9bf
1435 3a1823a761991fb53d640bd0dc6cd12bf8c5e6a275359970f","/director/host_templates.conf":"831e9b7e3ec1e33288e56a51e63c688da1d6316155349382a101f7fce6229ecc"}'.
1436 [2019-08-01 09:20:25 +0200] debug/ApiListener: Ignoring old internal file '/.checksums'.
1437 [2019-08-01 09:20:25 +0200] debug/ApiListener: Ignoring old internal file '/.timestamp'.
1438 [2019-08-01 09:20:25 +0200] debug/ApiListener: Checking /director/001-director-basics.conf for old checksum: f86583eec81c9bf3a1823a761991fb53d640bd0dc6cd12bf8c5e6a275359970f.
1439 [2019-08-01 09:20:25 +0200] debug/ApiListener: Checking /director/host_templates.conf for old checksum: 831e9b7e3ec1e33288e56a51e63c688da1d6316155349382a101f7fce6229ecc.
1440 [2019-08-01 09:20:25 +0200] debug/ApiListener: Ignoring new internal file '/.checksums'.
1441 [2019-08-01 09:20:25 +0200] debug/ApiListener: Ignoring new internal file '/.timestamp'.
1442 [2019-08-01 09:20:25 +0200] debug/ApiListener: Checking /director/001-director-basics.conf for new checksum: f86583eec81c9bf3a1823a761991fb53d640bd0dc6cd12bf8c5e6a275359970f.
1443 [2019-08-01 09:20:25 +0200] debug/ApiListener: Checking /director/host_templates.conf for new checksum: 831e9b7e3ec1e33288e56a51e63c688da1d6316155349382a101f7fce6229ecc.
1444 [2019-08-01 09:20:25 +0200] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/001-director-basics.c
1445 onf' for zone 'director-global'.
1446 [2019-08-01 09:20:25 +0200] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/host_templates.conf'
1447 for zone 'director-global'.
1448 [2019-08-01 09:20:25 +0200] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/director-global' (2209 Bytes).
1452 [2019-08-01 09:20:25 +0200] information/ApiListener: Received configuration updates (4) from endpoint 'icinga2-master1.localdomain' are different to production, triggering validation and reload.
1453 [2019-08-01 09:20:25 +0200] notice/Process: Running command '/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2' '--no-stack-rlimit' 'daemon' '--close-stdio' '-e' '/var/log/icinga2/e
1454 rror.log' '--validate' '--define' 'System.ZonesStageVarDir=/var/lib/icinga2/api/zones-stage/': PID 4532
1455 [2019-08-01 09:20:25 +0200] notice/Process: PID 4532 ('/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2' '--no-stack-rlimit' 'daemon' '--close-stdio' '-e' '/var/log/icinga2/error.l
1456 og' '--validate' '--define' 'System.ZonesStageVarDir=/var/lib/icinga2/api/zones-stage/') terminated with exit code 0
1457 [2019-08-01 09:20:25 +0200] information/ApiListener: Config validation for stage '/var/lib/icinga2/api/zones-stage/' was OK, replacing into '/var/lib/icinga2/api/zones/' and trig
1459 [2019-08-01 09:20:26 +0200] information/ApiListener: Copying file 'director-global//.checksums' from config sync staging to production zones directory.
1460 [2019-08-01 09:20:26 +0200] information/ApiListener: Copying file 'director-global//.timestamp' from config sync staging to production zones directory.
1461 [2019-08-01 09:20:26 +0200] information/ApiListener: Copying file 'director-global//director/001-director-basics.conf' from config sync staging to production zones directory.
1462 [2019-08-01 09:20:26 +0200] information/ApiListener: Copying file 'director-global//director/host_templates.conf' from config sync staging to production zones directory.
1466 [2019-08-01 09:20:26 +0200] notice/Application: Got reload command, forwarding to umbrella process (PID 4236)
1469 #### Syncing Binary Files is Denied <a id="troubleshooting-cluster-config-sync-binary-denied"></a>
1471 The config sync is built for syncing text configuration files, wrapped into JSON-RPC messages.
1472 Some users have started to use this as binary file sync instead of using tools built for this:
1473 rsync, git, Puppet, Ansible, etc.
1475 Starting with 2.11, this attempt is now prohibited and logged.
1478 [2019-08-02 16:03:19 +0200] critical/ApiListener: Ignoring file '/etc/icinga2/zones.d/global-templates/forbidden.exe' for cluster config sync: Does not contain valid UTF8. Binary files are not supported.
1480 (0) Creating config update for file '/etc/icinga2/zones.d/global-templates/forbidden.exe'
1481 (1) Activating object 'api' of type 'ApiListener'
1484 In order to solve this problem, remove the mentioned files from `zones.d` and use an alternate way
1485 of syncing plugin binaries to your satellites and agents.
1488 ### Cluster Troubleshooting Overdue Check Results <a id="troubleshooting-cluster-check-results"></a>
1490 If your master does not receive check results (or any other events) from the child zones
1491 (satellite, clients, etc.), make sure to check whether the client sending in events
1492 is allowed to do so.
1496 > General troubleshooting hints on late check results are documented [here](15-troubleshooting.md#late-check-results).
1498 The [distributed monitoring conventions](06-distributed-monitoring.md#distributed-monitoring-conventions)
1499 apply. So, if there's a mismatch between your client node's endpoint name and its provided
1500 certificate's CN, the master will deny all events.
1504 > [Icinga Web 2](02-installation.md#setting-up-icingaweb2) provides a dashboard view
1505 > for overdue check results.
1507 Enable the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) on the master
1508 for more verbose insights.
1510 If the client cannot authenticate, it's a more general [problem](15-troubleshooting.md#troubleshooting-cluster-unauthenticated-clients).
1512 The client's endpoint is not configured on nor trusted by the master node:
1515 Discarding 'check result' message from 'icinga2-agent1.localdomain': Invalid endpoint origin (client not allowed).
1518 The check result message sent by the client does not belong to the zone the checkable object is
1522 Discarding 'check result' message from 'icinga2-agent1.localdomain': Unauthorized access.
1526 ### Cluster Troubleshooting Replay Log <a id="troubleshooting-cluster-replay-log"></a>
1528 If your `/var/lib/icinga2/api/log` directory grows, it generally means that your cluster
1529 cannot replay the log on connection loss and re-establishment. A master node for example
1530 will store all events for not connected endpoints in the same and child zones.
1532 Check the following:
1534 * All clients are connected? (e.g. [cluster health check](06-distributed-monitoring.md#distributed-monitoring-health-checks)).
1535 * Check your [connection](15-troubleshooting.md#troubleshooting-cluster-connection-errors) in general.
1536 * Does the log replay work, e.g. are all events processed and the directory gets cleared up over time?
1537 * Decrease the `log_duration` attribute value for that specific [endpoint](09-object-types.md#objecttype-endpoint).
1539 The cluster health checks also measure the `slave_lag` metric. Use this data to correlate
1540 graphs with other events (e.g. disk I/O, network problems, etc).
1543 ### Cluster Troubleshooting: Windows Agents <a id="troubleshooting-cluster-windows-agents"></a>
1546 #### Windows Service Exe Path <a id="troubleshooting-cluster-windows-agents-service-exe-path"></a>
1548 Icinga agents can be installed either as x86 or x64 package. If you enable features, or wonder why
1549 logs are not written, the first step is to analyse which path the Windows service `icinga2` is using.
1551 Start a new administrative Powershell and ensure that the `icinga2` service is running.
1554 C:\Program Files\ICINGA2\sbin> net start icinga2
1557 Use the `Get-WmiObject` function to extract the windows service and its path name.
1560 C:\Program Files\ICINGA2\sbin> Get-WmiObject win32_service | ?{$_.Name -like '*icinga*'} | select Name, DisplayName, State, PathName
1562 Name DisplayName State PathName
1563 ---- ----------- ----- --------
1564 icinga2 Icinga 2 Running "C:\Program Files\ICINGA2\sbin\icinga2.exe" --scm "daemon"
1567 If you have used the `icinga2.exe` from a different path to enable e.g. the `debuglog` feature,
1568 navigate into `C:\Program Files\ICINGA2\sbin\` and use the correct exe to control the feature set.
1571 #### Windows Agents consuming 100% CPU <a id="troubleshooting-cluster-windows-agents-cpu"></a>
1575 > The network stack was rewritten in 2.11. This fixes several hanging connections and threads
1576 > on older Windows agents and master/satellite nodes. Prior to testing the below, plan an upgrade.
1578 Icinga 2 requires the `NodeName` [constant](17-language-reference.md#constants) in various places to run.
1579 This includes loading the TLS certificates, setting the proper check source,
1582 Typically the Windows setup wizard and also the CLI commands populate the [constants.conf](04-configuration.md#constants-conf)
1583 file with the auto-detected or user-provided FQDN/Common Name.
1585 If this constant is not set during startup, Icinga will try to resolve the
1586 FQDN, if that fails, fetch the hostname. If everything fails, it logs
1587 an error and sets this to `localhost`. This results in undefined behaviour
1588 if ignored by the admin.
1590 Querying the DNS when not reachable is CPU consuming, and may look like Icinga
1591 is doing lots of checks, etc. but actually really is just starting up.
1593 In order to fix this, edit the `constants.conf` file and populate
1594 the `NodeName` constant with the FQDN. Ensure this is the same value
1595 as the local endpoint object name.
1598 const NodeName = "windows-agent1.domain.com"
1603 #### Windows blocking Icinga 2 with ephemeral port range <a id="troubleshooting-cluster-windows-agents-ephemeral-port-range"></a>
1605 When you see a message like this in your Windows agent logs:
1608 critical/TcpSocket: Invalid socket: 10055, "An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full."
1611 Windows is blocking Icinga 2 and as such, no more TCP connection handling is possible.
1613 Depending on the version, patch level and installed applications, Windows is changing its
1614 range of [ephemeral ports](https://en.wikipedia.org/wiki/Ephemeral_port#Range).
1616 In order to solve this, raise the the `MaxUserPort` value in the registry.
1619 HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
1621 Value Name: MaxUserPort Value
1626 More details in [this blogpost](https://www.netways.de/blog/2019/01/24/windows-blocking-icinga-2-with-ephemeral-port-range/)
1627 and this [MS help entry](https://support.microsoft.com/en-us/help/196271/when-you-try-to-connect-from-tcp-ports-greater-than-5000-you-receive-t).