1 # Icinga 2 Troubleshooting <a id="troubleshooting"></a>
3 ## Required Information <a id="troubleshooting-information-required"></a>
5 Please ensure to provide any detail which may help reproduce and understand your issue.
6 Whether you ask on the community channels or you create an issue at [GitHub](https://github.com/Icinga), make sure
7 that others can follow your explanations. If necessary, draw a picture and attach it for
8 better illustration. This is especially helpful if you are troubleshooting a distributed
11 We've come around many community questions and compiled this list. Add your own
12 findings and details please.
14 * Describe the expected behavior in your own words.
15 * Describe the actual behavior in one or two sentences.
16 * Ensure to provide general information such as:
17 * How was Icinga 2 installed (and which repository in case) and which distribution are you using
19 * `icinga2 feature list`
21 * [Icinga Web 2](https://www.icinga.com/products/icinga-web-2/) version (screenshot from System - About)
22 * [Icinga Web 2 modules](https://www.icinga.com/products/icinga-web-2-modules/) e.g. the Icinga Director (optional)
23 * Configuration insights:
24 * Provide complete configuration snippets explaining your problem in detail
25 * Your [icinga2.conf](04-configuring-icinga-2.md#icinga2-conf) file
26 * If you run multiple Icinga 2 instances, the [zones.conf](04-configuring-icinga-2.md#zones-conf) file (or `icinga2 object list --type Endpoint` and `icinga2 object list --type Zone`) from all affected nodes.
28 * Relevant output from your main and [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) in `/var/log/icinga2`. Please add step-by-step explanations with timestamps if required.
29 * The newest Icinga 2 crash log if relevant, located in `/var/log/icinga2/crash`
31 * If the check command failed, what's the output of your manual plugin tests?
32 * In case of [debugging](20-development.md#development) Icinga 2, the full back traces and outputs
34 ## Analyze your Environment <a id="troubleshooting-analyze-environment"></a>
36 There are many components involved on a server running Icinga 2. When you
37 analyze a problem, keep in mind that basic system administration knowledge
38 is also key to identify bottlenecks and issues.
42 > [Monitor Icinga 2](08-advanced-topics.md#monitoring-icinga) and use the hints for further analysis.
44 * Analyze the system's performance and dentify bottlenecks and issues.
45 * Collect details about all applications (e.g. Icinga 2, MySQL, Apache, Graphite, Elastic, etc.).
46 * If data is exchanged via network (e.g. central MySQL cluster) ensure to monitor the bandwidth capabilities too.
47 * Add graphs and screenshots to your issue description
49 Install tools which help you to do so. Opinions differ, let us know if you have any additions here!
51 ### Analyse your Linux/Unix Environment <a id="troubleshooting-analyze-environment-linux"></a>
53 [htop](https://hisham.hm/htop/) is a better replacement for `top` and helps to analyze processes
61 If you are for example experiencing performance issues, open `htop` and take a screenshot.
62 Add it to your question and/or bug report.
64 Analyse disk I/O performance in Grafana, take a screenshot and obfuscate any sensitive details.
65 Attach it when posting a question to the community channels.
67 The [sysstat](https://github.com/sysstat/sysstat) package provides a number of tools to
68 analyze the performance on Linux. On FreeBSD you could use `systat` for example.
72 apt-get install sysstat
75 Example for `vmstat` (summary of memory, processes, etc.):
80 // print timestamps, format in MB, stats every 1 second, 5 times
98 `sysstat` also provides the `iostat` binary. On FreeBSD you could use `systat` for example.
100 If you are missing checks and metrics found in your analysis, add them to your monitoring!
102 ### Analyze your Windows Environment <a id="troubleshooting-analyze-environment-windows"></a>
104 A good tip for Windows are the tools found inside the [Sysinternals Suite](https://technet.microsoft.com/en-us/sysinternals/bb842062.aspx).
106 You can also start `perfmon` and analyze specific performance counters.
107 Keep notes which could be important for your monitoring, and add service
110 ## Enable Debug Output <a id="troubleshooting-enable-debug-output"></a>
112 ### Enable Debug Output on Linux/Unix <a id="troubleshooting-enable-debug-output-linux"></a>
114 Enable the `debuglog` feature:
116 # icinga2 feature enable debuglog
117 # service icinga2 restart
119 The debug log file can be found in `/var/log/icinga2/debug.log`.
121 Alternatively you may run Icinga 2 in the foreground with debugging enabled. Specify the console
122 log severity as an additional parameter argument to `-x`.
124 # /usr/sbin/icinga2 daemon -x notice
126 The [log severity](09-object-types.md#objecttype-filelogger) can be one of `critical`, `warning`, `information`, `notice`
129 ### Enable Debug Output on Windows <a id="troubleshooting-enable-debug-output-windows"></a>
131 Open a command prompt with administrative privileges and enable the debug log feature.
133 C:> icinga2.exe enable feature debuglog
135 Ensure that the Icinga 2 service already writes the main log into `C:\ProgramData\icinga2\var\log\icinga2`.
136 Restart the Icinga 2 service and open the newly created `debug.log` file.
139 C:> net start icinga2
141 ## List Configuration Objects <a id="list-configuration-objects"></a>
143 The `icinga2 object list` CLI command can be used to list all configuration objects and their
144 attributes. The tool also shows where each of the attributes was modified.
148 > Use the Icinga 2 API to access [config objects at runtime](12-icinga2-api.md#icinga2-api-config-objects) directly.
150 That way you can also identify which objects have been created from your [apply rules](17-language-reference.md#apply).
152 # icinga2 object list
154 Object 'localhost!ssh' of type 'Service':
155 * __name = 'localhost!ssh'
156 * check_command = 'ssh'
157 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 5:3-5:23
158 * check_interval = 60
159 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 24:3-24:21
160 * host_name = 'localhost'
161 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 4:3-4:25
162 * max_check_attempts = 3
163 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 23:3-23:24
165 * retry_interval = 30
166 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 25:3-25:22
167 * templates = [ 'ssh', 'generic-service' ]
168 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 1:0-7:1
169 % += modified in '/etc/icinga2/conf.d/templates.conf', lines 22:1-26:1
172 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
174 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
178 You can also filter by name and type:
180 # icinga2 object list --name *ssh* --type Service
181 Object 'localhost!ssh' of type 'Service':
182 * __name = 'localhost!ssh'
183 * check_command = 'ssh'
184 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 5:3-5:23
185 * check_interval = 60
186 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 24:3-24:21
187 * host_name = 'localhost'
188 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 4:3-4:25
189 * max_check_attempts = 3
190 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 23:3-23:24
192 * retry_interval = 30
193 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 25:3-25:22
194 * templates = [ 'ssh', 'generic-service' ]
195 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 1:0-7:1
196 % += modified in '/etc/icinga2/conf.d/templates.conf', lines 22:1-26:1
199 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
201 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
203 Found 1 Service objects.
205 [2014-10-15 14:27:19 +0200] information/cli: Parsed 175 objects.
207 Runtime modifications via the [REST API](12-icinga2-api.md#icinga2-api-config-objects)
208 are not immediately updated. Furthermore there is a known issue with
209 [group assign expressions](17-language-reference.md#group-assign) which are not reflected in the host object output.
210 You need to restart Icinga 2 in order to update the `icinga2.debug` cache file.
213 ## Where are the check command definitions? <a id="check-command-definitions"></a>
215 Icinga 2 features a number of built-in [check command definitions](10-icinga-template-library.md#icinga-template-library) which are
221 in the [icinga2.conf](04-configuring-icinga-2.md#icinga2-conf) configuration file. These files are not considered configuration files and will be overridden
222 on upgrade, so please send modifications as proposed patches upstream. The default include path is set to
223 `LocalStateDir + "/share/icinga2/includes"`.
225 You should add your own command definitions to a new file in `conf.d/` called `commands.conf`
228 ## Checks <a id="troubleshooting-checks"></a>
230 ### Executed Command for Checks <a id="checks-executed-command"></a>
232 * Use the Icinga 2 API to [query](12-icinga2-api.md#icinga2-api-config-objects-query) host/service objects
233 for their check result containing the executed shell command.
234 * Use the Icinga 2 [console cli command](11-cli-commands.md#cli-command-console)
235 to fetch the checkable object, its check result and the executed shell command.
236 * Alternatively enable the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) and look for the executed command.
238 Example for a service object query using a [regex match](18-library-reference.md#global-functions-regex)
241 $ curl -k -s -u root:icinga -H 'Accept: application/json' -H 'X-HTTP-Method-Override: GET' -X POST 'https://localhost:5665/v1/objects/services' \
242 -d '{ "filter": "regex(pattern, service.name)", "filter_vars": { "pattern": "^http" }, "attrs": [ "__name", "last_check_result" ] }' | python -m json.tool
247 "__name": "example.localdomain!http",
248 "last_check_result": {
250 "check_source": "example.localdomain",
252 "/usr/local/sbin/check_http",
265 "name": "example.localdomain!http",
271 Example for using the `icinga2 console` CLI command evaluation functionality:
273 $ ICINGA2_API_PASSWORD=icinga icinga2 console --connect 'https://root@localhost:5665/' \
274 --eval 'get_service("example.localdomain", "http").last_check_result.command' | python -m json.tool
276 "/usr/local/sbin/check_http",
283 Example for searching the debug log:
285 # icinga2 feature enable debuglog
286 # systemctl restart icinga2
287 # tail -f /var/log/icinga2/debug.log | grep "notice/Process"
290 ### Checks are not executed <a id="checks-not-executed"></a>
292 * Check the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) to see if the check command gets executed.
293 * Verify that failed depedencies do not prevent command execution.
294 * Make sure that the plugin is executable by the Icinga 2 user (run a manual test).
295 * Make sure the [checker](11-cli-commands.md#enable-features) feature is enabled.
296 * Use the Icinga 2 API [event streams](12-icinga2-api.md#icinga2-api-event-streams) to receive live check result streams.
300 # sudo -u icinga /usr/lib/nagios/plugins/check_ping -4 -H 127.0.0.1 -c 5000,100% -w 3000,80%
302 # icinga2 feature enable checker
303 The feature 'checker' is already enabled.
305 Fetch all check result events matching the `event.service` name `random`:
307 $ curl -k -s -u root:icinga -X POST 'https://localhost:5665/v1/events?queue=debugchecks&types=CheckResult&filter=match%28%22random*%22,event.service%29'
310 ### Check Fork Errors <a id="check-fork-errors"></a>
312 We've learned that newer kernel versions introduce a [fork limit for cgroups](https://lwn.net/Articles/663873/)
313 which is enabled in SLES 12 SP2+ for example. The default value
314 for `DefaultTasksMax` in Systemd is set to `512`.
316 Icinga 2 relies on forking child processes to execute commands
317 and might therefore hit this limit in larger setups.
319 The error message could look like this:
321 2017-01-12T11:55:40.742685+01:00 icinga2-master1 kernel: [65567.582895] cgroup: fork rejected by pids controller in /system.slice/icinga2.service
323 In order to solve the problem, increase the value for `DefaultTasksMax`
324 or set it to `infinity`:
327 [root@icinga2-master1.localdomain /]# cp /usr/lib/systemd/system/icinga2.service /etc/systemd/system/icinga2.service
328 [root@icinga2-master1.localdomain /]# vim /etc/systemd/system/icinga2.service
332 DefaultTasksMax=infinity
334 [root@icinga2-master1.localdomain /]# systemctl daemon-reload
335 [root@icinga2-master1.localdomain /]# systemctl restart icinga2
337 Please note that this setting is available since Systemd version 226.
341 > Future versions of Icinga 2 will add the setting as default.
343 ### Late Check Results <a id="late-check-results"></a>
345 [Icinga Web 2](https://www.icinga.com/products/icinga-web-2/) provides
346 a dashboard overview for `overdue checks`.
348 The REST API provides the [status] URL endpoint with some generic metrics
349 on Icinga and its features.
351 # curl -k -s -u root:icinga 'https://localhost:5665/v1/status' | python -m json.tool | less
353 You can also calculate late check results via the REST API:
355 * Fetch the `last_check` timestamp from each object
356 * Compare the timestamp with the current time and add `check_interval` multiple times (change it to see which results are really late, like five times check_interval)
358 You can use the [icinga2 console](11-cli-commands.md#cli-command-console) to connect to the instance, fetch all data
359 and calculate the differences. More infos can be found in [this blogpost](https://www.icinga.com/2016/08/11/analyse-icinga-2-problems-using-the-console-api/).
361 # ICINGA2_API_USERNAME=root ICINGA2_API_PASSWORD=icinga icinga2 console --connect 'https://localhost:5665/'
363 <1> => var res = []; for (s in get_objects(Service).filter(s => s.last_check < get_time() - 2 * s.check_interval)) { res.add([s.__name, DateTime(s.last_check).to_string()]) }; res
365 [ [ "10807-host!10807-service", "2016-06-10 15:54:55 +0200" ], [ "mbmif.int.netways.de!disk /", "2016-01-26 16:32:29 +0100" ] ]
367 Or if you are just interested in numbers, call [len](18-library-reference.md#array-len) on the result array `res`:
369 <2> => var res = []; for (s in get_objects(Service).filter(s => s.last_check < get_time() - 2 * s.check_interval)) { res.add([s.__name, DateTime(s.last_check).to_string()]) }; res.len()
373 If you need to analyze that problem multiple times, just add the current formatted timestamp
374 and repeat the commands.
376 <23> => DateTime(get_time()).to_string()
378 "2017-04-04 16:09:39 +0200"
380 <24> => var res = []; for (s in get_objects(Service).filter(s => s.last_check < get_time() - 2 * s.check_interval)) { res.add([s.__name, DateTime(s.last_check).to_string()]) }; res.len()
384 More details about the Icinga 2 DSL and its possibilities can be
385 found in the [language](17-language-reference.md#language-reference) and [library](18-library-reference.md#library-reference) reference chapters.
387 ### Late Check Results in Distributed Environments <a id="late-check-results-distributed"></a>
389 When it comes to a distributed HA setup, each node is responsible for a load-balanced amount of checks.
390 Host and Service objects provide the attribute `paused`. If this is set to `false`, the current node
391 actively attempts to schedule and execute checks. Otherwise the node does not feel responsible.
393 <3> => var res = {}; for (s in get_objects(Service).filter(s => s.last_check < get_time() - 2 * s.check_interval)) { res[s.paused] += 1 }; res
399 You may ask why this analysis is important? Fair enough - if the numbers are not inverted in a HA zone
400 with two members, this may give a hint that the cluster nodes are in a split-brain scenario, or you've
401 found a bug in the cluster.
404 If you are running a cluster setup where the master/satellite executes checks on the client via
405 [top down command endpoint](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint) mode,
406 you might want to know which zones are affected.
408 This analysis assumes that clients which are not connected, have the string `connected` in their
409 service check result output and their state is `UNKNOWN`.
411 <4> => var res = {}; for (s in get_objects(Service)) { if (s.state==3) { if (match("*connected*", s.last_check_result.output)) { res[s.zone] += [s.host_name] } } }; for (k => v in res) { res[k] = len(v.unique()) }; res
419 The result set shows the configured zones and their affected hosts in a unique list. The output also just prints the numbers
420 but you can adjust this by omitting the `len()` call inside the for loop.
422 ## Notifications are not sent <a id="notifications-not-sent"></a>
424 * Check the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) to see if a notification is triggered.
425 * If yes, verify that all conditions are satisfied.
426 * Are any errors on the notification command execution logged?
428 Please ensure to add these details with your own description
429 to any question or issue posted to the community channels.
431 Verify the following configuration:
433 * Is the host/service `enable_notifications` attribute set, and if so, to which value?
434 * Do the [notification](09-object-types.md#objecttype-notification) attributes `states`, `types`, `period` match the notification conditions?
435 * Do the [user](09-object-types.md#objecttype-user) attributes `states`, `types`, `period` match the notification conditions?
436 * Are there any notification `begin` and `end` times configured?
437 * Make sure the [notification](11-cli-commands.md#enable-features) feature is enabled.
438 * Does the referenced NotificationCommand work when executed as Icinga user on the shell?
440 If notifications are to be sent via mail, make sure that the mail program specified inside the
441 [NotificationCommand object](09-object-types.md#objecttype-notificationcommand) exists.
442 The name and location depends on the distribution so the preconfigured setting might have to be
443 changed on your system.
448 # icinga2 feature enable notification
449 The feature 'notification' is already enabled.
451 You can use the Icinga 2 API [event streams](12-icinga2-api.md#icinga2-api-event-streams) to receive live notification streams:
453 $ curl -k -s -u root:icinga -X POST 'https://localhost:5665/v1/events?queue=debugnotifications&types=Notification'
456 ## Feature is not working <a id="feature-not-working"></a>
458 * Make sure that the feature configuration is enabled by symlinking from `features-available/`
459 to `features-enabled` and that the latter is included in [icinga2.conf](04-configuring-icinga-2.md#icinga2-conf).
460 * Are the feature attributes set correctly according to the documentation?
461 * Any errors on the logs?
463 Look up the [object type](09-object-types.md#object-types) for the required feature and verify it is enabled:
465 # icinga2 object list --type <feature object type>
467 Example for the `graphite` feature:
469 # icinga2 object list --type GraphiteWriter
471 ## Configuration is ignored <a id="configuration-ignored"></a>
473 * Make sure that the line(s) are not [commented out](17-language-reference.md#comments) (starting with `//` or `#`, or
474 encapsulated by `/* ... */`).
475 * Is the configuration file included in [icinga2.conf](04-configuring-icinga-2.md#icinga2-conf)?
477 Run the [configuration validation](11-cli-commands.md#config-validation) and add `notice` as log severity.
478 Search for the file which should be included i.e. using the `grep` CLI command.
480 # icinga2 daemon -C -x notice | grep command
482 ## Configuration attributes are inherited from <a id="configuration-attribute-inheritance"></a>
484 Icinga 2 allows you to import templates using the [import](17-language-reference.md#template-imports) keyword. If these templates
485 contain additional attributes, your objects will automatically inherit them. You can override
486 or modify these attributes in the current object.
488 The [object list](15-troubleshooting.md#list-configuration-objects) CLI command allows you to verify the attribute origin.
490 ## Configuration Value with Single Dollar Sign <a id="configuration-value-dollar-sign"></a>
492 In case your configuration validation fails with a missing closing dollar sign error message, you
493 did not properly escape the single dollar sign preventing its usage as [runtime macro](03-monitoring-basics.md#runtime-macros).
495 critical/config: Error: Validation failed for Object 'ping4' (Type: 'Service') at /etc/icinga2/zones.d/global-templates/windows.conf:24: Closing $ not found in macro format string 'top-syntax=${list}'.
497 Correct the custom attribute value to
499 "top-syntax=$${list}"
501 ## Cluster and Clients Troubleshooting <a id="troubleshooting-cluster"></a>
503 This applies to any Icinga 2 node in a [distributed monitoring setup](06-distributed-monitoring.md#distributed-monitoring-scenarios).
505 You should configure the [cluster health checks](06-distributed-monitoring.md#distributed-monitoring-health-checks) if you haven't
510 > Some problems just exist due to wrong file permissions or applied packet filters. Make
511 > sure to check these in the first place.
513 ### Cluster Troubleshooting Connection Errors <a id="troubleshooting-cluster-connection-errors"></a>
515 General connection errors could be one of the following problems:
517 * Incorrect network configuration
519 * Firewall rules preventing traffic
521 Use tools like `netstat`, `tcpdump`, `nmap`, etc. to make sure that the cluster communication
522 works (default port is `5665`).
524 # tcpdump -n port 5665 -i any
526 # netstat -tulpen | grep icinga
528 # nmap yourclusternode.localdomain
530 ### Cluster Troubleshooting SSL Errors <a id="troubleshooting-cluster-ssl-errors"></a>
532 If the cluster communication fails with SSL error messages, make sure to check
535 * File permissions on the SSL certificate files
536 * Does the used CA match for all cluster endpoints?
537 * Verify the `Issuer` being your trusted CA
538 * Verify the `Subject` containing your endpoint's common name (CN)
539 * Check the validity of the certificate itself
541 Steps on the client `icinga2-node2.localdomain`:
543 # ls -la /etc/icinga2/pki
545 # cd /etc/icinga2/pki/
546 # openssl x509 -in icinga2-node2.localdomain.crt -text
550 Serial Number: 2 (0x2)
551 Signature Algorithm: sha1WithRSAEncryption
552 Issuer: C=DE, ST=Bavaria, L=Nuremberg, O=NETWAYS GmbH, OU=Monitoring, CN=Icinga CA
554 Not Before: Jan 7 13:17:38 2014 GMT
555 Not After : Jan 5 13:17:38 2024 GMT
556 Subject: C=DE, ST=Bavaria, L=Nuremberg, O=NETWAYS GmbH, OU=Monitoring, CN=icinga2-node2.localdomain
557 Subject Public Key Info:
558 Public Key Algorithm: rsaEncryption
559 Public-Key: (4096 bit)
563 Try to manually connect from `icinga2-node2.localdomain` to the master node `icinga2-node1.localdomain`:
565 # openssl s_client -CAfile /etc/icinga2/pki/ca.crt -cert /etc/icinga2/pki/icinga2-node2.localdomain.crt -key /etc/icinga2/pki/icinga2-node2.localdomain.key -connect icinga2-node1.localdomain:5665
571 If the connection attempt fails or your CA does not match, [verify the master and client certificates](15-troubleshooting.md#troubleshooting-cluster-ssl-certificate-verification).
573 #### Cluster Troubleshooting Unauthenticated Clients <a id="troubleshooting-cluster-unauthenticated-clients"></a>
575 Unauthenticated nodes are able to connect. This is required for client setups.
579 [2015-07-13 18:29:25 +0200] information/ApiListener: New client connection for identity 'icinga-client' (unauthenticated)
581 Client as command execution bridge:
583 [2015-07-13 18:29:26 +1000] notice/ApiEvents: Discarding 'execute command' message from 'icinga-master': Invalid endpoint origin (client not allowed).
585 If these messages do not go away, make sure to [verify the master and client certificates](15-troubleshooting.md#troubleshooting-cluster-ssl-certificate-verification).
587 #### Cluster Troubleshooting SSL Certificate Verification <a id="troubleshooting-cluster-ssl-certificate-verification"></a>
589 Make sure to verify the client's certificate and its received `ca.crt` in `/etc/icinga2/pki` and ensure that
590 both instances are signed by the **same CA**.
592 # openssl verify -verbose -CAfile /etc/icinga2/pki/ca.crt /etc/icinga2/pki/icinga2-node1.localdomain.crt
593 icinga2-node1.localdomain.crt: OK
595 # openssl verify -verbose -CAfile /etc/icinga2/pki/ca.crt /etc/icinga2/pki/icinga2-node2.localdomain.crt
596 icinga2-node2.localdomain.crt: OK
598 Fetch the `ca.crt` file from the client node and compare it to your master's `ca.crt` file:
600 # scp icinga2-node2:/etc/icinga2/pki/ca.crt test-client-ca.crt
601 # diff -ur /etc/icinga2/pki/ca.crt test-client-ca.crt
603 On SLES11 you'll need to use the `openssl1` command instead of `openssl`.
605 ### Cluster Troubleshooting Message Errors <a id="troubleshooting-cluster-message-errors"></a>
607 At some point, when the network connection is broken or gone, the Icinga 2 instances
608 will be disconnected. If the connection can't be re-established between endpoints in the same HA zone,
609 they remain in a Split-Brain-mode and history may differ.
611 Although the Icinga 2 cluster protocol stores historical events in a [replay log](15-troubleshooting.md#troubleshooting-cluster-replay-log)
612 for later synchronisation, you should make sure to check why the network connection failed.
614 ### Cluster Troubleshooting Command Endpoint Errors <a id="troubleshooting-cluster-command-endpoint-errors"></a>
616 Command endpoints can be used [for clients](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint)
617 as well as inside an [High-Availability cluster](06-distributed-monitoring.md#distributed-monitoring-scenarios).
619 There is no cli command for manually executing the check, but you can verify
620 the following (e.g. by invoking a forced check from the web interface):
622 * `/var/log/icinga2/icinga2.log` contains connection and execution errors.
623 * The ApiListener is not enabled to [accept commands](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint).
624 * `CheckCommand` definition not found on the remote client.
625 * Referenced check plugin not found on the remote client.
626 * Runtime warnings and errors, e.g. unresolved runtime macros or configuration problems.
627 * Specific error messages are also populated into `UNKNOWN` check results including a detailed error message in their output.
628 * Verify the `check_source` object attribute. This is populated by the node executing the check.
629 * More verbose logs are found inside the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output).
631 * Use the Icinga 2 API [event streams](12-icinga2-api.md#icinga2-api-event-streams) to receive live check result streams.
633 Fetch all check result events matching the `event.service` name `remote-client`:
635 $ curl -k -s -u root:icinga -X POST 'https://localhost:5665/v1/events?queue=debugcommandendpoint&types=CheckResult&filter=match%28%22remote-client*%22,event.service%29'
639 ### Cluster Troubleshooting Config Sync <a id="troubleshooting-cluster-config-sync"></a>
641 If the cluster zones do not sync their configuration, make sure to check the following:
643 * Within a config master zone, only one configuration master is allowed to have its config in `/etc/icinga2/zones.d`.
644 ** The master syncs the configuration to `/var/lib/icinga2/api/zones/` during startup and only syncs valid configuration to the other nodes.
645 ** The other nodes receive the configuration into `/var/lib/icinga2/api/zones/`.
646 * The `icinga2.log` log file in `/var/log/icinga2` will indicate whether this ApiListener
647 [accepts config](06-distributed-monitoring.md#distributed-monitoring-top-down-config-sync), or not.
649 Verify the object's [version](09-object-types.md#object-types) attribute on all nodes to
650 check whether the config update and reload was succesful or not.
652 ### Cluster Troubleshooting Overdue Check Results <a id="troubleshooting-cluster-check-results"></a>
654 If your master does not receive check results (or any other events) from the child zones
655 (satellite, clients, etc.), make sure to check whether the client sending in events
658 The [distributed monitoring conventions](06-distributed-monitoring.md#distributed-monitoring-conventions)
659 apply. So, if there's a mismatch between your client node's endpoint name and its provided
660 certificate's CN, the master will deny all events.
664 > [Icinga Web 2](02-getting-started.md#setting-up-icingaweb2) provides a dashboard view
665 > for overdue check results.
667 Enable the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) on the master
668 for more verbose insights.
670 If the client cannot authenticate, it's a more general [problem](15-troubleshooting.md#troubleshooting-cluster-unauthenticated-clients).
672 The client's endpoint is not configured on nor trusted by the master node:
674 Discarding 'check result' message from 'icinga2b': Invalid endpoint origin (client not allowed).
676 The check result message sent by the client does not belong to the zone the checkable object is
679 Discarding 'check result' message from 'icinga2b': Unauthorized access.
682 ### Cluster Troubleshooting Replay Log <a id="troubleshooting-cluster-replay-log"></a>
684 If your `/var/lib/icinga2/api/log` directory grows, it generally means that your cluster
685 cannot replay the log on connection loss and re-establishment. A master node for example
686 will store all events for not connected endpoints in the same and child zones.
690 * All clients are connected? (e.g. [cluster health check](06-distributed-monitoring.md#distributed-monitoring-health-checks)).
691 * Check your [connection](15-troubleshooting.md#troubleshooting-cluster-connection-errors) in general.
692 * Does the log replay work, e.g. are all events processed and the directory gets cleared up over time?
693 * Decrease the `log_duration` attribute value for that specific [endpoint](09-object-types.md#objecttype-endpoint).