1 # <a id="troubleshooting"></a> Icinga 2 Troubleshooting
3 ## <a id="troubleshooting-information-required"></a> Which information is required
5 * Run `icinga2 troubleshoot` to collect required troubleshooting information
6 * Alternative, manual steps:
8 * `icinga2 feature list`
9 * `icinga2 daemon --validate`
10 * Relevant output from your main and debug log ( `icinga2 object list --type='filelogger'` )
11 * The newest Icinga 2 crash log if relevant
12 * Your icinga2.conf and, if you run multiple Icinga 2 instances, your zones.conf
13 * How was Icinga 2 installed (and which repository in case) and which distribution are you using
14 * Provide complete configuration snippets explaining your problem in detail
15 * If the check command failed, what's the output of your manual plugin tests?
16 * In case of [debugging](21-development.md#development) Icinga 2, the full back traces and outputs
18 ## <a id="troubleshooting-enable-debug-output"></a> Enable Debug Output
20 Enable the `debuglog` feature:
22 # icinga2 feature enable debuglog
23 # service icinga2 restart
25 You can find the debug log file in `/var/log/icinga2/debug.log`.
27 Alternatively you may run Icinga 2 in the foreground with debugging enabled. Specify the console
28 log severity as an additional parameter argument to `-x`.
30 # /usr/sbin/icinga2 daemon -x notice
32 The log level can be one of `critical`, `warning`, `information`, `notice`
35 ## <a id="list-configuration-objects"></a> List Configuration Objects
37 The `icinga2 object list` CLI command can be used to list all configuration objects and their
38 attributes. The tool also shows where each of the attributes was modified.
42 > Use the Icinga 2 API to access [config objects at runtime](9-icinga2-api.md#icinga2-api-config-objects) directly.
44 That way you can also identify which objects have been created from your [apply rules](18-language-reference.md#apply).
48 Object 'localhost!ssh' of type 'Service':
49 * __name = 'localhost!ssh'
50 * check_command = 'ssh'
51 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 5:3-5:23
53 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 24:3-24:21
54 * host_name = 'localhost'
55 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 4:3-4:25
56 * max_check_attempts = 3
57 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 23:3-23:24
60 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 25:3-25:22
61 * templates = [ 'ssh', 'generic-service' ]
62 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 1:0-7:1
63 % += modified in '/etc/icinga2/conf.d/templates.conf', lines 22:1-26:1
66 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
68 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
72 You can also filter by name and type:
74 # icinga2 object list --name *ssh* --type Service
75 Object 'localhost!ssh' of type 'Service':
76 * __name = 'localhost!ssh'
77 * check_command = 'ssh'
78 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 5:3-5:23
80 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 24:3-24:21
81 * host_name = 'localhost'
82 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 4:3-4:25
83 * max_check_attempts = 3
84 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 23:3-23:24
87 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 25:3-25:22
88 * templates = [ 'ssh', 'generic-service' ]
89 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 1:0-7:1
90 % += modified in '/etc/icinga2/conf.d/templates.conf', lines 22:1-26:1
93 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
95 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
97 Found 1 Service objects.
99 [2014-10-15 14:27:19 +0200] information/cli: Parsed 175 objects.
101 ## <a id="check-command-definitions"></a> Where are the check command definitions?
103 Icinga 2 features a number of built-in [check command definitions](7-icinga-template-library.md#plugin-check-commands) which are
109 in the [icinga2.conf](4-configuring-icinga-2.md#icinga2-conf) configuration file. These files are not considered configuration files and will be overridden
110 on upgrade, so please send modifications as proposed patches upstream. The default include path is set to
111 `LocalStateDir + "/share/icinga2/includes"`.
113 You should add your own command definitions to a new file in `conf.d/` called `commands.conf`
116 ## <a id="troubleshooting-checks"></a> Checks
118 ### <a id="checks-executed-command"></a> Executed Command for Checks
120 * Use the Icinga 2 API to [query](9-icinga2-api.md#icinga2-api-config-objects-query) host/service objects
121 for their check result containing the executed shell command.
122 * Use the Icinga 2 [console cli command](8-cli-commands.md#cli-command-console)
123 to fetch the checkable object, its check result and the executed shell command.
124 * Alternatively enable the [debug log](16-troubleshooting.md#troubleshooting-enable-debug-output) and look for the executed command.
126 ### <a id="checks-not-executed"></a> Checks are not executed
128 * Check the [debug log](16-troubleshooting.md#troubleshooting-enable-debug-output) to see if the check command gets executed.
129 * Verify that failed depedencies do not prevent command execution.
130 * Make sure that the plugin is executable by the Icinga 2 user (run a manual test).
131 * Make sure the [checker](8-cli-commands.md#enable-features) feature is enabled.
132 * Use the Icinga 2 API [event streams](9-icinga2-api.md#icinga2-api-event-streams) to receive live check result streams.
136 # sudo -u icinga /usr/lib/nagios/plugins/check_ping -4 -H 127.0.0.1 -c 5000,100% -w 3000,80%
138 # icinga2 feature enable checker
139 The feature 'checker' is already enabled.
141 Fetch all check result events matching the `event.service` name `random`:
143 $ curl -k -s -u root:icinga -X POST 'https://localhost:5665/v1/events?queue=debugchecks&types=CheckResult&filter=match%28%22random*%22,event.service%29'
146 ## <a id="notifications-not-sent"></a> Notifications are not sent
148 * Check the debug log to see if a notification is triggered.
149 * If yes, verify that all conditions are satisfied.
150 * Are any errors on the notification command execution logged?
152 Verify the following configuration:
154 * Is the host/service `enable_notifications` attribute set, and if so, to which value?
155 * Do the notification attributes `states`, `types`, `period` match the notification conditions?
156 * Do the user attributes `states`, `types`, `period` match the notification conditions?
157 * Are there any notification `begin` and `end` times configured?
158 * Make sure the [notification](8-cli-commands.md#enable-features) feature is enabled.
159 * Does the referenced NotificationCommand work when executed as Icinga user on the shell?
161 If notifications are to be sent via mail, make sure that the mail program specified inside the
162 [NotificationCommand object](6-object-types.md#objecttype-notificationcommand) exists.
163 The name and location depends on the distribution so the preconfigured setting might have to be
164 changed on your system.
169 # icinga2 feature enable notification
170 The feature 'notification' is already enabled.
172 You can use the Icinga 2 API [event streams](9-icinga2-api.md#icinga2-api-event-streams) to receive live notification streams:
174 $ curl -k -s -u root:icinga -X POST 'https://localhost:5665/v1/events?queue=debugnotifications&types=Notification'
177 ## <a id="feature-not-working"></a> Feature is not working
179 * Make sure that the feature configuration is enabled by symlinking from `features-available/`
180 to `features-enabled` and that the latter is included in [icinga2.conf](4-configuring-icinga-2.md#icinga2-conf).
181 * Are the feature attributes set correctly according to the documentation?
182 * Any errors on the logs?
184 ## <a id="configuration-ignored"></a> Configuration is ignored
186 * Make sure that the line(s) are not [commented out](18-language-reference.md#comments) (starting with `//` or `#`, or
187 encapsulated by `/* ... */`).
188 * Is the configuration file included in [icinga2.conf](4-configuring-icinga-2.md#icinga2-conf)?
190 ## <a id="configuration-attribute-inheritance"></a> Configuration attributes are inherited from
192 Icinga 2 allows you to import templates using the [import](18-language-reference.md#template-imports) keyword. If these templates
193 contain additional attributes, your objects will automatically inherit them. You can override
194 or modify these attributes in the current object.
196 ## <a id="configuration-value-dollar-sign"></a> Configuration Value with Single Dollar Sign
198 In case your configuration validation fails with a missing closing dollar sign error message, you
199 did not properly escape the single dollar sign preventing its usage as [runtime macro](3-monitoring-basics.md#runtime-macros).
201 critical/config: Error: Validation failed for Object 'ping4' (Type: 'Service') at /etc/icinga2/zones.d/global-templates/windows.conf:24: Closing $ not found in macro format string 'top-syntax=${list}'.
204 ## <a id="troubleshooting-cluster"></a> Cluster and Clients Troubleshooting
206 This applies to anything using the cluster protocol:
208 * [Distributed and High-Availability](13-distributed-monitoring-ha.md#distributed-monitoring-high-availability) scenarios
209 * [Remote client](11-icinga2-client.md#icinga2-client-scenarios) scenarios
211 You should configure the [cluster health checks](13-distributed-monitoring-ha.md#cluster-health-check) if you haven't
216 > Some problems just exist due to wrong file permissions or packet filters applied. Make
217 > sure to check these in the first place.
219 ### <a id="troubleshooting-cluster-connection-errors"></a> Cluster Troubleshooting Connection Errors
221 General connection errors normally lead you to one of the following problems:
223 * Wrong network configuration
224 * Packet loss on the connection
225 * Firewall rules preventing traffic
227 Use tools like `netstat`, `tcpdump`, `nmap`, etc. to make sure that the cluster communication
228 happens (default port is `5665`).
230 # tcpdump -n port 5665 -i any
232 # netstat -tulpen | grep icinga
234 # nmap yourclusternode.localdomain
236 ### <a id="troubleshooting-cluster-ssl-errors"></a> Cluster Troubleshooting SSL Errors
238 If the cluster communication fails with SSL error messages, make sure to check
241 * File permissions on the SSL certificate files
242 * Does the used CA match for all cluster endpoints?
243 * Verify the `Issuer` being your trusted CA
244 * Verify the `Subject` containing your endpoint's common name (CN)
245 * Check the validity of the certificate itself
247 Steps on the client `icinga2-node2.localdomain`:
249 # ls -la /etc/icinga2/pki
251 # cd /etc/icinga2/pki/
252 # openssl x509 -in icinga2-node2.localdomain.crt -text
256 Serial Number: 2 (0x2)
257 Signature Algorithm: sha1WithRSAEncryption
258 Issuer: C=DE, ST=Bavaria, L=Nuremberg, O=NETWAYS GmbH, OU=Monitoring, CN=Icinga CA
260 Not Before: Jan 7 13:17:38 2014 GMT
261 Not After : Jan 5 13:17:38 2024 GMT
262 Subject: C=DE, ST=Bavaria, L=Nuremberg, O=NETWAYS GmbH, OU=Monitoring, CN=icinga2-node2.localdomain
263 Subject Public Key Info:
264 Public Key Algorithm: rsaEncryption
265 Public-Key: (4096 bit)
269 Try to manually connect from `icinga2-node2.localdomain` to the master node `icinga2-node1.localdomain`:
271 # openssl s_client -CAfile /etc/icinga2/pki/ca.crt -cert /etc/icinga2/pki/icinga2-node2.localdomain.crt -key /etc/icinga2/pki/icinga2-node2.localdomain.key -connect icinga2-node1.localdomain:5665
277 If the connection attempt fails or your CA does not match, [verify the master and client certificates](16-troubleshooting.md#troubleshooting-cluster-ssl-certificate-verification).
279 #### <a id="troubleshooting-cluster-unauthenticated-clients"></a> Cluster Troubleshooting Unauthenticated Clients
281 Unauthenticated nodes are able to connect required by the
282 [CSR auto-signing](11-icinga2-client.md#csr-autosigning-requirements) functionality.
286 [2015-07-13 18:29:25 +0200] information/ApiListener: New client connection for identity 'icinga-client' (unauthenticated)
288 Client as command execution bridge:
290 [2015-07-13 18:29:26 +1000] notice/ApiEvents: Discarding 'execute command' message from 'icinga-master': Invalid endpoint origin (client not allowed).
292 If these messages do not go away, make sure to [verify the master and client certificates](16-troubleshooting.md#troubleshooting-cluster-ssl-certificate-verification).
294 #### <a id="troubleshooting-cluster-ssl-certificate-verification"></a> Cluster Troubleshooting SSL Certificate Verification
296 Make sure to verify the client's certificate and its received `ca.crt` in `/etc/icinga2/pki` and ensure that
297 both instances are signed by the **same CA**.
299 # openssl verify -verbose -CAfile /etc/icinga2/pki/ca.crt /etc/icinga2/pki/icinga2-node1.localdomain.crt
300 icinga2-node1.localdomain.crt: OK
302 # openssl verify -verbose -CAfile /etc/icinga2/pki/ca.crt /etc/icinga2/pki/icinga2-node2.localdomain.crt
303 icinga2-node2.localdomain.crt: OK
305 Fetch the `ca.crt` file from the client node and compare it to your master's `ca.crt` file:
307 # scp icinga2-node2:/etc/icinga2/pki/ca.crt test-client-ca.crt
308 # diff -ur /etc/icinga2/pki/ca.crt test-client-ca.crt
310 On SLES11 you'll need to use the `openssl1` command instead of `openssl`.
312 ### <a id="troubleshooting-cluster-message-errors"></a> Cluster Troubleshooting Message Errors
314 At some point, when the network connection is broken or gone, the Icinga 2 instances
315 will be disconnected. If the connection can't be re-established between endpoints in the same HA zone,
316 they remain in a Split-Brain-mode and history may differ.
318 Although the Icinga 2 cluster protocol stores historical events in a [replay log](16-troubleshooting.md#troubleshooting-cluster-replay-log)
319 for later synchronisation, you should make sure to check why the network connection failed.
321 ### <a id="troubleshooting-cluster-command-endpoint-errors"></a> Cluster Troubleshooting Command Endpoint Errors
323 Command endpoints can be used for clients acting as [remote command execution bridge](11-icinga2-client.md#icinga2-client-configuration-command-bridge)
324 as well as inside an [High-Availability cluster](13-distributed-monitoring-ha.md#distributed-monitoring-high-availability).
326 There is no cli command for manually executing the check, but you can verify
327 the following (e.g. by invoking a forced check from the web interface):
329 * `/var/log/icinga2/icinga2.log` contains connection and execution errors.
330 * The ApiListener is not enabled to [accept commands](11-icinga2-client.md#icinga2-client-configuration-command-bridge).
331 * `CheckCommand` definition not found on the remote client.
332 * Referenced check plugin not found on the remote client.
333 * Runtime warnings and errors, e.g. unresolved runtime macros or configuration problems.
334 * Specific error messages are also populated into `UNKNOWN` check results including a detailed error message in their output.
335 * Verify the `check_source` object attribute. This is populated by the node executing the check.
336 * More verbose logs are found inside the [debug log](16-troubleshooting.md#troubleshooting-enable-debug-output).
338 * Use the Icinga 2 API [event streams](9-icinga2-api.md#icinga2-api-event-streams) to receive live check result streams.
340 Fetch all check result events matching the `event.service` name `remote-client`:
342 $ curl -k -s -u root:icinga -X POST 'https://localhost:5665/v1/events?queue=debugcommandendpoint&types=CheckResult&filter=match%28%22remote-client*%22,event.service%29'
346 ### <a id="troubleshooting-cluster-config-sync"></a> Cluster Troubleshooting Config Sync
348 If the cluster zones do not sync their configuration, make sure to check the following:
350 * Within a config master zone, only one configuration master is allowed to have its config in `/etc/icinga2/zones.d`.
351 ** The master syncs the configuration to `/var/lib/icinga2/api/zones/` during startup and only syncs valid configuration to the other nodes.
352 ** The other nodes receive the configuration into `/var/lib/icinga2/api/zones/`.
353 * The `icinga2.log` log file in `/var/log/icinga2` will indicate whether this ApiListener
354 [accepts config](13-distributed-monitoring-ha.md#zone-config-sync-permissions), or not.
356 Verify the object's [version](6-object-types.md#object-types) attribute on all nodes to
357 check whether the config update and reload was succesful or not.
359 ### <a id="troubleshooting-cluster-check-results"></a> Cluster Troubleshooting Overdue Check Results
361 If your master does not receive check results (or any other events) from the child zones
362 (satellite, clients, etc.), make sure to check whether the client sending in events
365 The [cluster naming convention](13-distributed-monitoring-ha.md#cluster-naming-convention)
366 applies. So, if there's a mismatch between your client node's endpoint name and its provided
367 certificate's CN, the master will deny all events.
371 > [Icinga Web 2](2-getting-started.md#setting-up-icingaweb2) provides a dashboard view
372 > for overdue check results.
374 Enable the [debug log](16-troubleshooting.md#troubleshooting-enable-debug-output) on the master
375 for more verbose insights.
377 If the client cannot authenticate, it's a more general [problem](16-troubleshooting.md#troubleshooting-cluster-unauthenticated-clients).
379 The client's endpoint is not configured on nor trusted by the master node:
381 Discarding 'check result' message from 'icinga2b': Invalid endpoint origin (client not allowed).
383 The check result message sent by the client does not belong to the zone the checkable object is
386 Discarding 'check result' message from 'icinga2b': Unauthorized access.
389 ### <a id="troubleshooting-cluster-replay-log"></a> Cluster Troubleshooting Replay Log
391 If your `/var/lib/icinga2/api/log` directory grows, it generally means that your cluster
392 cannot replay the log on connection loss and re-establishment. A master node for example
393 will store all events for not connected endpoints in the same and child zones.
397 * All clients are connected? (e.g. [cluster health check](13-distributed-monitoring-ha.md#cluster-health-check)).
398 * Check your [connection](16-troubleshooting.md#troubleshooting-cluster-connection-errors) in general.
399 * Does the log replay work, e.g. are all events processed and the directory gets cleared up over time?
400 * Decrease the `log_duration` attribute value for that specific [endpoint](6-object-types.md#objecttype-endpoint).