1 # <a id="troubleshooting"></a> Icinga 2 Troubleshooting
3 ## <a id="troubleshooting-information-required"></a> Which information is required
5 * Run `icinga2 troubleshoot` to collect required troubleshooting information
6 * Alternative, manual steps:
8 * `icinga2 feature list`
9 * `icinga2 daemon --validate`
10 * Relevant output from your main and debug log ( `icinga2 object list --type='filelogger'` )
11 * The newest Icinga 2 crash log, if relevant
12 * Your icinga2.conf and, if you run multiple Icinga 2 instances, your zones.conf
13 * How was Icinga 2 installed (and which repository in case) and which distribution are you using
14 * Provide complete configuration snippets explaining your problem in detail
15 * If the check command failed - what's the output of your manual plugin tests?
16 * In case of [debugging](21-debug.md#debug) Icinga 2, the full back traces and outputs
18 ## <a id="troubleshooting-enable-debug-output"></a> Enable Debug Output
20 Run Icinga 2 in the foreground with debugging enabled. Specify the console
21 log severity as an additional parameter argument to `-x`.
23 # /usr/sbin/icinga2 daemon -x notice
25 The log level can be one of `critical`, `warning`, `information`, `notice`
28 Alternatively you can enable the debug log:
30 # icinga2 feature enable debuglog
31 # service icinga2 restart
33 You can find the debug log file in `/var/log/icinga2/debug.log`.
35 ## <a id="list-configuration-objects"></a> List Configuration Objects
37 The `icinga2 object list` CLI command can be used to list all configuration objects and their
38 attributes. The tool also shows where each of the attributes was modified.
40 That way you can also identify which objects have been created from your [apply rules](19-language-reference.md#apply).
44 Object 'localhost!ssh' of type 'Service':
45 * __name = 'localhost!ssh'
46 * check_command = 'ssh'
47 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 5:3-5:23
49 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 24:3-24:21
50 * host_name = 'localhost'
51 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 4:3-4:25
52 * max_check_attempts = 3
53 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 23:3-23:24
56 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 25:3-25:22
57 * templates = [ 'ssh', 'generic-service' ]
58 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 1:0-7:1
59 % += modified in '/etc/icinga2/conf.d/templates.conf', lines 22:1-26:1
62 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
64 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
68 You can also filter by name and type:
70 # icinga2 object list --name *ssh* --type Service
71 Object 'localhost!ssh' of type 'Service':
72 * __name = 'localhost!ssh'
73 * check_command = 'ssh'
74 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 5:3-5:23
76 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 24:3-24:21
77 * host_name = 'localhost'
78 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 4:3-4:25
79 * max_check_attempts = 3
80 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 23:3-23:24
83 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 25:3-25:22
84 * templates = [ 'ssh', 'generic-service' ]
85 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 1:0-7:1
86 % += modified in '/etc/icinga2/conf.d/templates.conf', lines 22:1-26:1
89 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
91 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
93 Found 1 Service objects.
95 [2014-10-15 14:27:19 +0200] information/cli: Parsed 175 objects.
97 ## <a id="check-command-definitions"></a> Where are the check command definitions?
99 Icinga 2 features a number of built-in [check command definitions](7-icinga-template-library.md#plugin-check-commands) which are
105 in the [icinga2.conf](5-configuring-icinga-2.md#icinga2-conf) configuration file. These files are not considered configuration files and will be overridden
106 on upgrade, so please send modifications as proposed patches upstream. The default include path is set to
107 `LocalStateDir + "/share/icinga2/includes"`.
109 You should add your own command definitions to a new file in `conf.d/` called `commands.conf`
112 ## <a id="checks-not-executed"></a> Checks are not executed
114 * Check the debug log to see if the check command gets executed
115 * Verify that failed depedencies do not prevent command execution
116 * Make sure that the plugin is executable by the Icinga 2 user (run a manual test)
117 * Make sure the [checker](8-cli-commands.md#features) feature is enabled.
121 # sudo -u icinga /usr/lib/nagios/plugins/check_ping -4 -H 127.0.0.1 -c 5000,100% -w 3000,80%
123 # icinga2 feature enable checker
124 The feature 'checker' is already enabled.
127 ## <a id="notifications-not-sent"></a> Notifications are not sent
129 * Check the debug log to see if a notification is triggered
130 * If yes, verify that all conditions are satisfied
131 * Are any errors on the notification command execution logged?
133 Verify the following configuration
135 * Is the host/service `enable_notifications` attribute set, and if, to which value?
136 * Do the notification attributes `states`, `types`, `period` match the notification conditions?
137 * Do the user attributes `states`, `types`, `period` match the notification conditions?
138 * Are there any notification `begin` and `end` times configured?
139 * Make sure the [notification](8-cli-commands.md#features) feature is enabled.
140 * Does the referenced NotificationCommand work when executed as Icinga user on the shell?
142 If notifications are to be sent via mail make sure that the mail program specified exists.
143 The name and location depends on the distribution so the preconfigured setting might have to be
144 changed on your system.
148 # icinga2 feature enable notification
149 The feature 'notification' is already enabled.
151 ## <a id="feature-not-working"></a> Feature is not working
153 * Make sure that the feature configuration is enabled by symlinking from `features-available/`
154 to `features-enabled` and that the latter is included in [icinga2.conf](5-configuring-icinga-2.md#icinga2-conf).
155 * Are the feature attributes set correctly according to the documentation?
156 * Any errors on the logs?
158 ## <a id="configuration-ignored"></a> Configuration is ignored
160 * Make sure that the line(s) are not [commented out](19-language-reference.md#comments) (starting with `//` or `#`, or
161 encapsulated by `/* ... */`).
162 * Is the configuration file included in [icinga2.conf](5-configuring-icinga-2.md#icinga2-conf)?
164 ## <a id="configuration-attribute-inheritance"></a> Configuration attributes are inherited from
166 Icinga 2 allows you to import templates using the [import](19-language-reference.md#template-imports) keyword. If these templates
167 contain additional attributes, your objects will automatically inherit them. You can override
168 or modify these attributes in the current object.
170 ## <a id="troubleshooting-cluster"></a> Cluster Troubleshooting
172 You should configure the [cluster health checks](12-distributed-monitoring-ha.md#cluster-health-check) if you haven't
177 > Some problems just exist due to wrong file permissions or packet filters applied. Make
178 > sure to check these in the first place.
180 ### <a id="troubleshooting-cluster-connection-errors"></a> Cluster Troubleshooting Connection Errors
182 General connection errors normally lead you to one of the following problems:
184 * Wrong network configuration
185 * Packet loss on the connection
186 * Firewall rules preventing traffic
188 Use tools like `netstat`, `tcpdump`, `nmap`, etc to make sure that the cluster communication
189 happens (default port is `5665`).
191 # tcpdump -n port 5665 -i any
193 # netstat -tulpen | grep icinga
195 # nmap yourclusternode.localdomain
197 ### <a id="troubleshooting-cluster-ssl-errors"></a> Cluster Troubleshooting SSL Errors
199 If the cluster communication fails with cryptic SSL error messages, make sure to check
202 * File permissions on the SSL certificate files
203 * Does the used CA match for all cluster endpoints?
207 # ls -la /etc/icinga2/pki
210 ### <a id="troubleshooting-cluster-message-errors"></a> Cluster Troubleshooting Message Errors
212 At some point, when the network connection is broken or gone, the Icinga 2 instances
213 will be disconnected. If the connection can't be re-established between zones and endpoints,
214 they remain in a Split-Brain-mode and history may differ.
216 Although the Icinga 2 cluster protocol stores historical events in a replay log for later synchronisation,
217 you should make sure to check why the network connection failed.
219 ### <a id="troubleshooting-cluster-config-sync"></a> Cluster Troubleshooting Config Sync
221 If the cluster zones do not sync their configuration, make sure to check the following:
223 * Within a config master zone, only one configuration master is allowed to have its config in `/etc/icinga2/zones.d`.
224 ** The master syncs the configuration to `/var/lib/icinga2/api/zones/` during startup and only syncs valid configuration to the other nodes
225 ** The other nodes receive the configuration into `/var/lib/icinga2/api/zones/`
226 * The `icinga2.log` log file will indicate whether this ApiListener [accepts config](12-distributed-monitoring-ha.md#zone-config-sync-permissions), or not