1 # <a id="troubleshooting"></a> Icinga 2 Troubleshooting
3 ## <a id="troubleshooting-information-required"></a> Which information is required
5 * Which distribution and version
6 * How was Icinga 2 installed (and which repository in case)
7 * Run `icinga2 --version`
8 * Provide complete configuration snippets explaining your problem in detail
9 * Provide complete logs targetting your problem
10 * If the check command failed - what's the output of your manual plugin tests?
11 * In case of [debugging](12-troubleshooting.md#debug) Icinga 2, the full back traces and outputs
13 ## <a id="troubleshooting-enable-debug-output"></a> Enable Debug Output
15 Run Icinga 2 in the foreground with debugging enabled. Specify the console
16 log severity as an additional parameter argument to `-x`.
18 # /usr/sbin/icinga2 daemon -x notice
20 The log level can be one of `critical`, `warning`, `information`, `notice`
23 Alternatively you can enable the debug log:
25 # icinga2 feature enable debuglog
26 # service icinga2 restart
28 You can find the debug log file in `/var/log/icinga2/debug.log`.
30 ## <a id="list-configuration-objects"></a> List Configuration Objects
32 The `icinga2 object list` CLI command can be used to list all configuration objects and their
33 attributes. The tool also shows where each of the attributes was modified.
35 That way you can also identify which objects have been created from your [apply rules](15-language-reference.md#apply).
39 Object 'localhost!ssh' of type 'Service':
40 * __name = 'localhost!ssh'
41 * check_command = 'ssh'
42 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 5:3-5:23
44 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 24:3-24:21
45 * host_name = 'localhost'
46 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 4:3-4:25
47 * max_check_attempts = 3
48 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 23:3-23:24
51 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 25:3-25:22
52 * templates = [ 'ssh', 'generic-service' ]
53 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 1:0-7:1
54 % += modified in '/etc/icinga2/conf.d/templates.conf', lines 22:1-26:1
57 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
59 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
63 You can also filter by name and type:
65 # icinga2 object list --name *ssh* --type Service
66 Object 'localhost!ssh' of type 'Service':
67 * __name = 'localhost!ssh'
68 * check_command = 'ssh'
69 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 5:3-5:23
71 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 24:3-24:21
72 * host_name = 'localhost'
73 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 4:3-4:25
74 * max_check_attempts = 3
75 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 23:3-23:24
78 % = modified in '/etc/icinga2/conf.d/templates.conf', lines 25:3-25:22
79 * templates = [ 'ssh', 'generic-service' ]
80 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 1:0-7:1
81 % += modified in '/etc/icinga2/conf.d/templates.conf', lines 22:1-26:1
84 % += modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
86 % = modified in '/etc/icinga2/conf.d/hosts/localhost/ssh.conf', lines 6:3-6:19
88 Found 1 Service objects.
90 [2014-10-15 14:27:19 +0200] information/cli: Parsed 175 objects.
92 ## <a id="check-command-definitions"></a> Where are the check command definitions?
94 Icinga 2 features a number of built-in [check command definitions](6-icinga-template-library.md#plugin-check-commands) which are
100 in the [icinga2.conf](4-configuring-icinga-2.md#icinga2-conf) configuration file. These files are not considered configuration files and will be overridden
101 on upgrade, so please send modifications as proposed patches upstream. The default include path is set to
102 `LocalStateDir + "/share/icinga2/includes"`.
104 You should add your own command definitions to a new file in `conf.d/` called `commands.conf`
107 ## <a id="checks-not-executed"></a> Checks are not executed
109 * Check the debug log to see if the check command gets executed
110 * Verify that failed depedencies do not prevent command execution
111 * Make sure that the plugin is executable by the Icinga 2 user (run a manual test)
112 * Make sure the [checker](7-cli-commands.md#features) feature is enabled.
116 # sudo -u icinga /usr/lib/nagios/plugins/check_ping -4 -H 127.0.0.1 -c 5000,100% -w 3000,80%
118 # icinga2 feature enable checker
119 The feature 'checker' is already enabled.
122 ## <a id="notifications-not-sent"></a> Notifications are not sent
124 * Check the debug log to see if a notification is triggered
125 * If yes, verify that all conditions are satisfied
126 * Are any errors on the notification command execution logged?
128 Verify the following configuration
130 * Is the host/service `enable_notifications` attribute set, and if, to which value?
131 * Do the notification attributes `states`, `types`, `period` match the notification conditions?
132 * Do the user attributes `states`, `types`, `period` match the notification conditions?
133 * Are there any notification `begin` and `end` times configured?
134 * Make sure the [notification](7-cli-commands.md#features) feature is enabled.
135 * Does the referenced NotificationCommand work when executed as Icinga user on the shell?
137 If notifications are to be sent via mail make sure that the mail program specified exists.
138 The name and location depends on the distribution so the preconfigured setting might have to be
139 changed on your system.
143 # icinga2 feature enable notification
144 The feature 'notification' is already enabled.
146 ## <a id="feature-not-working"></a> Feature is not working
148 * Make sure that the feature configuration is enabled by symlinking from `features-available/`
149 to `features-enabled` and that the latter is included in [icinga2.conf](4-configuring-icinga-2.md#icinga2-conf).
150 * Are the feature attributes set correctly according to the documentation?
151 * Any errors on the logs?
153 ## <a id="configuration-ignored"></a> Configuration is ignored
155 * Make sure that the line(s) are not [commented out](15-language-reference.md#comments) (starting with `//` or `#`, or
156 encapsulated by `/* ... */`).
157 * Is the configuration file included in [icinga2.conf](4-configuring-icinga-2.md#icinga2-conf)?
159 ## <a id="configuration-attribute-inheritance"></a> Configuration attributes are inherited from
161 Icinga 2 allows you to import templates using the [import](15-language-reference.md#template-imports) keyword. If these templates
162 contain additional attributes, your objects will automatically inherit them. You can override
163 or modify these attributes in the current object.
165 ## <a id="troubleshooting-cluster"></a> Cluster Troubleshooting
167 You should configure the [cluster health checks](8-monitoring-remote-systems.md#cluster-health-check) if you haven't
172 > Some problems just exist due to wrong file permissions or packet filters applied. Make
173 > sure to check these in the first place.
175 ### <a id="troubleshooting-cluster-connection-errors"></a> Cluster Troubleshooting Connection Errors
177 General connection errors normally lead you to one of the following problems:
179 * Wrong network configuration
180 * Packet loss on the connection
181 * Firewall rules preventing traffic
183 Use tools like `netstat`, `tcpdump`, `nmap`, etc to make sure that the cluster communication
184 happens (default port is `5665`).
186 # tcpdump -n port 5665 -i any
188 # netstat -tulpen | grep icinga
190 # nmap yourclusternode.localdomain
192 ### <a id="troubleshooting-cluster-ssl-errors"></a> Cluster Troubleshooting SSL Errors
194 If the cluster communication fails with cryptic SSL error messages, make sure to check
197 * File permissions on the SSL certificate files
198 * Does the used CA match for all cluster endpoints?
202 # ls -la /etc/icinga2/pki
205 ### <a id="troubleshooting-cluster-message-errors"></a> Cluster Troubleshooting Message Errors
207 At some point, when the network connection is broken or gone, the Icinga 2 instances
208 will be disconnected. If the connection can't be re-established between zones and endpoints,
209 they remain in a Split-Brain-mode and history may differ.
211 Although the Icinga 2 cluster protocol stores historical events in a replay log for later synchronisation,
212 you should make sure to check why the network connection failed.
214 ### <a id="troubleshooting-cluster-config-sync"></a> Cluster Troubleshooting Config Sync
216 If the cluster zones do not sync their configuration, make sure to check the following:
218 * Within a config master zone, only one configuration master is allowed to have its config in `/etc/icinga2/zones.d`.
219 ** The master syncs the configuration to `/var/lib/icinga2/api/zones/` during startup and only syncs valid configuration to the other nodes
220 ** The other nodes receive the configuration into `/var/lib/icinga2/api/zones/`
221 * The `icinga2.log` log file will indicate whether this ApiListener [accepts config](8-monitoring-remote-systems.md#zone-config-sync-permissions), or not
224 ## <a id="debug"></a> Debug Icinga 2
228 > If you are planning to build your own development environment,
229 > please consult the `INSTALL.md` file from the source tree.
231 ### <a id="debug-requirements"></a> Debug Requirements
233 Make sure that the debug symbols are available for Icinga 2.
234 The Icinga 2 packages provide a debug package which must be
235 installed separately for all involved binaries, like `icinga2-bin`
236 or `icinga2-ido-mysql`.
240 # apt-get install icinga2-dbg
244 # yum install icinga2-debuginfo
248 # zypper install icinga2-bin-debuginfo icinga2-ido-mysql-debuginfo
251 Furthermore, you may also have to install debug symbols for Boost and your C library.
253 If you're building your own binaries you should use the `-DCMAKE_BUILD_TYPE=Debug` cmake
254 build flag for debug builds.
257 ### <a id="development-debug-gdb"></a> GDB
263 # apt-get install gdb
274 Install the `boost`, `python` and `icinga2` pretty printers. Absolute paths are required,
275 so please make sure to update the installation paths accordingly (`pwd`).
277 Boost Pretty Printers:
279 $ mkdir -p ~/.gdb_printers && cd ~/.gdb_printers
280 $ git clone https://github.com/ruediger/Boost-Pretty-Printer.git && cd Boost-Pretty-Printer
282 /home/michi/.gdb_printers/Boost-Pretty-Printer
284 Python Pretty Printers:
287 $ svn co svn://gcc.gnu.org/svn/gcc/trunk/libstdc++-v3/python
289 Icinga 2 Pretty Printers:
291 $ mkdir -p ~/.gdb_printers/icinga2 && cd ~/.gdb_printers/icinga2
292 $ wget https://raw.githubusercontent.com/Icinga/icinga2/master/tools/debug/gdb/icingadbg.py
294 Now you'll need to modify/setup your `~/.gdbinit` configuration file.
295 You can download the one from Icinga 2 and modify all paths.
299 $ wget https://raw.githubusercontent.com/Icinga/icinga2/master/tools/debug/gdb/gdbinit -O ~/.gdbinit
306 sys.path.insert(0, '/home/michi/.gdb_printers/icinga2')
307 from icingadbg import register_icinga_printers
308 register_icinga_printers()
313 sys.path.insert(0, '/home/michi/.gdb_printers/python')
314 from libstdcxx.v6.printers import register_libstdcxx_printers
315 register_libstdcxx_printers(None)
320 sys.path.insert(0, '/home/michi/.gdb_printers/Boost-Pretty-Printer')
321 from boost.printers import register_printer_gen
322 register_printer_gen(None)
325 If you are getting the following error when running gdb, the `libstdcxx`
326 printers are already preloaded in your environment and you can remove
327 the duplicate import in your `~/.gdbinit` file.
329 RuntimeError: pretty-printer already registered: libstdc++-v6
331 ### <a id="development-debug-gdb-run"></a> GDB Run
333 Call GDB with the binary and all arguments and run it in foreground.
335 # gdb --args /usr/sbin/icinga2 daemon -x debug
339 > If gdb tells you it's missing debug symbols, quit gdb and install
340 > them: `Missing separate debuginfos, use: debuginfo-install ...`
346 Kill the running application.
350 Continue after breakpoint.
354 ### <a id="development-debug-gdb-backtrace"></a> GDB Backtrace
356 If Icinga 2 aborted its operation abnormally, generate a backtrace.
363 > If you're opening an issue at [https://dev.icinga.org] make sure
364 > to attach as much detail as possible.
367 ### <a id="development-debug-gdb-backtrace-stepping"></a> GDB Backtrace Stepping
369 Identifying the problem may require stepping into the backtrace, analysing
370 the current scope, attributes, and possible unmet requirements. `p` prints
371 the value of the selected variable or function call result.
376 (gdb) p checkable.px->m_Name
379 ### <a id="development-debug-gdb-breakpoint"></a> GDB Breakpoints
381 To set a breakpoint to a specific function call, or file specific line.
383 (gdb) b checkable.cpp:125
384 (gdb) b icinga::Checkable::SetEnablePerfdata
386 GDB will ask about loading the required symbols later, select `yes` instead
389 Then run Icinga 2 until it reaches the first breakpoint. Continue with `c`
395 If you want to delete all breakpoints, use `d` and select `yes`.
401 > When debugging exceptions, set your breakpoint like this: `b __cxa_throw`.
410 #11 0x00007ffff7cbf9ff in icinga::Utility::GlobRecursive(icinga::String const&, icinga::String const&, boost::function<void (icinga::String const&)> const&, int) (path=..., pattern=..., callback=..., type=1)
411 at /home/michi/coding/icinga/icinga2/lib/base/utility.cpp:609
415 605 #endif /* _WIN32 */
417 607 std::sort(files.begin(), files.end());
418 608 BOOST_FOREACH(const String& cpath, files) {
422 612 std::sort(dirs.begin(), dirs.end());
423 613 BOOST_FOREACH(const String& cpath, dirs) {
425 $3 = std::vector of length 11, capacity 16 = {{static NPos = 18446744073709551615, m_Data = "/etc/icinga2/conf.d/agent.conf"}, {static NPos = 18446744073709551615,
426 m_Data = "/etc/icinga2/conf.d/commands.conf"}, {static NPos = 18446744073709551615, m_Data = "/etc/icinga2/conf.d/downtimes.conf"}, {static NPos = 18446744073709551615,
427 m_Data = "/etc/icinga2/conf.d/groups.conf"}, {static NPos = 18446744073709551615, m_Data = "/etc/icinga2/conf.d/notifications.conf"}, {static NPos = 18446744073709551615,
428 m_Data = "/etc/icinga2/conf.d/satellite.conf"}, {static NPos = 18446744073709551615, m_Data = "/etc/icinga2/conf.d/services.conf"}, {static NPos = 18446744073709551615,
429 m_Data = "/etc/icinga2/conf.d/templates.conf"}, {static NPos = 18446744073709551615, m_Data = "/etc/icinga2/conf.d/test.conf"}, {static NPos = 18446744073709551615,
430 m_Data = "/etc/icinga2/conf.d/timeperiods.conf"}, {static NPos = 18446744073709551615, m_Data = "/etc/icinga2/conf.d/users.conf"}}