* [Icinga Web 2 modules](https://icinga.com/products/icinga-web-2-modules/) e.g. the Icinga Director (optional)
* Configuration insights:
* Provide complete configuration snippets explaining your problem in detail
- * Your [icinga2.conf](04-configuring-icinga-2.md#icinga2-conf) file
- * If you run multiple Icinga 2 instances, the [zones.conf](04-configuring-icinga-2.md#zones-conf) file (or `icinga2 object list --type Endpoint` and `icinga2 object list --type Zone`) from all affected nodes.
+ * Your [icinga2.conf](04-configuration.md#icinga2-conf) file
+ * If you run multiple Icinga 2 instances, the [zones.conf](04-configuration.md#zones-conf) file (or `icinga2 object list --type Endpoint` and `icinga2 object list --type Zone`) from all affected nodes.
* Logs
* Relevant output from your main and [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) in `/var/log/icinga2`. Please add step-by-step explanations with timestamps if required.
* The newest Icinga 2 crash log if relevant, located in `/var/log/icinga2/crash`
include <plugins>
```
-in the [icinga2.conf](04-configuring-icinga-2.md#icinga2-conf) configuration file. These files are not considered
+in the [icinga2.conf](04-configuration.md#icinga2-conf) configuration file. These files are not considered
configuration files and will be overridden on upgrade, so please send modifications as proposed patches upstream.
The default include path is set to `/usr/share/icinga2/includes` with the constant `IncludeConfDir`.
* Make sure that the line(s) are not [commented out](17-language-reference.md#comments) (starting with `//` or `#`, or
encapsulated by `/* ... */`).
-* Is the configuration file included in [icinga2.conf](04-configuring-icinga-2.md#icinga2-conf)?
+* Is the configuration file included in [icinga2.conf](04-configuration.md#icinga2-conf)?
Run the [configuration validation](11-cli-commands.md#config-validation) and add `notice` as log severity.
Search for the file which should be included i.e. using the `grep` CLI command.
critical/config: Error: Validation failed for Object 'ping4' (Type: 'Service') at /etc/icinga2/zones.d/global-templates/windows.conf:24: Closing $ not found in macro format string 'top-syntax=${list}'.
```
-Correct the custom attribute value to
+Correct the custom variable value to
```
"top-syntax=$${list}"
"results": [
{
"attrs": {
- "__name": "icinga2-client1.localdomain!disk",
+ "__name": "icinga2-agent1.localdomain!disk",
"last_check_result": {
"active": true,
- "check_source": "icinga2-client1.localdomain",
+ "check_source": "icinga2-agent1.localdomain",
...
},
"joins": {},
"meta": {},
- "name": "icinga2-client1.localdomain!disk",
+ "name": "icinga2-agent1.localdomain!disk",
"type": "Service"
}
]
```
$ ICINGA2_API_PASSWORD=icinga icinga2 console --connect 'https://root@localhost:5665/' \
---eval 'get_service("icinga2-client1.localdomain", "disk").last_check_result.check_source' | python -m json.tool
+--eval 'get_service("icinga2-agent1.localdomain", "disk").last_check_result.check_source' | python -m json.tool
-"icinga2-client1.localdomain"
+"icinga2-agent1.localdomain"
```
This could happen with [clients as command endpoint execution](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint).
-If you have for example a client host `icinga2-client1.localdomain`
+If you have for example a client host `icinga2-agent1.localdomain`
and a service `disk` check defined on the master, the warning and
critical thresholds are sometimes to applied and unwanted notification
alerts are raised.
This happens because the client itself includes a host object with
-its `NodeName` and a basic set of checks in the [conf.d](04-configuring-icinga-2.md#conf-d)
+its `NodeName` and a basic set of checks in the [conf.d](04-configuration.md#conf-d)
directory, i.e. `disk` with the default thresholds.
Clients which have the `checker` feature enabled will attempt
Solution:
* Disable the `checker` feature on clients: `icinga2 feature disable checker`.
-* Remove the inclusion of [conf.d](04-configuring-icinga-2.md#conf-d) as suggested in the [client setup docs](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint).
+* Remove the inclusion of [conf.d](04-configuration.md#conf-d) as suggested in the [client setup docs](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint).
### Check Fork Errors <a id="check-fork-errors"></a>
### Feature is not working <a id="feature-not-working"></a>
* Make sure that the feature configuration is enabled by symlinking from `features-available/`
-to `features-enabled` and that the latter is included in [icinga2.conf](04-configuring-icinga-2.md#icinga2-conf).
+to `features-enabled` and that the latter is included in [icinga2.conf](04-configuration.md#icinga2-conf).
* Are the feature attributes set correctly according to the documentation?
* Any errors on the logs?
}
```
-## REST API Troubleshooting: No Objects Found <a id="troubleshooting-api-no-objects-found"></a>
+##
# REST API Troubleshooting: No Objects Found <a id="troubleshooting-api-no-objects-found"></a>
Please note that the `404` status with no objects being found can also originate
from missing or too strict object permissions for the authenticated user.
- use an administrative account with full permissions to check whether the objects are actually there.
- verify the permissions on the affected ApiUser object and fix them.
+### Missing Runtime Objects (Hosts, Downtimes, etc.) <a id="troubleshooting-api-missing-runtime-objects"></a>
+
+Runtime objects consume the internal config packages shared with
+the REST API config packages. Each host, downtime, comment, service, etc. created
+via the REST API is stored in the `_api` package.
+
+This includes downtimes and comments, which where sometimes stored in the wrong
+directory path, because the active-stage file was empty/truncated/unreadable at
+this point.
+
+Wrong:
+
+```
+/var/lib/icinga2/api/packages/_api//conf.d/downtimes/1234-5678-9012-3456.conf
+```
+
+Correct:
+
+```
+/var/lib/icinga2/api/packages/_api/dbe0bef8-c72c-4cc9-9779-da7c4527c5b2/conf.d/downtimes/1234-5678-9012-3456.conf
+```
+
+At creation time, the object lives in memory but its storage is broken. Upon restart,
+it is missing and e.g. a missing downtime will re-enable unwanted notifications.
+
+`abcd-ef12-3456-7890` is the active stage name which wasn't correctly
+read by the Icinga daemon. This information is stored in `/var/lib/icinga2/api/packages/_api/active-stage`.
+
+2.11 now limits the direct active-stage file access (this is hidden from the user),
+and caches active stages for packages in-memory.
+
+It also tries to repair the broken package, and logs a new message:
+
+```
+systemctl restart icinga2
+
+tail -f /var/log/icinga2/icinga2.log
+
+[2019-05-10 12:27:15 +0200] information/ConfigObjectUtility: Repairing config package '_api' with stage 'dbe0bef8-c72c-4cc9-9779-da7c4527c5b2'.
+```
+
+If this does not happen, you can manually fix the broken config package, and mark a deployed stage as active
+again, carefully do the following steps with creating a backup before:
+
+Navigate into the API package prefix.
+
+```
+cd /var/lib/icinga2/api/packages
+```
+
+Change into the broken package directory and list all directories and files
+ordered by latest changes.
+
+```
+cd _api
+ls -lahtr
+
+drwx------ 4 michi wheel 128B Mar 27 14:39 ..
+-rw-r--r-- 1 michi wheel 25B Mar 27 14:39 include.conf
+-rw-r--r-- 1 michi wheel 405B Mar 27 14:39 active.conf
+drwx------ 7 michi wheel 224B Mar 27 15:01 dbe0bef8-c72c-4cc9-9779-da7c4527c5b2
+drwx------ 5 michi wheel 160B Apr 26 12:47 .
+```
+
+As you can see, the `active-stage` file is missing. When it is there, verify that its content
+is set to the stage directory as follows.
+
+If you have more than one stage directory here, pick the latest modified
+directory. Copy the directory name `abcd-ef12-3456-7890` and
+add it into a new file `active-stage`. This can be done like this:
+
+```
+echo "dbe0bef8-c72c-4cc9-9779-da7c4527c5b2" > active-stage
+```
+
+`active.conf` needs to have the correct active stage too, add it again
+like this. Note: This is deep down in the code, use with care!
+
+```
+sed -i 's/ActiveStages\["_api"\] = .*/ActiveStages\["_api"\] = "dbe0bef8-c72c-4cc9-9779-da7c4527c5b2"/g' /var/lib/icinga2/api/packages/_api/active.conf
+```
+
+Restart Icinga 2.
+
+```
+systemctl restart icinga2
+```
+
+
+> **Note**
+>
+> The internal `_api` config package structure may change in the future. Do not modify
+> things in there manually or with scripts unless guided here or asked by a developer.
+
## Certificate Troubleshooting <a id="troubleshooting-certificate"></a>
Client public certificate:
```
-# openssl x509 -in icinga2-client1.localdomain.crt -text
+# openssl x509 -in icinga2-agent1.localdomain.crt -text
Certificate:
Data:
Validity
Not Before: Aug 20 16:20:05 2016 GMT
Not After : Aug 17 16:20:05 2031 GMT
- Subject: CN=icinga2-client1.localdomain
+ Subject: CN=icinga2-agent1.localdomain
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (4096 bit)
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Alternative Name:
- DNS:icinga2-client1.localdomain
+ DNS:icinga2-agent1.localdomain
Signature Algorithm: sha256WithRSAEncryption
...
```
# openssl verify -verbose -CAfile /var/lib/icinga2/certs/ca.crt /var/lib/icinga2/certs/icinga2-master1.localdomain.crt
icinga2-master1.localdomain.crt: OK
-# openssl verify -verbose -CAfile /var/lib/icinga2/certs/ca.crt /var/lib/icinga2/certs/icinga2-client1.localdomain.crt
-icinga2-client1.localdomain.crt: OK
+# openssl verify -verbose -CAfile /var/lib/icinga2/certs/ca.crt /var/lib/icinga2/certs/icinga2-agent1.localdomain.crt
+icinga2-agent1.localdomain.crt: OK
```
Fetch the `ca.crt` file from the client node and compare it to your master's `ca.crt` file:
```
-# scp icinga2-client1:/var/lib/icinga2/certs/ca.crt test-client-ca.crt
+# scp icinga2-agent1:/var/lib/icinga2/certs/ca.crt test-client-ca.crt
# diff -ur /var/lib/icinga2/certs/ca.crt test-client-ca.crt
```
-On SLES11 you'll need to use the `openssl1` command instead of `openssl`.
-
<!--
### Certificate Signing <a id="troubleshooting-certificate-signing"></a>
-->
+### TLS Handshake: Ciphers <a id="troubleshooting-certificate-handshake-ciphers"></a>
+
+Starting with v2.11, the default configured ciphers have been hardened to modern
+standards. This includes TLS v1.2 as minimum protocol version too.
+
+In case the TLS handshake fails with `no shared cipher`, first analyse whether both
+instances support the same ciphers.
+
+Connect using `openssl s_client` and try to reproduce the connection problem.
+
+> **Important**
+>
+> The endpoint with the server role **accepting** the connection picks the preferred
+> cipher. E.g. when a satellite connects to the master, the master chooses the cipher.
+>
+> Keep this in mind where to simulate the client role connecting to a server with
+> CLI tools such as `openssl s_client`.
+
+
+`openssl s_client` tells you about the supported and shared cipher suites
+on the remove server. `openssl ciphers` lists locally available ciphers.
+
+```
+$ openssl s_client -connect 192.168.33.5:5665
+...
+
+---
+SSL handshake has read 2899 bytes and written 786 bytes
+---
+New, TLSv1/SSLv3, Cipher is AES256-GCM-SHA384
+Server public key is 4096 bit
+Secure Renegotiation IS supported
+Compression: NONE
+Expansion: NONE
+No ALPN negotiated
+SSL-Session:
+ Protocol : TLSv1.2
+ Cipher : AES256-GCM-SHA384
+
+...
+```
+
+You can specifically use one cipher or a list with the `-cipher` parameter:
+
+```
+openssl s_client -connect 192.168.33.5:5665 -cipher 'ECDHE-RSA-AES256-GCM-SHA384'
+```
+
+In order to fully simulate a connecting client, provide the certificates too:
+
+```
+CERTPATH='/var/lib/icinga2/certs'
+HOSTNAME='icinga2.vagrant.demo.icinga.com'
+openssl s_client -connect 192.168.33.5:5665 -cert "${CERTPATH}/${HOSTNAME}.crt" -key "${CERTPATH}/${HOSTNAME}.key" -CAfile "${CERTPATH}/ca.crt" -cipher 'ECDHE-RSA-AES256-GCM-SHA384'
+```
+
+In case to need to change the default cipher list,
+set the [cipher_list](09-object-types.md#objecttype-apilistener) attribute
+in the `api` feature configuration accordingly.
+
+Beware of using insecure ciphers, this may become a
+security risk in your organisation.
+
+
+#### Cipher Scan Tools
+
+You can also use different tools to test the available cipher suites, this is what SSL Labs, etc.
+provide for TLS enabled websites as well. [This post](https://superuser.com/questions/109213/how-do-i-list-the-ssl-tls-cipher-suites-a-particular-website-offers)
+highlights some tools and scripts such as [sslscan](https://github.com/rbsec/sslscan) or [testssl.sh](https://github.com/drwetter/testssl.sh/)
+
+Example for sslscan on macOS against a Debian 10 Buster instance
+running v2.11:
+
+```
+$ brew install sslscan
+
+$ sslscan 192.168.33.22:5665
+Version: 1.11.13-static
+OpenSSL 1.0.2f 28 Jan 2016
+
+Connected to 192.168.33.22
+
+Testing SSL server 192.168.33.22 on port 5665 using SNI name 192.168.33.22
+
+ TLS Fallback SCSV:
+Server supports TLS Fallback SCSV
+
+ TLS renegotiation:
+Session renegotiation not supported
+
+ TLS Compression:
+Compression disabled
+
+ Heartbleed:
+TLS 1.2 not vulnerable to heartbleed
+TLS 1.1 not vulnerable to heartbleed
+TLS 1.0 not vulnerable to heartbleed
+
+ Supported Server Cipher(s):
+Preferred TLSv1.2 256 bits ECDHE-RSA-AES256-GCM-SHA384 Curve P-256 DHE 256
+Accepted TLSv1.2 128 bits ECDHE-RSA-AES128-GCM-SHA256 Curve P-256 DHE 256
+Accepted TLSv1.2 256 bits ECDHE-RSA-AES256-SHA384 Curve P-256 DHE 256
+Accepted TLSv1.2 128 bits ECDHE-RSA-AES128-SHA256 Curve P-256 DHE 256
+
+ SSL Certificate:
+Signature Algorithm: sha256WithRSAEncryption
+RSA Key Strength: 4096
+
+Subject: icinga2-debian10.vagrant.demo.icinga.com
+Altnames: DNS:icinga2-debian10.vagrant.demo.icinga.com
+Issuer: Icinga CA
+
+Not valid before: Jul 12 07:39:55 2019 GMT
+Not valid after: Jul 8 07:39:55 2034 GMT
+```
+
### Certificate Problems with OpenSSL 1.1.0 <a id="troubleshooting-certificate-openssl-1-1-0"></a>
# netstat -tulpen | grep icinga
-# nmap icinga2-client1.localdomain
+# nmap icinga2-agent1.localdomain
```
### Cluster Troubleshooting SSL Errors <a id="troubleshooting-cluster-ssl-errors"></a>
* Verify the `Subject` containing your endpoint's common name (CN)
* Check the validity of the certificate itself
-Try to manually connect from `icinga2-client1.localdomain` to the master node `icinga2-master1.localdomain`:
+Try to manually connect from `icinga2-agent1.localdomain` to the master node `icinga2-master1.localdomain`:
```
-# openssl s_client -CAfile /var/lib/icinga2/certs/ca.crt -cert /var/lib/icinga2/certs/icinga2-client1.localdomain.crt -key /var/lib/icinga2/certs/icinga2-client1.localdomain.key -connect icinga2-master1.localdomain:5665
+# openssl s_client -CAfile /var/lib/icinga2/certs/ca.crt -cert /var/lib/icinga2/certs/icinga2-agent1.localdomain.crt -key /var/lib/icinga2/certs/icinga2-agent1.localdomain.key -connect icinga2-master1.localdomain:5665
CONNECTED(00000003)
---
Master:
```
-[2015-07-13 18:29:25 +0200] information/ApiListener: New client connection for identity 'icinga2-client1.localdomain' (unauthenticated)
+[2015-07-13 18:29:25 +0200] information/ApiListener: New client connection for identity 'icinga2-agent1.localdomain' (unauthenticated)
```
Client as command execution bridge:
> **Tip**
>
-> [Icinga Web 2](02-getting-started.md#setting-up-icingaweb2) provides a dashboard view
+> [Icinga Web 2](02-installation.md#setting-up-icingaweb2) provides a dashboard view
> for overdue check results.
Enable the [debug log](15-troubleshooting.md#troubleshooting-enable-debug-output) on the master
The client's endpoint is not configured on nor trusted by the master node:
```
-Discarding 'check result' message from 'icinga2-client1.localdomain': Invalid endpoint origin (client not allowed).
+Discarding 'check result' message from 'icinga2-agent1.localdomain': Invalid endpoint origin (client not allowed).
```
The check result message sent by the client does not belong to the zone the checkable object is
in on the master:
```
-Discarding 'check result' message from 'icinga2-client1.localdomain': Unauthorized access.
+Discarding 'check result' message from 'icinga2-agent1.localdomain': Unauthorized access.
```
### Cluster Troubleshooting: Windows Agents <a id="troubleshooting-cluster-windows-agents"></a>
+#### Windows Agents consuming 100% CPU <a id="troubleshooting-cluster-windows-agents-cpu"></a>
+
+Icinga 2 requires the `NodeName` [constant](17-language-reference.md#constants) in various places to run.
+This includes loading the TLS certificates, setting the proper check source,
+and so on.
+
+Typically the Windows setup wizard and also the CLI commands populate the [constants.conf](04-configuration.md#constants-conf)
+file with the auto-detected or user-provided FQDN/Common Name.
+
+If this constant is not set during startup, Icinga will try to resolve the
+FQDN, if that fails, fetch the hostname. If everything fails, it logs
+an error and sets this to `localhost`. This results in undefined behaviour
+if ignored by the admin.
+
+Querying the DNS when not reachable is CPU consuming, and may look like Icinga
+is doing lots of checks, etc. but actually really is just starting up.
+
+In order to fix this, edit the `constants.conf` file and populate
+the `NodeName` constant with the FQDN. Ensure this is the same value
+as the local endpoint object name.
+
+```
+const NodeName = "windows-agent1.domain.com"
+```
+
+
+
#### Windows blocking Icinga 2 with ephemeral port range <a id="troubleshooting-cluster-windows-agents-ephemeral-port-range"></a>
When you see a message like this in your Windows agent logs:
projects / icinga2 / blobdiff