1 # Icinga 2 Features <a id="icinga2-features"></a>
3 ## Logging <a id="logging"></a>
5 Icinga 2 supports three different types of logging:
8 * Syslog (on Linux/UNIX)
9 * Console logging (`STDOUT` on tty)
11 You can enable additional loggers using the `icinga2 feature enable`
12 and `icinga2 feature disable` commands to configure loggers:
15 ---------|------------
16 debuglog | Debug log (path: `/var/log/icinga2/debug.log`, severity: `debug` or higher)
17 mainlog | Main log (path: `/var/log/icinga2/icinga2.log`, severity: `information` or higher)
18 syslog | Syslog (severity: `warning` or higher)
20 By default file the `mainlog` feature is enabled. When running Icinga 2
21 on a terminal log messages with severity `information` or higher are
22 written to the console.
24 ### Log Rotation <a id="logging-logrotate"></a>
26 Packages provide a configuration file for [logrotate](https://linux.die.net/man/8/logrotate)
27 on Linux/Unix. Typically this is installed into `/etc/logrotate.d/icinga2`
28 and modifications won't be overridden on upgrade.
30 Instead of sending the reload HUP signal, logrotate
31 sends the USR1 signal to notify the Icinga daemon
32 that it has rotate the log file. Icinga reopens the log
35 * `/var/log/icinga2/icinga2.log` (requires `mainlog` enabled)
36 * `/var/log/icinga2/debug.log` (requires `debuglog` enabled)
37 * `/var/log/icinga2/erorr.log`
39 By default, log files will be rotated daily.
41 ## Core Backends <a id="core-backends"></a>
43 ### REST API <a id="core-backends-api"></a>
45 The REST API is documented [here](12-icinga2-api.md#icinga2-api) as a core feature.
47 ### IDO Database (DB IDO) <a id="db-ido"></a>
49 The IDO (Icinga Data Output) feature for Icinga 2 takes care of exporting all
50 configuration and status information into a database. The IDO database is used
51 by Icinga Web 2 as data backend.
53 Details on the installation can be found in the [Configuring DB IDO](02-getting-started.md#configuring-db-ido-mysql)
54 chapter. Details on the configuration can be found in the
55 [IdoMysqlConnection](09-object-types.md#objecttype-idomysqlconnection) and
56 [IdoPgsqlConnection](09-object-types.md#objecttype-idopgsqlconnection)
57 object configuration documentation.
59 #### DB IDO Health <a id="db-ido-health"></a>
61 If the monitoring health indicator is critical in Icinga Web 2,
62 you can use the following queries to manually check whether Icinga 2
63 is actually updating the IDO database.
65 Icinga 2 writes its current status to the `icinga_programstatus` table
66 every 10 seconds. The query below checks 60 seconds into the past which is a reasonable
67 amount of time -- adjust it for your requirements. If the condition is not met,
68 the query returns an empty result.
72 > Use [check plugins](05-service-monitoring.md#service-monitoring-plugins) to monitor the backend.
74 Replace the `default` string with your instance name if different.
79 # mysql -u root -p icinga -e "SELECT status_update_time FROM icinga_programstatus ps
80 JOIN icinga_instances i ON ps.instance_id=i.instance_id
81 WHERE (UNIX_TIMESTAMP(ps.status_update_time) > UNIX_TIMESTAMP(NOW())-60)
82 AND i.instance_name='default';"
84 +---------------------+
85 | status_update_time |
86 +---------------------+
87 | 2014-05-29 14:29:56 |
88 +---------------------+
91 Example for PostgreSQL:
94 # export PGPASSWORD=icinga; psql -U icinga -d icinga -c "SELECT ps.status_update_time FROM icinga_programstatus AS ps
95 JOIN icinga_instances AS i ON ps.instance_id=i.instance_id
96 WHERE ((SELECT extract(epoch from status_update_time) FROM icinga_programstatus) > (SELECT extract(epoch from now())-60))
97 AND i.instance_name='default'";
100 ------------------------
101 2014-05-29 15:11:38+02
105 A detailed list on the available table attributes can be found in the [DB IDO Schema documentation](24-appendix.md#schema-db-ido).
107 #### DB IDO in Cluster HA Zones <a id="db-ido-cluster-ha"></a>
109 The DB IDO feature supports [High Availability](06-distributed-monitoring.md#distributed-monitoring-high-availability-db-ido) in
110 the Icinga 2 cluster.
112 By default, both endpoints in a zone calculate the
113 endpoint which activates the feature, the other endpoint
114 automatically pauses it. If the cluster connection
115 breaks at some point, the paused IDO feature automatically
118 You can disable this behaviour by setting `enable_ha = false`
119 in both feature configuration files.
121 #### DB IDO Cleanup <a id="db-ido-cleanup"></a>
123 Objects get deactivated when they are deleted from the configuration.
124 This is visible with the `is_active` column in the `icinga_objects` table.
125 Therefore all queries need to join this table and add `WHERE is_active=1` as
126 condition. Deleted objects preserve their history table entries for later SLA
129 Historical data isn't purged by default. You can enable the least
130 kept data age inside the `cleanup` configuration attribute for the
131 IDO features [IdoMysqlConnection](09-object-types.md#objecttype-idomysqlconnection)
132 and [IdoPgsqlConnection](09-object-types.md#objecttype-idopgsqlconnection).
134 Example if you prefer to keep notification history for 30 days:
138 notifications_age = 30d
139 contactnotifications_age = 30d
143 The historical tables are populated depending on the data `categories` specified.
144 Some tables are empty by default.
146 #### DB IDO Tuning <a id="db-ido-tuning"></a>
148 As with any application database, there are ways to optimize and tune the database performance.
150 General tips for performance tuning:
152 * [MariaDB KB](https://mariadb.com/kb/en/library/optimization-and-tuning/)
153 * [PostgreSQL Wiki](https://wiki.postgresql.org/wiki/Performance_Optimization)
155 Re-creation of indexes, changed column values, etc. will increase the database size. Ensure to
156 add health checks for this, and monitor the trend in your Grafana dashboards.
158 In order to optimize the tables, there are different approaches. Always keep in mind to have a
159 current backup and schedule maintenance downtime for these kind of tasks!
164 mariadb> OPTIMIZE TABLE icinga_statehistory;
169 > Tables might not support optimization at runtime. This can take a **long** time.
171 > `Table does not support optimize, doing recreate + analyze instead`.
173 If you want to optimize all tables in a specified database, there is a script called `mysqlcheck`.
174 This also allows to repair broken tables in the case of emergency.
177 mysqlcheck --optimize icinga
189 > Don't use `VACUUM FULL` as this has a severe impact on performance.
192 ## Metrics <a id="metrics"></a>
194 Whenever a host or service check is executed, or received via the REST API,
195 best practice is to provide performance data.
197 This data is parsed by features sending metrics to time series databases (TSDB):
199 * [Graphite](14-features.md#graphite-carbon-cache-writer)
200 * [InfluxDB](14-features.md#influxdb-writer)
201 * [OpenTSDB](14-features.md#opentsdb-writer)
203 Metrics, state changes and notifications can be managed with the following integrations:
205 * [Elastic Stack](14-features.md#elastic-stack-integration)
206 * [Graylog](14-features.md#graylog-integration)
209 ### Graphite Writer <a id="graphite-carbon-cache-writer"></a>
211 [Graphite](13-addons.md#addons-graphing-graphite) is a tool stack for storing
212 metrics and needs to be running prior to enabling the `graphite` feature.
214 Icinga 2 writes parsed metrics directly to Graphite's Carbon Cache
215 TCP port, defaulting to `2003`.
217 You can enable the feature using
220 # icinga2 feature enable graphite
223 By default the [GraphiteWriter](09-object-types.md#objecttype-graphitewriter) feature
224 expects the Graphite Carbon Cache to listen at `127.0.0.1` on TCP port `2003`.
226 #### Graphite Schema <a id="graphite-carbon-cache-writer-schema"></a>
228 The current naming schema is defined as follows. The [Icinga Web 2 Graphite module](https://github.com/icinga/icingaweb2-module-graphite)
229 depends on this schema.
231 The default prefix for hosts and services is configured using
232 [runtime macros](03-monitoring-basics.md#runtime-macros)like this:
235 icinga2.$host.name$.host.$host.check_command$
236 icinga2.$host.name$.services.$service.name$.$service.check_command$
239 You can customize the prefix name by using the `host_name_template` and
240 `service_name_template` configuration attributes.
242 The additional levels will allow fine granular filters and also template
243 capabilities, e.g. by using the check command `disk` for specific
244 graph templates in web applications rendering the Graphite data.
246 The following characters are escaped in prefix labels:
248 Character | Escaped character
249 --------------|--------------------------
255 Metric values are stored like this:
258 <prefix>.perfdata.<perfdata-label>.value
261 The following characters are escaped in performance labels
262 parsed from plugin output:
264 Character | Escaped character
265 --------------|--------------------------
271 Note that labels may contain dots (`.`) allowing to
272 add more subsequent levels inside the Graphite tree.
273 `::` adds support for [multi performance labels](http://my-plugin.de/wiki/projects/check_multi/configuration/performance)
274 and is therefore replaced by `.`.
276 By enabling `enable_send_thresholds` Icinga 2 automatically adds the following threshold metrics:
279 <prefix>.perfdata.<perfdata-label>.min
280 <prefix>.perfdata.<perfdata-label>.max
281 <prefix>.perfdata.<perfdata-label>.warn
282 <prefix>.perfdata.<perfdata-label>.crit
285 By enabling `enable_send_metadata` Icinga 2 automatically adds the following metadata metrics:
288 <prefix>.metadata.current_attempt
289 <prefix>.metadata.downtime_depth
290 <prefix>.metadata.acknowledgement
291 <prefix>.metadata.execution_time
292 <prefix>.metadata.latency
293 <prefix>.metadata.max_check_attempts
294 <prefix>.metadata.reachable
295 <prefix>.metadata.state
296 <prefix>.metadata.state_type
299 Metadata metric overview:
302 -------------------|------------------------------------------
303 current_attempt | current check attempt
304 max_check_attempts | maximum check attempts until the hard state is reached
305 reachable | checked object is reachable
306 downtime_depth | number of downtimes this object is in
307 acknowledgement | whether the object is acknowledged or not
308 execution_time | check execution time
309 latency | check latency
310 state | current state of the checked object
311 state_type | 0=SOFT, 1=HARD state
313 The following example illustrates how to configure the storage schemas for Graphite Carbon
318 # intervals like PNP4Nagios uses them per default
320 retentions = 1m:2d,5m:10d,30m:90d,360m:4y
323 #### Graphite in Cluster HA Zones <a id="graphite-carbon-cache-writer-cluster-ha"></a>
325 The Graphite feature supports [high availability](06-distributed-monitoring.md#distributed-monitoring-high-availability-features)
326 in cluster zones since 2.11.
328 By default, all endpoints in a zone will activate the feature and start
329 writing metrics to a Carbon Cache socket. In HA enabled scenarios,
330 it is possible to set `enable_ha = true` in all feature configuration
331 files. This allows each endpoint to calculate the feature authority,
332 and only one endpoint actively writes metrics, the other endpoints
335 When the cluster connection breaks at some point, the remaining endpoint(s)
336 in that zone will automatically resume the feature. This built-in failover
337 mechanism ensures that metrics are written even if the cluster fails.
339 The recommended way of running Graphite in this scenario is a dedicated server
340 where Carbon Cache/Relay is running as receiver.
343 ### InfluxDB Writer <a id="influxdb-writer"></a>
345 Once there are new metrics available, Icinga 2 will directly write them to the
346 defined InfluxDB HTTP API.
348 You can enable the feature using
351 # icinga2 feature enable influxdb
354 By default the [InfluxdbWriter](09-object-types.md#objecttype-influxdbwriter) feature
355 expects the InfluxDB daemon to listen at `127.0.0.1` on port `8086`.
357 Measurement names and tags are fully configurable by the end user. The InfluxdbWriter
358 object will automatically add a `metric` tag to each data point. This correlates to the
359 perfdata label. Fields (value, warn, crit, min, max, unit) are created from data if available
360 and the configuration allows it. If a value associated with a tag is not able to be
361 resolved, it will be dropped and not sent to the target host.
363 Backslashes are allowed in tag keys, tag values and field keys, however they are also
364 escape characters when followed by a space or comma, but cannot be escaped themselves.
365 As a result all trailling slashes in these fields are replaced with an underscore. This
366 predominantly affects Windows paths e.g. `C:\` becomes `C:_`.
368 The database is assumed to exist so this object will make no attempt to create it currently.
370 If [SELinux](22-selinux.md#selinux) is enabled, it will not allow access for Icinga 2 to InfluxDB until the [boolean](22-selinux.md#selinux-policy-booleans)
371 `icinga2_can_connect_all` is set to true as InfluxDB is not providing its own policy.
373 More configuration details can be found [here](09-object-types.md#objecttype-influxdbwriter).
375 #### Instance Tagging <a id="influxdb-writer-instance-tags"></a>
377 Consider the following service check:
380 apply Service "disk" for (disk => attributes in host.vars.disks) {
381 import "generic-service"
382 check_command = "disk"
383 display_name = "Disk " + disk
384 vars.disk_partitions = disk
385 assign where host.vars.disks
389 This is a typical pattern for checking individual disks, NICs, SSL certificates etc associated
390 with a host. What would be useful is to have the data points tagged with the specific instance
391 for that check. This would allow you to query time series data for a check on a host and for a
392 specific instance e.g. /dev/sda. To do this quite simply add the instance to the service variables:
395 apply Service "disk" for (disk => attributes in host.vars.disks) {
402 Then modify your writer configuration to add this tag to your data points if the instance variable
403 is associated with the service:
406 object InfluxdbWriter "influxdb" {
409 measurement = "$service.check_command$"
411 hostname = "$host.name$"
412 service = "$service.name$"
413 instance = "$service.vars.instance$"
420 #### InfluxDB in Cluster HA Zones <a id="influxdb-writer-cluster-ha"></a>
422 The InfluxDB feature supports [high availability](06-distributed-monitoring.md#distributed-monitoring-high-availability-features)
423 in cluster zones since 2.11.
425 By default, all endpoints in a zone will activate the feature and start
426 writing metrics to the InfluxDB HTTP API. In HA enabled scenarios,
427 it is possible to set `enable_ha = true` in all feature configuration
428 files. This allows each endpoint to calculate the feature authority,
429 and only one endpoint actively writes metrics, the other endpoints
432 When the cluster connection breaks at some point, the remaining endpoint(s)
433 in that zone will automatically resume the feature. This built-in failover
434 mechanism ensures that metrics are written even if the cluster fails.
436 The recommended way of running InfluxDB in this scenario is a dedicated server
437 where the InfluxDB HTTP API or Telegraf as Proxy are running.
439 ### Elastic Stack Integration <a id="elastic-stack-integration"></a>
441 [Icingabeat](https://github.com/icinga/icingabeat) is an Elastic Beat that fetches data
442 from the Icinga 2 API and sends it either directly to [Elasticsearch](https://www.elastic.co/products/elasticsearch)
443 or [Logstash](https://www.elastic.co/products/logstash).
447 * [Logstash output](https://github.com/Icinga/logstash-output-icinga) for the Icinga 2 API.
448 * [Logstash Grok Pattern](https://github.com/Icinga/logstash-grok-pattern) for Icinga 2 logs.
450 #### Elasticsearch Writer <a id="elasticsearch-writer"></a>
452 This feature forwards check results, state changes and notification events
453 to an [Elasticsearch](https://www.elastic.co/products/elasticsearch) installation over its HTTP API.
455 The check results include parsed performance data metrics if enabled.
459 > Elasticsearch 5.x or 6.x are required. This feature has been successfully tested with
460 > Elasticsearch 5.6.7 and 6.3.1.
464 Enable the feature and restart Icinga 2.
467 # icinga2 feature enable elasticsearch
470 The default configuration expects an Elasticsearch instance running on `localhost` on port `9200
471 and writes to an index called `icinga2`.
473 More configuration details can be found [here](09-object-types.md#objecttype-elasticsearchwriter).
475 #### Current Elasticsearch Schema <a id="elastic-writer-schema"></a>
477 The following event types are written to Elasticsearch:
479 * icinga2.event.checkresult
480 * icinga2.event.statechange
481 * icinga2.event.notification
483 Performance data metrics must be explicitly enabled with the `enable_send_perfdata`
486 Metric values are stored like this:
489 check_result.perfdata.<perfdata-label>.value
492 The following characters are escaped in perfdata labels:
494 Character | Escaped character
495 ------------|--------------------------
501 Note that perfdata labels may contain dots (`.`) allowing to
502 add more subsequent levels inside the tree.
503 `::` adds support for [multi performance labels](http://my-plugin.de/wiki/projects/check_multi/configuration/performance)
504 and is therefore replaced by `.`.
506 Icinga 2 automatically adds the following threshold metrics
510 check_result.perfdata.<perfdata-label>.min
511 check_result.perfdata.<perfdata-label>.max
512 check_result.perfdata.<perfdata-label>.warn
513 check_result.perfdata.<perfdata-label>.crit
516 #### Elasticsearch in Cluster HA Zones <a id="elasticsearch-writer-cluster-ha"></a>
518 The Elasticsearch feature supports [high availability](06-distributed-monitoring.md#distributed-monitoring-high-availability-features)
519 in cluster zones since 2.11.
521 By default, all endpoints in a zone will activate the feature and start
522 writing events to the Elasticsearch HTTP API. In HA enabled scenarios,
523 it is possible to set `enable_ha = true` in all feature configuration
524 files. This allows each endpoint to calculate the feature authority,
525 and only one endpoint actively writes events, the other endpoints
528 When the cluster connection breaks at some point, the remaining endpoint(s)
529 in that zone will automatically resume the feature. This built-in failover
530 mechanism ensures that events are written even if the cluster fails.
532 The recommended way of running Elasticsearch in this scenario is a dedicated server
533 where you either have the Elasticsearch HTTP API, or a TLS secured HTTP proxy,
534 or Logstash for additional filtering.
536 ### Graylog Integration <a id="graylog-integration"></a>
538 #### GELF Writer <a id="gelfwriter"></a>
540 The `Graylog Extended Log Format` (short: [GELF](http://docs.graylog.org/en/latest/pages/gelf.html))
541 can be used to send application logs directly to a TCP socket.
543 While it has been specified by the [Graylog](https://www.graylog.org) project as their
544 [input resource standard](http://docs.graylog.org/en/latest/pages/sending_data.html), other tools such as
545 [Logstash](https://www.elastic.co/products/logstash) also support `GELF` as
546 [input type](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-gelf.html).
548 You can enable the feature using
551 # icinga2 feature enable gelf
554 By default the `GelfWriter` object expects the GELF receiver to listen at `127.0.0.1` on TCP port `12201`.
555 The default `source` attribute is set to `icinga2`. You can customize that for your needs if required.
557 Currently these events are processed:
562 #### Graylog/GELF in Cluster HA Zones <a id="gelf-writer-cluster-ha"></a>
564 The Gelf feature supports [high availability](06-distributed-monitoring.md#distributed-monitoring-high-availability-features)
565 in cluster zones since 2.11.
567 By default, all endpoints in a zone will activate the feature and start
568 writing events to the Graylog HTTP API. In HA enabled scenarios,
569 it is possible to set `enable_ha = true` in all feature configuration
570 files. This allows each endpoint to calculate the feature authority,
571 and only one endpoint actively writes events, the other endpoints
574 When the cluster connection breaks at some point, the remaining endpoint(s)
575 in that zone will automatically resume the feature. This built-in failover
576 mechanism ensures that events are written even if the cluster fails.
578 The recommended way of running Graylog in this scenario is a dedicated server
579 where you have the Graylog HTTP API listening.
581 ### OpenTSDB Writer <a id="opentsdb-writer"></a>
583 While there are some OpenTSDB collector scripts and daemons like tcollector available for
584 Icinga 1.x it's more reasonable to directly process the check and plugin performance
585 in memory in Icinga 2. Once there are new metrics available, Icinga 2 will directly
586 write them to the defined TSDB TCP socket.
588 You can enable the feature using
591 # icinga2 feature enable opentsdb
594 By default the `OpenTsdbWriter` object expects the TSD to listen at
595 `127.0.0.1` on port `4242`.
597 The current naming schema is
600 icinga.host.<metricname>
601 icinga.service.<servicename>.<metricname>
604 for host and service checks. The tag host is always applied.
606 To make sure Icinga 2 writes a valid metric into OpenTSDB some characters are replaced
607 with `_` in the target name:
613 The resulting name in OpenTSDB might look like:
616 www-01 / http-cert / response time
617 icinga.http_cert.response_time
620 In addition to the performance data retrieved from the check plugin, Icinga 2 sends
621 internal check statistic data to OpenTSDB:
624 -------------------|------------------------------------------
625 current_attempt | current check attempt
626 max_check_attempts | maximum check attempts until the hard state is reached
627 reachable | checked object is reachable
628 downtime_depth | number of downtimes this object is in
629 acknowledgement | whether the object is acknowledged or not
630 execution_time | check execution time
631 latency | check latency
632 state | current state of the checked object
633 state_type | 0=SOFT, 1=HARD state
635 While reachable, state and state_type are metrics for the host or service the
636 other metrics follow the current naming schema
639 icinga.check.<metricname>
642 with the following tags
645 --------|------------------------------------------
646 type | the check type, one of [host, service]
647 host | hostname, the check ran on
648 service | the service name (if type=service)
652 > You might want to set the tsd.core.auto_create_metrics setting to `true`
653 > in your opentsdb.conf configuration file.
655 #### OpenTSDB in Cluster HA Zones <a id="opentsdb-writer-cluster-ha"></a>
657 The OpenTSDB feature supports [high availability](06-distributed-monitoring.md#distributed-monitoring-high-availability-features)
658 in cluster zones since 2.11.
660 By default, all endpoints in a zone will activate the feature and start
661 writing events to the OpenTSDB listener. In HA enabled scenarios,
662 it is possible to set `enable_ha = true` in all feature configuration
663 files. This allows each endpoint to calculate the feature authority,
664 and only one endpoint actively writes metrics, the other endpoints
667 When the cluster connection breaks at some point, the remaining endpoint(s)
668 in that zone will automatically resume the feature. This built-in failover
669 mechanism ensures that metrics are written even if the cluster fails.
671 The recommended way of running OpenTSDB in this scenario is a dedicated server
672 where you have OpenTSDB running.
675 ### Writing Performance Data Files <a id="writing-performance-data-files"></a>
677 PNP and Graphios use performance data collector daemons to fetch
678 the current performance files for their backend updates.
680 Therefore the Icinga 2 [PerfdataWriter](09-object-types.md#objecttype-perfdatawriter)
681 feature allows you to define the output template format for host and services helped
682 with Icinga 2 runtime vars.
685 host_format_template = "DATATYPE::HOSTPERFDATA\tTIMET::$icinga.timet$\tHOSTNAME::$host.name$\tHOSTPERFDATA::$host.perfdata$\tHOSTCHECKCOMMAND::$host.check_command$\tHOSTSTATE::$host.state$\tHOSTSTATETYPE::$host.state_type$"
686 service_format_template = "DATATYPE::SERVICEPERFDATA\tTIMET::$icinga.timet$\tHOSTNAME::$host.name$\tSERVICEDESC::$service.name$\tSERVICEPERFDATA::$service.perfdata$\tSERVICECHECKCOMMAND::$service.check_command$\tHOSTSTATE::$host.state$\tHOSTSTATETYPE::$host.state_type$\tSERVICESTATE::$service.state$\tSERVICESTATETYPE::$service.state_type$"
689 The default templates are already provided with the Icinga 2 feature configuration
690 which can be enabled using
693 # icinga2 feature enable perfdata
696 By default all performance data files are rotated in a 15 seconds interval into
697 the `/var/spool/icinga2/perfdata/` directory as `host-perfdata.<timestamp>` and
698 `service-perfdata.<timestamp>`.
699 External collectors need to parse the rotated performance data files and then
700 remove the processed files.
702 #### Perfdata Files in Cluster HA Zones <a id="perfdata-writer-cluster-ha"></a>
704 The Perfdata feature supports [high availability](06-distributed-monitoring.md#distributed-monitoring-high-availability-features)
705 in cluster zones since 2.11.
707 By default, all endpoints in a zone will activate the feature and start
708 writing metrics to the local spool directory. In HA enabled scenarios,
709 it is possible to set `enable_ha = true` in all feature configuration
710 files. This allows each endpoint to calculate the feature authority,
711 and only one endpoint actively writes metrics, the other endpoints
714 When the cluster connection breaks at some point, the remaining endpoint(s)
715 in that zone will automatically resume the feature. This built-in failover
716 mechanism ensures that metrics are written even if the cluster fails.
718 The recommended way of running Perfdata is to mount the perfdata spool
719 directory via NFS on a central server where PNP with the NPCD collector
725 ## Livestatus <a id="setting-up-livestatus"></a>
727 The [MK Livestatus](https://mathias-kettner.de/checkmk_livestatus.html) project
728 implements a query protocol that lets users query their Icinga instance for
729 status information. It can also be used to send commands.
731 The Livestatus component that is distributed as part of Icinga 2 is a
732 re-implementation of the Livestatus protocol which is compatible with MK
737 > Only install the Livestatus feature if your web interface or addon requires
739 > [Icinga Web 2](02-getting-started.md#setting-up-icingaweb2) does not need
742 Details on the available tables and attributes with Icinga 2 can be found
743 in the [Livestatus Schema](24-appendix.md#schema-livestatus) section.
745 You can enable Livestatus using icinga2 feature enable:
748 # icinga2 feature enable livestatus
751 After that you will have to restart Icinga 2:
754 # systemctl restart icinga2
757 By default the Livestatus socket is available in `/var/run/icinga2/cmd/livestatus`.
759 In order for queries and commands to work you will need to add your query user
760 (e.g. your web server) to the `icingacmd` group:
763 # usermod -a -G icingacmd www-data
766 The Debian packages use `nagios` as the user and group name. Make sure to change `icingacmd` to
767 `nagios` if you're using Debian.
769 Change `www-data` to the user you're using to run queries.
771 In order to use the historical tables provided by the livestatus feature (for example, the
772 `log` table) you need to have the `CompatLogger` feature enabled. By default these logs
773 are expected to be in `/var/log/icinga2/compat`. A different path can be set using the
774 `compat_log_path` configuration attribute.
777 # icinga2 feature enable compatlog
780 ### Livestatus Sockets <a id="livestatus-sockets"></a>
782 Other to the Icinga 1.x Addon, Icinga 2 supports two socket types
784 * Unix socket (default)
787 Details on the configuration can be found in the [LivestatusListener](09-object-types.md#objecttype-livestatuslistener)
788 object configuration.
790 ### Livestatus GET Queries <a id="livestatus-get-queries"></a>
794 > All Livestatus queries require an additional empty line as query end identifier.
795 > The `nc` tool (`netcat`) provides the `-U` parameter to communicate using
798 There also is a Perl module available in CPAN for accessing the Livestatus socket
799 programmatically: [Monitoring::Livestatus](http://search.cpan.org/~nierlein/Monitoring-Livestatus-0.74/)
802 Example using the unix socket:
805 # echo -e "GET services\n" | /usr/bin/nc -U /var/run/icinga2/cmd/livestatus
807 Example using the tcp socket listening on port `6558`:
809 # echo -e 'GET services\n' | netcat 127.0.0.1 6558
811 # cat servicegroups <<EOF
816 (cat servicegroups; sleep 1) | netcat 127.0.0.1 6558
819 ### Livestatus COMMAND Queries <a id="livestatus-command-queries"></a>
821 A list of available external commands and their parameters can be found [here](24-appendix.md#external-commands-list-detail)
824 $ echo -e 'COMMAND <externalcommandstring>' | netcat 127.0.0.1 6558
827 ### Livestatus Filters <a id="livestatus-filters"></a>
831 Operator | Negate | Description
832 ----------|----------|-------------
835 =~ | !=~ | Equality ignoring case
836 ~~ | !~~ | Regex ignoring case
839 <= | | Less than or equal
840 >= | | Greater than or equal
843 ### Livestatus Stats <a id="livestatus-stats"></a>
845 Schema: "Stats: aggregatefunction aggregateattribute"
847 Aggregate Function | Description
848 -------------------|--------------
853 std | standard deviation
854 suminv | sum (1 / value)
855 avginv | suminv / count
856 count | ordinary default for any stats query if not aggregate function defined
862 Filter: has_been_checked = 1
863 Filter: check_type = 0
864 Stats: sum execution_time
866 Stats: sum percent_state_change
867 Stats: min execution_time
869 Stats: min percent_state_change
870 Stats: max execution_time
872 Stats: max percent_state_change
874 ResponseHeader: fixed16
877 ### Livestatus Output <a id="livestatus-output"></a>
881 CSV output uses two levels of array separators: The members array separator
882 is a comma (1st level) while extra info and host|service relation separator
883 is a pipe (2nd level).
885 Separators can be set using ASCII codes like:
888 Separators: 10 59 44 124
895 ### Livestatus Error Codes <a id="livestatus-error-codes"></a>
898 ----------|--------------
900 404 | Table does not exist
901 452 | Exception on query
903 ### Livestatus Tables <a id="livestatus-tables"></a>
905 Table | Join |Description
906 --------------|-----------|----------------------------
907 hosts | | host config and status attributes, services counter
908 hostgroups | | hostgroup config, status attributes and host/service counters
909 services | hosts | service config and status attributes
910 servicegroups | | servicegroup config, status attributes and service counters
911 contacts | | contact config and status attributes
912 contactgroups | | contact config, members
913 commands | | command name and line
914 status | | programstatus, config and stats
915 comments | services | status attributes
916 downtimes | services | status attributes
917 timeperiods | | name and is inside flag
918 endpoints | | config and status attributes
919 log | services, hosts, contacts, commands | parses [compatlog](09-object-types.md#objecttype-compatlogger) and shows log attributes
920 statehist | hosts, services | parses [compatlog](09-object-types.md#objecttype-compatlogger) and aggregates state change attributes
921 hostsbygroup | hostgroups | host attributes grouped by hostgroup and its attributes
922 servicesbygroup | servicegroups | service attributes grouped by servicegroup and its attributes
923 servicesbyhostgroup | hostgroups | service attributes grouped by hostgroup and its attributes
925 The `commands` table is populated with `CheckCommand`, `EventCommand` and `NotificationCommand` objects.
927 A detailed list on the available table attributes can be found in the [Livestatus Schema documentation](24-appendix.md#schema-livestatus).
930 ## Deprecated Features <a id="deprecated-features"></a>
932 ### Status Data Files <a id="status-data"></a>
936 > This feature is DEPRECATED and will be removed in future releases.
937 > Check the [roadmap](https://github.com/Icinga/icinga2/milestones).
939 Icinga 1.x writes object configuration data and status data in a cyclic
940 interval to its `objects.cache` and `status.dat` files. Icinga 2 provides
941 the `StatusDataWriter` object which dumps all configuration objects and
942 status updates in a regular interval.
945 # icinga2 feature enable statusdata
948 If you are not using any web interface or addon which uses these files,
949 you can safely disable this feature.
951 ### Compat Log Files <a id="compat-logging"></a>
955 > This feature is DEPRECATED and will be removed in future releases.
956 > Check the [roadmap](https://github.com/Icinga/icinga2/milestones).
958 The Icinga 1.x log format is considered being the `Compat Log`
959 in Icinga 2 provided with the `CompatLogger` object.
961 These logs are used for informational representation in
962 external web interfaces parsing the logs, but also to generate
963 SLA reports and trends.
964 The [Livestatus](14-features.md#setting-up-livestatus) feature uses these logs
965 for answering queries to historical tables.
967 The `CompatLogger` object can be enabled with
970 # icinga2 feature enable compatlog
973 By default, the Icinga 1.x log file called `icinga.log` is located
974 in `/var/log/icinga2/compat`. Rotated log files are moved into
975 `var/log/icinga2/compat/archives`.
977 ### External Command Pipe <a id="external-commands"></a>
981 > Please use the [REST API](12-icinga2-api.md#icinga2-api) as modern and secure alternative
982 > for external actions.
986 > This feature is DEPRECATED and will be removed in future releases.
987 > Check the [roadmap](https://github.com/Icinga/icinga2/milestones).
989 Icinga 2 provides an external command pipe for processing commands
990 triggering specific actions (for example rescheduling a service check
991 through the web interface).
993 In order to enable the `ExternalCommandListener` configuration use the
994 following command and restart Icinga 2 afterwards:
997 # icinga2 feature enable command
1000 Icinga 2 creates the command pipe file as `/var/run/icinga2/cmd/icinga2.cmd`
1001 using the default configuration.
1003 Web interfaces and other Icinga addons are able to send commands to
1004 Icinga 2 through the external command pipe, for example for rescheduling
1005 a forced service check:
1008 # /bin/echo "[`date +%s`] SCHEDULE_FORCED_SVC_CHECK;localhost;ping4;`date +%s`" >> /var/run/icinga2/cmd/icinga2.cmd
1010 # tail -f /var/log/messages
1012 Oct 17 15:01:25 icinga-server icinga2: Executing external command: [1382014885] SCHEDULE_FORCED_SVC_CHECK;localhost;ping4;1382014885
1013 Oct 17 15:01:25 icinga-server icinga2: Rescheduling next check for service 'ping4'
1016 A list of currently supported external commands can be found [here](24-appendix.md#external-commands-list-detail).
1018 Detailed information on the commands and their required parameters can be found
1019 on the [Icinga 1.x documentation](https://docs.icinga.com/latest/en/extcommands2.html).
1022 ### Check Result Files <a id="check-result-files"></a>
1026 > This feature is DEPRECATED and will be removed in future releases.
1027 > Check the [roadmap](https://github.com/Icinga/icinga2/milestones).
1029 Icinga 1.x writes its check result files to a temporary spool directory
1030 where they are processed in a regular interval.
1031 While this is extremely inefficient in performance regards it has been
1032 rendered useful for passing passive check results directly into Icinga 1.x
1033 skipping the external command pipe.
1035 Several clustered/distributed environments and check-aggregation addons
1036 use that method. In order to support step-by-step migration of these
1037 environments, Icinga 2 supports the `CheckResultReader` object.
1039 There is no feature configuration available, but it must be defined
1040 on-demand in your Icinga 2 objects configuration.
1043 object CheckResultReader "reader" {
1044 spool_dir = "/data/check-results"