1 # Icinga 2 Features <a id="icinga2-features"></a>
3 ## Logging <a id="logging"></a>
5 Icinga 2 supports three different types of logging:
8 * Syslog (on Linux/UNIX)
9 * Console logging (`STDOUT` on tty)
11 You can enable additional loggers using the `icinga2 feature enable`
12 and `icinga2 feature disable` commands to configure loggers:
15 ---------|------------
16 debuglog | Debug log (path: `/var/log/icinga2/debug.log`, severity: `debug` or higher)
17 mainlog | Main log (path: `/var/log/icinga2/icinga2.log`, severity: `information` or higher)
18 syslog | Syslog (severity: `warning` or higher)
20 By default file the `mainlog` feature is enabled. When running Icinga 2
21 on a terminal log messages with severity `information` or higher are
22 written to the console.
24 Packages will install a configuration file for logrotate on supported
25 platforms. This configuration ensures that the `icinga2.log`, `error.log` and
26 `debug.log` files are rotated on a daily basis.
28 ## DB IDO <a id="db-ido"></a>
30 The IDO (Icinga Data Output) modules for Icinga 2 take care of exporting all
31 configuration and status information into a database. The IDO database is used
34 Details on the installation can be found in the [Configuring DB IDO](02-getting-started.md#configuring-db-ido-mysql)
35 chapter. Details on the configuration can be found in the
36 [IdoMysqlConnection](09-object-types.md#objecttype-idomysqlconnection) and
37 [IdoPgsqlConnection](09-object-types.md#objecttype-idopgsqlconnection)
38 object configuration documentation.
39 The DB IDO feature supports [High Availability](06-distributed-monitoring.md#distributed-monitoring-high-availability-db-ido) in
42 The following example query checks the health of the current Icinga 2 instance
43 writing its current status to the DB IDO backend table `icinga_programstatus`
44 every 10 seconds. By default it checks 60 seconds into the past which is a reasonable
45 amount of time -- adjust it for your requirements. If the condition is not met,
46 the query returns an empty result.
50 > Use [check plugins](05-service-monitoring.md#service-monitoring-plugins) to monitor the backend.
52 Replace the `default` string with your instance name if different.
56 # mysql -u root -p icinga -e "SELECT status_update_time FROM icinga_programstatus ps
57 JOIN icinga_instances i ON ps.instance_id=i.instance_id
58 WHERE (UNIX_TIMESTAMP(ps.status_update_time) > UNIX_TIMESTAMP(NOW())-60)
59 AND i.instance_name='default';"
61 +---------------------+
62 | status_update_time |
63 +---------------------+
64 | 2014-05-29 14:29:56 |
65 +---------------------+
68 Example for PostgreSQL:
70 # export PGPASSWORD=icinga; psql -U icinga -d icinga -c "SELECT ps.status_update_time FROM icinga_programstatus AS ps
71 JOIN icinga_instances AS i ON ps.instance_id=i.instance_id
72 WHERE ((SELECT extract(epoch from status_update_time) FROM icinga_programstatus) > (SELECT extract(epoch from now())-60))
73 AND i.instance_name='default'";
76 ------------------------
77 2014-05-29 15:11:38+02
81 A detailed list on the available table attributes can be found in the [DB IDO Schema documentation](24-appendix.md#schema-db-ido).
84 ## External Commands <a id="external-commands"></a>
88 > Please use the [REST API](12-icinga2-api.md#icinga2-api) as modern and secure alternative
89 > for external actions.
91 Icinga 2 provides an external command pipe for processing commands
92 triggering specific actions (for example rescheduling a service check
93 through the web interface).
95 In order to enable the `ExternalCommandListener` configuration use the
96 following command and restart Icinga 2 afterwards:
98 # icinga2 feature enable command
100 Icinga 2 creates the command pipe file as `/var/run/icinga2/cmd/icinga2.cmd`
101 using the default configuration.
103 Web interfaces and other Icinga addons are able to send commands to
104 Icinga 2 through the external command pipe, for example for rescheduling
105 a forced service check:
107 # /bin/echo "[`date +%s`] SCHEDULE_FORCED_SVC_CHECK;localhost;ping4;`date +%s`" >> /var/run/icinga2/cmd/icinga2.cmd
109 # tail -f /var/log/messages
111 Oct 17 15:01:25 icinga-server icinga2: Executing external command: [1382014885] SCHEDULE_FORCED_SVC_CHECK;localhost;ping4;1382014885
112 Oct 17 15:01:25 icinga-server icinga2: Rescheduling next check for service 'ping4'
114 A list of currently supported external commands can be found [here](24-appendix.md#external-commands-list-detail).
116 Detailed information on the commands and their required parameters can be found
117 on the [Icinga 1.x documentation](https://docs.icinga.com/latest/en/extcommands2.html).
119 ## Performance Data <a id="performance-data"></a>
121 When a host or service check is executed plugins should provide so-called
122 `performance data`. Next to that additional check performance data
123 can be fetched using Icinga 2 runtime macros such as the check latency
124 or the current service state (or additional custom attributes).
126 The performance data can be passed to external applications which aggregate and
127 store them in their backends. These tools usually generate graphs for historical
128 reporting and trending.
130 Well-known addons processing Icinga performance data are [PNP4Nagios](13-addons.md#addons-graphing-pnp),
131 [Graphite](13-addons.md#addons-graphing-graphite) or [OpenTSDB](14-features.md#opentsdb-writer).
133 ### Writing Performance Data Files <a id="writing-performance-data-files"></a>
135 PNP4Nagios and Graphios use performance data collector daemons to fetch
136 the current performance files for their backend updates.
138 Therefore the Icinga 2 [PerfdataWriter](09-object-types.md#objecttype-perfdatawriter)
139 feature allows you to define the output template format for host and services helped
140 with Icinga 2 runtime vars.
142 host_format_template = "DATATYPE::HOSTPERFDATA\tTIMET::$icinga.timet$\tHOSTNAME::$host.name$\tHOSTPERFDATA::$host.perfdata$\tHOSTCHECKCOMMAND::$host.check_command$\tHOSTSTATE::$host.state$\tHOSTSTATETYPE::$host.state_type$"
143 service_format_template = "DATATYPE::SERVICEPERFDATA\tTIMET::$icinga.timet$\tHOSTNAME::$host.name$\tSERVICEDESC::$service.name$\tSERVICEPERFDATA::$service.perfdata$\tSERVICECHECKCOMMAND::$service.check_command$\tHOSTSTATE::$host.state$\tHOSTSTATETYPE::$host.state_type$\tSERVICESTATE::$service.state$\tSERVICESTATETYPE::$service.state_type$"
145 The default templates are already provided with the Icinga 2 feature configuration
146 which can be enabled using
148 # icinga2 feature enable perfdata
150 By default all performance data files are rotated in a 15 seconds interval into
151 the `/var/spool/icinga2/perfdata/` directory as `host-perfdata.<timestamp>` and
152 `service-perfdata.<timestamp>`.
153 External collectors need to parse the rotated performance data files and then
154 remove the processed files.
156 ### Graphite Carbon Cache Writer <a id="graphite-carbon-cache-writer"></a>
158 While there are some [Graphite](13-addons.md#addons-graphing-graphite)
159 collector scripts and daemons like Graphios available for Icinga 1.x it's more
160 reasonable to directly process the check and plugin performance
161 in memory in Icinga 2. Once there are new metrics available, Icinga 2 will directly
162 write them to the defined Graphite Carbon daemon tcp socket.
164 You can enable the feature using
166 # icinga2 feature enable graphite
168 By default the [GraphiteWriter](09-object-types.md#objecttype-graphitewriter) feature
169 expects the Graphite Carbon Cache to listen at `127.0.0.1` on TCP port `2003`.
171 #### Current Graphite Schema <a id="graphite-carbon-cache-writer-schema"></a>
173 The current naming schema is defined as follows. The [Icinga Web 2 Graphite module](https://github.com/icinga/icingaweb2-module-graphite)
174 depends on this schema.
176 The default prefix for hosts and services is configured using
177 [runtime macros](03-monitoring-basics.md#runtime-macros)like this:
179 icinga2.$host.name$.host.$host.check_command$
180 icinga2.$host.name$.services.$service.name$.$service.check_command$
182 You can customize the prefix name by using the `host_name_template` and
183 `service_name_template` configuration attributes.
185 The additional levels will allow fine granular filters and also template
186 capabilities, e.g. by using the check command `disk` for specific
187 graph templates in web applications rendering the Graphite data.
189 The following characters are escaped in prefix labels:
191 Character | Escaped character
192 --------------|--------------------------
198 Metric values are stored like this:
200 <prefix>.perfdata.<perfdata-label>.value
202 The following characters are escaped in perfdata labels:
204 Character | Escaped character
205 --------------|--------------------------
211 Note that perfdata labels may contain dots (`.`) allowing to
212 add more subsequent levels inside the Graphite tree.
213 `::` adds support for [multi performance labels](http://my-plugin.de/wiki/projects/check_multi/configuration/performance)
214 and is therefore replaced by `.`.
216 By enabling `enable_send_thresholds` Icinga 2 automatically adds the following threshold metrics:
218 <prefix>.perfdata.<perfdata-label>.min
219 <prefix>.perfdata.<perfdata-label>.max
220 <prefix>.perfdata.<perfdata-label>.warn
221 <prefix>.perfdata.<perfdata-label>.crit
223 By enabling `enable_send_metadata` Icinga 2 automatically adds the following metadata metrics:
225 <prefix>.metadata.current_attempt
226 <prefix>.metadata.downtime_depth
227 <prefix>.metadata.acknowledgement
228 <prefix>.metadata.execution_time
229 <prefix>.metadata.latency
230 <prefix>.metadata.max_check_attempts
231 <prefix>.metadata.reachable
232 <prefix>.metadata.state
233 <prefix>.metadata.state_type
235 Metadata metric overview:
238 -------------------|------------------------------------------
239 current_attempt | current check attempt
240 max_check_attempts | maximum check attempts until the hard state is reached
241 reachable | checked object is reachable
242 downtime_depth | number of downtimes this object is in
243 acknowledgement | whether the object is acknowledged or not
244 execution_time | check execution time
245 latency | check latency
246 state | current state of the checked object
247 state_type | 0=SOFT, 1=HARD state
249 The following example illustrates how to configure the storage schemas for Graphite Carbon
253 # intervals like PNP4Nagios uses them per default
255 retentions = 1m:2d,5m:10d,30m:90d,360m:4y
258 ### InfluxDB Writer <a id="influxdb-writer"></a>
260 Once there are new metrics available, Icinga 2 will directly write them to the
261 defined InfluxDB HTTP API.
263 You can enable the feature using
265 # icinga2 feature enable influxdb
267 By default the [InfluxdbWriter](09-object-types.md#objecttype-influxdbwriter) feature
268 expects the InfluxDB daemon to listen at `127.0.0.1` on port `8086`.
270 Measurement names and tags are fully configurable by the end user. The InfluxdbWriter
271 object will automatically add a `metric` tag to each data point. This correlates to the
272 perfdata label. Fields (value, warn, crit, min, max, unit) are created from data if available
273 and the configuration allows it. If a value associated with a tag is not able to be
274 resolved, it will be dropped and not sent to the target host.
276 Backslashes are allowed in tag keys, tag values and field keys, however they are also
277 escape characters when followed by a space or comma, but cannot be escaped themselves.
278 As a result all trailling slashes in these fields are replaced with an underscore. This
279 predominantly affects Windows paths e.g. `C:\` becomes `C:_`.
281 The database is assumed to exist so this object will make no attempt to create it currently.
283 More configuration details can be found [here](09-object-types.md#objecttype-influxdbwriter).
285 #### Instance Tagging <a id="influxdb-writer-instance-tags"></a>
287 Consider the following service check:
290 apply Service "disk" for (disk => attributes in host.vars.disks) {
291 import "generic-service"
292 check_command = "disk"
293 display_name = "Disk " + disk
294 vars.disk_partitions = disk
295 assign where host.vars.disks
299 This is a typical pattern for checking individual disks, NICs, SSL certificates etc associated
300 with a host. What would be useful is to have the data points tagged with the specific instance
301 for that check. This would allow you to query time series data for a check on a host and for a
302 specific instance e.g. /dev/sda. To do this quite simply add the instance to the service variables:
305 apply Service "disk" for (disk => attributes in host.vars.disks) {
312 Then modify your writer configuration to add this tag to your data points if the instance variable
313 is associated with the service:
316 object InfluxdbWriter "influxdb" {
319 measurement = "$service.check_command$"
321 hostname = "$host.name$"
322 service = "$service.name$"
323 instance = "$service.vars.instance$"
330 ### Elastic Stack Integration <a id="elastic-stack-integration"></a>
332 [Icingabeat](https://github.com/icinga/icingabeat) is an Elastic Beat that fetches data
333 from the Icinga 2 API and sends it either directly to [Elasticsearch](https://www.elastic.co/products/elasticsearch)
334 or [Logstash](https://www.elastic.co/products/logstash).
338 * [Logstash output](https://github.com/Icinga/logstash-output-icinga) for the Icinga 2 API.
339 * [Logstash Grok Pattern](https://github.com/Icinga/logstash-grok-pattern) for Icinga 2 logs.
341 #### Elasticsearch Writer <a id="elasticsearch-writer"></a>
343 This feature forwards check results, state changes and notification events
344 to an [Elasticsearch](https://www.elastic.co/products/elasticsearch) installation over its HTTP API.
346 The check results include parsed performance data metrics if enabled.
350 > Elasticsearch 5.x is required. This feature has been successfully tested with Elasticsearch 5.6.4.
352 Enable the feature and restart Icinga 2.
355 # icinga2 feature enable elasticsearch
358 The default configuration expects an Elasticsearch instance running on `localhost` on port `9200
359 and writes to an index called `icinga2`.
361 More configuration details can be found [here](09-object-types.md#objecttype-elasticsearchwriter).
363 #### Current Elasticsearch Schema <a id="elastic-writer-schema"></a>
365 The following event types are written to Elasticsearch:
367 * icinga2.event.checkresult
368 * icinga2.event.statechange
369 * icinga2.event.notification
371 Performance data metrics must be explicitly enabled with the `enable_send_perfdata`
374 Metric values are stored like this:
376 check_result.perfdata.<perfdata-label>.value
378 The following characters are escaped in perfdata labels:
380 Character | Escaped character
381 --------------|--------------------------
387 Note that perfdata labels may contain dots (`.`) allowing to
388 add more subsequent levels inside the tree.
389 `::` adds support for [multi performance labels](http://my-plugin.de/wiki/projects/check_multi/configuration/performance)
390 and is therefore replaced by `.`.
392 Icinga 2 automatically adds the following threshold metrics
395 check_result.perfdata.<perfdata-label>.min
396 check_result.perfdata.<perfdata-label>.max
397 check_result.perfdata.<perfdata-label>.warn
398 check_result.perfdata.<perfdata-label>.crit
400 ### Graylog Integration <a id="graylog-integration"></a>
402 #### GELF Writer <a id="gelfwriter"></a>
404 The `Graylog Extended Log Format` (short: [GELF](http://docs.graylog.org/en/latest/pages/gelf.html))
405 can be used to send application logs directly to a TCP socket.
407 While it has been specified by the [Graylog](https://www.graylog.org) project as their
408 [input resource standard](http://docs.graylog.org/en/latest/pages/sending_data.html), other tools such as
409 [Logstash](https://www.elastic.co/products/logstash) also support `GELF` as
410 [input type](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-gelf.html).
412 You can enable the feature using
414 # icinga2 feature enable gelf
416 By default the `GelfWriter` object expects the GELF receiver to listen at `127.0.0.1` on TCP port `12201`.
417 The default `source` attribute is set to `icinga2`. You can customize that for your needs if required.
419 Currently these events are processed:
425 ### OpenTSDB Writer <a id="opentsdb-writer"></a>
427 While there are some OpenTSDB collector scripts and daemons like tcollector available for
428 Icinga 1.x it's more reasonable to directly process the check and plugin performance
429 in memory in Icinga 2. Once there are new metrics available, Icinga 2 will directly
430 write them to the defined TSDB TCP socket.
432 You can enable the feature using
434 # icinga2 feature enable opentsdb
436 By default the `OpenTsdbWriter` object expects the TSD to listen at
437 `127.0.0.1` on port `4242`.
439 The current naming schema is
441 icinga.host.<metricname>
442 icinga.service.<servicename>.<metricname>
444 for host and service checks. The tag host is always applied.
446 To make sure Icinga 2 writes a valid metric into OpenTSDB some characters are replaced
447 with `_` in the target name:
451 The resulting name in OpenTSDB might look like:
453 www-01 / http-cert / response time
454 icinga.http_cert.response_time
456 In addition to the performance data retrieved from the check plugin, Icinga 2 sends
457 internal check statistic data to OpenTSDB:
460 -------------------|------------------------------------------
461 current_attempt | current check attempt
462 max_check_attempts | maximum check attempts until the hard state is reached
463 reachable | checked object is reachable
464 downtime_depth | number of downtimes this object is in
465 acknowledgement | whether the object is acknowledged or not
466 execution_time | check execution time
467 latency | check latency
468 state | current state of the checked object
469 state_type | 0=SOFT, 1=HARD state
471 While reachable, state and state_type are metrics for the host or service the
472 other metrics follow the current naming schema
474 icinga.check.<metricname>
476 with the following tags
479 --------|------------------------------------------
480 type | the check type, one of [host, service]
481 host | hostname, the check ran on
482 service | the service name (if type=service)
486 > You might want to set the tsd.core.auto_create_metrics setting to `true`
487 > in your opentsdb.conf configuration file.
490 ## Livestatus <a id="setting-up-livestatus"></a>
492 The [MK Livestatus](https://mathias-kettner.de/checkmk_livestatus.html) project
493 implements a query protocol that lets users query their Icinga instance for
494 status information. It can also be used to send commands.
496 The Livestatus component that is distributed as part of Icinga 2 is a
497 re-implementation of the Livestatus protocol which is compatible with MK
502 > Only install the Livestatus feature if your web interface or addon requires
504 > [Icinga Web 2](02-getting-started.md#setting-up-icingaweb2) does not need
507 Details on the available tables and attributes with Icinga 2 can be found
508 in the [Livestatus Schema](24-appendix.md#schema-livestatus) section.
510 You can enable Livestatus using icinga2 feature enable:
512 # icinga2 feature enable livestatus
514 After that you will have to restart Icinga 2:
516 # systemctl restart icinga2
518 By default the Livestatus socket is available in `/var/run/icinga2/cmd/livestatus`.
520 In order for queries and commands to work you will need to add your query user
521 (e.g. your web server) to the `icingacmd` group:
523 # usermod -a -G icingacmd www-data
525 The Debian packages use `nagios` as the user and group name. Make sure to change `icingacmd` to
526 `nagios` if you're using Debian.
528 Change `www-data` to the user you're using to run queries.
530 In order to use the historical tables provided by the livestatus feature (for example, the
531 `log` table) you need to have the `CompatLogger` feature enabled. By default these logs
532 are expected to be in `/var/log/icinga2/compat`. A different path can be set using the
533 `compat_log_path` configuration attribute.
535 # icinga2 feature enable compatlog
538 ### Livestatus Sockets <a id="livestatus-sockets"></a>
540 Other to the Icinga 1.x Addon, Icinga 2 supports two socket types
542 * Unix socket (default)
545 Details on the configuration can be found in the [LivestatusListener](09-object-types.md#objecttype-livestatuslistener)
546 object configuration.
548 ### Livestatus GET Queries <a id="livestatus-get-queries"></a>
552 > All Livestatus queries require an additional empty line as query end identifier.
553 > The `nc` tool (`netcat`) provides the `-U` parameter to communicate using
556 There also is a Perl module available in CPAN for accessing the Livestatus socket
557 programmatically: [Monitoring::Livestatus](http://search.cpan.org/~nierlein/Monitoring-Livestatus-0.74/)
560 Example using the unix socket:
562 # echo -e "GET services\n" | /usr/bin/nc -U /var/run/icinga2/cmd/livestatus
564 Example using the tcp socket listening on port `6558`:
566 # echo -e 'GET services\n' | netcat 127.0.0.1 6558
568 # cat servicegroups <<EOF
573 (cat servicegroups; sleep 1) | netcat 127.0.0.1 6558
576 ### Livestatus COMMAND Queries <a id="livestatus-command-queries"></a>
578 A list of available external commands and their parameters can be found [here](24-appendix.md#external-commands-list-detail)
580 $ echo -e 'COMMAND <externalcommandstring>' | netcat 127.0.0.1 6558
583 ### Livestatus Filters <a id="livestatus-filters"></a>
587 Operator | Negate | Description
588 ----------|------------------------
591 =~ | !=~ | Equality ignoring case
592 ~~ | !~~ | Regex ignoring case
595 <= | | Less than or equal
596 >= | | Greater than or equal
599 ### Livestatus Stats <a id="livestatus-stats"></a>
601 Schema: "Stats: aggregatefunction aggregateattribute"
603 Aggregate Function | Description
604 -------------------|--------------
609 std | standard deviation
610 suminv | sum (1 / value)
611 avginv | suminv / count
612 count | ordinary default for any stats query if not aggregate function defined
617 Filter: has_been_checked = 1
618 Filter: check_type = 0
619 Stats: sum execution_time
621 Stats: sum percent_state_change
622 Stats: min execution_time
624 Stats: min percent_state_change
625 Stats: max execution_time
627 Stats: max percent_state_change
629 ResponseHeader: fixed16
631 ### Livestatus Output <a id="livestatus-output"></a>
635 CSV output uses two levels of array separators: The members array separator
636 is a comma (1st level) while extra info and host|service relation separator
637 is a pipe (2nd level).
639 Separators can be set using ASCII codes like:
641 Separators: 10 59 44 124
647 ### Livestatus Error Codes <a id="livestatus-error-codes"></a>
650 ----------|--------------
652 404 | Table does not exist
653 452 | Exception on query
655 ### Livestatus Tables <a id="livestatus-tables"></a>
657 Table | Join |Description
658 --------------|-----------|----------------------------
659 hosts | | host config and status attributes, services counter
660 hostgroups | | hostgroup config, status attributes and host/service counters
661 services | hosts | service config and status attributes
662 servicegroups | | servicegroup config, status attributes and service counters
663 contacts | | contact config and status attributes
664 contactgroups | | contact config, members
665 commands | | command name and line
666 status | | programstatus, config and stats
667 comments | services | status attributes
668 downtimes | services | status attributes
669 timeperiods | | name and is inside flag
670 endpoints | | config and status attributes
671 log | services, hosts, contacts, commands | parses [compatlog](09-object-types.md#objecttype-compatlogger) and shows log attributes
672 statehist | hosts, services | parses [compatlog](09-object-types.md#objecttype-compatlogger) and aggregates state change attributes
673 hostsbygroup | hostgroups | host attributes grouped by hostgroup and its attributes
674 servicesbygroup | servicegroups | service attributes grouped by servicegroup and its attributes
675 servicesbyhostgroup | hostgroups | service attributes grouped by hostgroup and its attributes
677 The `commands` table is populated with `CheckCommand`, `EventCommand` and `NotificationCommand` objects.
679 A detailed list on the available table attributes can be found in the [Livestatus Schema documentation](24-appendix.md#schema-livestatus).
682 ## Status Data Files <a id="status-data"></a>
684 Icinga 1.x writes object configuration data and status data in a cyclic
685 interval to its `objects.cache` and `status.dat` files. Icinga 2 provides
686 the `StatusDataWriter` object which dumps all configuration objects and
687 status updates in a regular interval.
689 # icinga2 feature enable statusdata
691 If you are not using any web interface or addon which uses these files,
692 you can safely disable this feature.
695 ## Compat Log Files <a id="compat-logging"></a>
697 The Icinga 1.x log format is considered being the `Compat Log`
698 in Icinga 2 provided with the `CompatLogger` object.
700 These logs are used for informational representation in
701 external web interfaces parsing the logs, but also to generate
702 SLA reports and trends.
703 The [Livestatus](14-features.md#setting-up-livestatus) feature uses these logs
704 for answering queries to historical tables.
706 The `CompatLogger` object can be enabled with
708 # icinga2 feature enable compatlog
710 By default, the Icinga 1.x log file called `icinga.log` is located
711 in `/var/log/icinga2/compat`. Rotated log files are moved into
712 `var/log/icinga2/compat/archives`.
714 ## Check Result Files <a id="check-result-files"></a>
716 Icinga 1.x writes its check result files to a temporary spool directory
717 where they are processed in a regular interval.
718 While this is extremely inefficient in performance regards it has been
719 rendered useful for passing passive check results directly into Icinga 1.x
720 skipping the external command pipe.
722 Several clustered/distributed environments and check-aggregation addons
723 use that method. In order to support step-by-step migration of these
724 environments, Icinga 2 supports the `CheckResultReader` object.
726 There is no feature configuration available, but it must be defined
727 on-demand in your Icinga 2 objects configuration.
729 object CheckResultReader "reader" {
730 spool_dir = "/data/check-results"