1 # Technical Concepts <a id="technical-concepts"></a>
3 This chapter provides technical concepts and design insights
4 into specific Icinga 2 components such as:
6 * [Application](19-technical-concepts.md#technical-concepts-application)
7 * [Configuration](19-technical-concepts.md#technical-concepts-configuration)
8 * [Features](19-technical-concepts.md#technical-concepts-features)
9 * [Check Scheduler](19-technical-concepts.md#technical-concepts-check-scheduler)
10 * [Cluster](19-technical-concepts.md#technical-concepts-cluster)
11 * [TLS Network IO](19-technical-concepts.md#technical-concepts-tls-network-io)
13 ## Application <a id="technical-concepts-application"></a>
15 ### CLI Commands <a id="technical-concepts-application-cli-commands"></a>
17 The Icinga 2 application is managed with different CLI sub commands.
18 `daemon` takes care about loading the configuration files, running the
19 application as daemon, etc.
20 Other sub commands allow to enable features, generate and request
21 TLS certificates or enter the debug console.
23 The main entry point for each CLI command parses the command line
24 parameters and then triggers the required actions.
26 ### daemon CLI command <a id="technical-concepts-application-cli-commands-daemon"></a>
28 This CLI command loads the configuration files, starting with `icinga2.conf`.
29 The [configuration compiler](19-technical-concepts.md#technical-concepts-configuration) parses the
30 file and detects additional file includes, constants, and any other DSL
33 At this stage, the configuration will already be checked against the
34 defined grammar in the scanner, and custom object validators will also be
37 If the user provided `-C/--validate`, the CLI command returns with the
40 When running as daemon, additional parameters are checked, e.g. whether
41 this application was triggered by a reload, needs to daemonize with fork()
42 involved and update the object's authority. The latter is important for
43 HA-enabled cluster zones.
45 ## Configuration <a id="technical-concepts-configuration"></a>
47 ### Lexer <a id="technical-concepts-configuration-lexer"></a>
49 The lexer stage does not understand the DSL itself, it only
50 maps specific character sequences into identifiers.
52 This allows Icinga to detect the beginning of a string with `"`,
53 reading the following characters and determining the end of the
54 string with again `"`.
56 Other parts covered by the lexer a escape sequences insides a string,
59 The lexer also identifiers logical operators, e.g. `&` or `in`,
60 specific keywords like `object`, `import`, etc. and comment blocks.
62 Please check `lib/config/config_lexer.ll` for details.
64 Icinga uses [Flex](https://github.com/westes/flex) in the first stage.
66 > Flex (The Fast Lexical Analyzer)
68 > Flex is a fast lexical analyser generator. It is a tool for generating programs
69 > that perform pattern-matching on text. Flex is a free (but non-GNU) implementation
70 > of the original Unix lex program.
72 ### Parser <a id="technical-concepts-configuration-parser"></a>
74 The parser stage puts the identifiers from the lexer into more
75 context with flow control and sequences.
77 The following comparison is parsed into a left term, an operator
84 The DSL contains many elements which require a specific order,
85 and sometimes only a left term for example.
87 The parser also takes care of parsing an object declaration for
88 example. It already knows from the lexer that `object` marks the
89 beginning of an object. It then expects a type string afterwards,
90 and the object name - which can be either a string with double quotes
91 or a previously defined constant.
93 An opening bracket `{` in this specific context starts the object
94 scope, which also is stored for later scope specific variable access.
96 If there's an apply rule defined, this follows the same principle.
97 The config parser detects the scope of an apply rule and generates
98 Icinga 2 C++ code for the parsed string tokens.
101 assign where host.vars.sla == "24x7"
104 is parsed into an assign token identifier, and the string expression
105 is compiled into a new `ApplyExpression` object.
107 The flow control inside the parser ensures that for example `ignore where`
108 can only be defined when a previous `assign where` was given - or when
109 inside an apply for rule.
111 Another example are specific object types which allow assign expression,
112 specifically group objects. Others objects must throw a configuration error.
114 Please check `lib/config/config_parser.yy` for more details,
115 and the [language reference](17-language-reference.md#language-reference) chapter for
116 documented DSL keywords and sequences.
118 > Icinga uses [Bison](https://en.wikipedia.org/wiki/GNU_bison) as parser generator
119 > which reads a specification of a context-free language, warns about any parsing
120 > ambiguities, and generates a parser in C++ which reads sequences of tokens and
121 > decides whether the sequence conforms to the syntax specified by the grammar.
124 ### Compiler <a id="technical-concepts-configuration-compiler"></a>
126 The config compiler initializes the scanner inside the [lexer](19-technical-concepts.md#technical-concepts-configuration-lexer)
129 The configuration files are parsed into memory from inside the [daemon CLI command](19-technical-concepts.md#technical-concepts-application-cli-commands-daemon)
130 which invokes the config validation in `ValidateConfigFiles()`. This compiles the
131 files into an AST expression which is executed.
133 At this stage, the expressions generate so-called "config items" which
134 are a pre-stage of the later compiled object.
136 `ConfigItem::CommitItems` takes care of committing the items, and doing a
137 rollback on failure. It also checks against matching apply rules from the previous run
138 and generates statistics about the objects which can be seen by the config validation.
140 `ConfigItem::CommitNewItems` collects the registered types and items,
141 and checks for a specific required order, e.g. a service object needs
144 The following stages happen then:
146 - **Commit**: A workqueue then commits the items in a parallel fashion for this specific type. The object gets its name, and the AST expression is executed. It is then registered into the item into `m_Object` as reference.
147 - **OnAllConfigLoaded**: Special signal for each object to pre-load required object attributes, resolve group membership, initialize functions and timers.
148 - **CreateChildObjects**: Run apply rules for this specific type.
149 - **CommitNewItems**: Apply rules may generate new config items, this is to ensure that they again run through the stages.
151 Note that the items are now committed and the configuration is validated and loaded
152 into memory. The final config objects are not yet activated though.
154 This only happens after the validation, when the application is about to be run
155 with `ConfigItem::ActivateItems`.
157 Each item has an object created in `m_Object` which is checked in a loop.
158 Again, the dependency order of activated objects is important here, e.g. logger features come first, then
159 config objects and last the checker, api, etc. features. This is done by sorting the objects
160 based on their type specific activation priority.
162 The following signals are triggered in the stages:
164 - **PreActivate**: Setting the `active` flag for the config object.
165 - **Activate**: Calls `Start()` on the object, sets the local HA authority and notifies subscribers that this object is now activated (e.g. for config updates in the DB backend).
169 ## Features <a id="technical-concepts-features"></a>
171 Features are implemented in specific libraries and can be enabled
174 Features either write specific data or receive data.
176 Examples for writing data: [DB IDO](14-features.md#db-ido), [Graphite](14-features.md#graphite-carbon-cache-writer), [InfluxDB](14-features.md#influxdb-writer). [GELF](14-features.md#gelfwriter), etc.
177 Examples for receiving data: [REST API](12-icinga2-api.md#icinga2-api), etc.
179 The implementation of features makes use of existing libraries
180 and functionality. This makes the code more abstract, but shorter
183 Features register callback functions on specific events they want
184 to handle. For example the `GraphiteWriter` feature subscribes to
185 new CheckResult events.
187 Each time Icinga 2 receives and processes a new check result, this
188 event is triggered and forwarded to all subscribers.
190 The GraphiteWriter feature calls the registered function and processes
191 the received data. Features which connect Icinga 2 to external interfaces
192 normally parse and reformat the received data into an applicable format.
194 Since this check result signal is blocking, many of the features include a work queue
195 with asynchronous task handling.
197 The GraphiteWriter uses a TCP socket to communicate with the carbon cache
198 daemon of Graphite. The InfluxDBWriter is instead writing bulk metric messages
199 to InfluxDB's HTTP API, similar to Elasticsearch.
202 ## Check Scheduler <a id="technical-concepts-check-scheduler"></a>
204 The check scheduler starts a thread which loops forever. It waits for
205 check events being inserted into `m_IdleCheckables`.
207 If the current pending check event number is larger than the configured
208 max concurrent checks, the thread waits up until it there's slots again.
210 In addition, further checks on enabled checks, check periods, etc. are
211 performed. Once all conditions have passed, the next check timestamp is
212 calculated and updated. This also is the timestamp where Icinga expects
213 a new check result ("freshness check").
215 The object is removed from idle checkables, and inserted into the
216 pending checkables list. This can be seen via REST API metrics for the
217 checker component feature as well.
219 The actual check execution happens asynchronously using the application's
222 Once the check returns, it is removed from pending checkables and again
223 inserted into idle checkables. This ensures that the scheduler takes this
224 checkable event into account in the next iteration.
226 ### Start <a id="technical-concepts-check-scheduler-start"></a>
228 When checkable objects get activated during the startup phase,
229 the checker feature registers a handler for this event. This is due
230 to the fact that the `checker` feature is fully optional, and e.g. not
231 used on command endpoint clients.
233 Whenever such an object activation signal is triggered, Icinga 2 checks
234 whether it is [authoritative for this object](19-technical-concepts.md#technical-concepts-cluster-ha-object-authority).
235 This means that inside an HA enabled zone with two endpoints, only non-paused checkable objects are
236 actively inserted into the idle checkable list for the check scheduler.
238 ### Initial Check <a id="technical-concepts-check-scheduler-initial"></a>
240 When a new checkable object (host or service) is initially added to the
241 configuration, Icinga 2 performs the following during startup:
243 * `Checkable::Start()` is called and calculates the first check time
244 * With a spread delta, the next check time is actually set.
246 If the next check should happen within a time frame of 60 seconds,
247 Icinga 2 calculates a delta from a random value. The minimum of `check_interval`
248 and 60 seconds is used as basis, multiplied with a random value between 0 and 1.
250 In the best case, this check gets immediately executed after application start.
251 The worst case scenario is that the check is scheduled 60 seconds after start
254 The reasons for delaying and spreading checks during startup is that
255 the application typically needs more resources at this time (cluster connections,
256 feature warmup, initial syncs, etc.). Immediate check execution with
257 thousands of checks could lead into performance problems, and additional
258 events for each received check results.
260 Therefore the initial check window is 60 seconds on application startup,
261 random seed for all checkables. This is not predictable over multiple restarts
262 for specific checkable objects, the delta changes every time.
264 ### Scheduling Offset <a id="technical-concepts-check-scheduler-offset"></a>
266 There's a high chance that many checkable objects get executed at the same time
267 and interval after startup. The initial scheduling spreads that a little, but
268 Icinga 2 also attempts to ensure to keep fixed intervals, even with high check latency.
270 During startup, Icinga 2 calculates the scheduling offset from a random number:
272 * `Checkable::Checkable()` calls `SetSchedulingOffset()` with `Utility::Random()`
273 * The offset is a pseudo-random integral value between `0` and `RAND_MAX`.
275 Whenever the next check time is updated with `Checkable::UpdateNextCheck()`,
276 the scheduling offset is taken into account.
278 Depending on the state type (SOFT or HARD), either the `retry_interval` or `check_interval`
279 is used. If the interval is greater than 1 second, the time adjustment is calculated in the
282 `now * 100 + offset` divided by `interval * 100`, using the remainder (that's what `fmod()` is for)
283 and dividing this again onto base 100.
285 Example: offset is 6500, interval 300, now is 1542190472.
288 1542190472 * 100 + 6500 = 154219053714
290 154219053714 / 30000 = 5140635.1238
292 (5140635.1238 - 5140635.0) * 30000 = 3714
296 37.15 seconds as an offset would be far too much, so this is again used as a calculation divider for the
297 real offset with the base of 5 times the actual interval.
299 Again, the remainder is calculated from the offset and `interval * 5`. This is divided onto base 100 again,
300 with an additional 0.5 seconds delay.
302 Example: offset is 6500, interval 300.
305 6500 / 300 = 21.666666666666667
306 (21.666666666666667 - 21.0) * 300 = 200
311 The minimum value between the first adjustment and the second offset calculation based on the interval is
312 taken, in the above example `2.5` wins.
314 The actual next check time substracts the adjusted time from the future interval addition to provide
315 a more widespread scheduling time among all checkable objects.
317 `nextCheck = now - adj + interval`
319 You may ask, what other values can happen with this offset calculation. Consider calculating more examples
320 with different interval settings.
322 Example: offset is 34567, interval 60, now is 1542190472.
325 1542190472 * 100 + 34567 = 154219081767
327 154219081767 / 6000 = 25703180.2945
328 (25703180.2945 - 25703180.0) * 6000 / 100 = 17.67
330 34567 / 60 = 576.116666666666667
331 (576.116666666666667 - 576.0) * 60 / 100 + 0.5 = 1.2
334 `1m` interval starts at `now + 1.2s`.
336 Example: offset is 12345, interval 86400, now is 1542190472.
339 1542190472 * 100 + 12345 = 154219059545
340 86400 * 100 = 8640000
341 154219059545 / 8640000 = 17849.428188078703704
342 (17849.428188078703704 - 17849) * 8640000 = 3699545
343 3699545 / 100 = 36995.45
345 12345 / 86400 = 0.142881944444444
346 0.142881944444444 * 86400 / 100 + 0.5 = 123.95
349 `1d` interval starts at `now + 2m4s`.
353 > In case you have a better algorithm at hand, feel free to discuss this in a PR on GitHub.
354 > It needs to fulfill two things: 1) spread and shuffle execution times on each `next_check` update
355 > 2) not too narrowed window for both long and short intervals
356 > Application startup and initial checks need to be handled with care in a slightly different
359 When `SetNextCheck()` is called, there are signals registered. One of them sits
360 inside the `CheckerComponent` class whose handler `CheckerComponent::NextCheckChangedHandler()`
361 deletes/inserts the next check event from the scheduling queue. This basically
362 is a list with multiple indexes with the keys for scheduling info and the object.
365 ### Check Latency and Execution Time <a id="technical-concepts-check-scheduler-latency"></a>
367 Each check command execution logs the start and end time where
368 Icinga 2 (and the end user) is able to calculate the plugin execution time from it.
371 GetExecutionEnd() - GetExecutionStart()
374 The higher the execution time, the higher the command timeout must be set. Furthermore
375 users and developers are encouraged to look into plugin optimizations to minimize the
376 execution time. Sometimes it is better to let an external daemon/script do the checks
377 and feed them back via REST API.
379 Icinga 2 stores the scheduled start and end time for a check. If the actual
380 check execution time differs from the scheduled time, e.g. due to performance
381 problems or limited execution slots (concurrent checks), this value is stored
382 and computed from inside the check result.
384 The difference between the two deltas is called `check latency`.
387 (GetScheduleEnd() - GetScheduleStart()) - CalculateExecutionTime()
391 ## Cluster <a id="technical-concepts-cluster"></a>
393 ### Communication <a id="technical-concepts-cluster-communication"></a>
395 Icinga 2 uses its own certificate authority (CA) by default. The
396 public and private CA keys can be generated on the signing master.
398 Each node certificate must be signed by the private CA key.
400 Note: The following description uses `parent node` and `child node`.
401 This also applies to nodes in the same cluster zone.
403 During the connection attempt, an SSL handshake is performed.
404 If the public certificate of a child node is not signed by the same
405 CA, the child node is not trusted and the connection will be closed.
407 If the SSL handshake succeeds, the parent node reads the
408 certificate's common name (CN) of the child node and looks for
409 a local Endpoint object name configuration.
411 If there is no Endpoint object found, further communication
412 (runtime and config sync, etc.) is terminated.
414 The child node also checks the CN from the parent node's public
415 certificate. If the child node does not find any local Endpoint
416 object name configuration, it will not trust the parent node.
418 Both checks prevent accepting cluster messages from an untrusted
421 If an Endpoint match was found, there is one additional security
422 mechanism in place: Endpoints belong to a Zone hierarchy.
424 Several cluster messages can only be sent "top down", others like
425 check results are allowed being sent from the child to the parent node.
427 Once this check succeeds the cluster messages are exchanged and processed.
430 ### CSR Signing <a id="technical-concepts-cluster-csr-signing"></a>
432 In order to make things easier, Icinga 2 provides built-in methods
433 to allow child nodes to request a signed certificate from the
436 Icinga 2 v2.8 introduces the possibility to request certificates
437 from indirectly connected nodes. This is required for multi level
438 cluster environments with masters, satellites and clients.
440 CSR Signing in general starts with the master setup. This step
441 ensures that the master is in a working CSR signing state with:
443 * public and private CA key in `/var/lib/icinga2/ca`
444 * private `TicketSalt` constant defined inside the `api` feature
445 * Cluster communication is ready and Icinga 2 listens on port 5665
447 The child node setup which is run with CLI commands will now
448 attempt to connect to the parent node. This is not necessarily
449 the signing master instance, but could also be a parent satellite node.
451 During this process the child node asks the user to verify the
452 parent node's public certificate to prevent MITM attacks.
454 There are two methods to request signed certificates:
456 * Add the ticket into the request. This ticket was generated on the master
457 beforehand and contains hashed details for which client it has been created.
458 The signing master uses this information to automatically sign the certificate
461 * Do not add a ticket into the request. It will be sent to the signing master
462 which stores the pending request. Manual user interaction with CLI commands
463 is necessary to sign the request.
465 The certificate request is sent as `pki::RequestCertificate` cluster
466 message to the parent node.
468 If the parent node is not the signing master, it stores the request
469 in `/var/lib/icinga2/certificate-requests` and forwards the
470 cluster message to its parent node.
472 Once the message arrives on the signing master, it first verifies that
473 the sent certificate request is valid. This is to prevent unwanted errors
474 or modified requests from the "proxy" node.
476 After verification, the signing master checks if the request contains
477 a valid signing ticket. It hashes the certificate's common name and
478 compares the value to the received ticket number.
480 If the ticket is valid, the certificate request is immediately signed
481 with CA key. The request is sent back to the client inside a `pki::UpdateCertificate`
484 If the child node was not the certificate request origin, it only updates
485 the cached request for the child node and send another cluster message
486 down to its child node (e.g. from a satellite to a client).
489 If no ticket was specified, the signing master waits until the
490 `ca sign` CLI command manually signed the certificate.
494 > Push notifications for manual request signing is not yet implemented (TODO).
496 Once the child node reconnects it synchronizes all signed certificate requests.
497 This takes some minutes and requires all nodes to reconnect to each other.
500 #### CSR Signing: Clients without parent connection <a id="technical-concepts-cluster-csr-signing-clients-no-connection"></a>
502 There is an additional scenario: The setup on a child node does
503 not necessarily need a connection to the parent node.
505 This mode leaves the node in a semi-configured state. You need
506 to manually copy the master's public CA key into `/var/lib/icinga2/certs/ca.crt`
507 on the client before starting Icinga 2.
509 The parent node needs to actively connect to the child node.
510 Once this connections succeeds, the child node will actively
511 request a signed certificate.
513 The update procedure works the same way as above.
515 ### High Availability <a id="technical-concepts-cluster-ha"></a>
517 General high availability is automatically enabled between two endpoints in the same
520 **This requires the same configuration and enabled features on both nodes.**
522 HA zone members trust each other and share event updates as cluster messages.
523 This includes for example check results, next check timestamp updates, acknowledgements
526 This ensures that both nodes are synchronized. If one node goes away, the
527 remaining node takes over and continues as normal.
529 #### High Availability: Object Authority <a id="technical-concepts-cluster-ha-object-authority"></a>
531 Cluster nodes automatically determine the authority for configuration
532 objects. By default, all config objects are set to `HARunEverywhere` and
533 as such the object authority is true for any config object on any instance.
535 Specific objects can override and influence this setting, e.g. with `HARunOnce`
536 instead prior to config object activation.
538 This is done when the daemon starts and in a regular interval inside
539 the ApiListener class, specifically calling `ApiListener::UpdateObjectAuthority()`.
541 The algorithm works like this:
543 * Determine whether this instance is assigned to a local zone and endpoint.
544 * Collects all endpoints in this zone if they are connected.
545 * If there's two endpoints, but only us seeing ourselves and the application start is less than 60 seconds in the past, do nothing (wait for cluster reconnect to take place, grace period).
546 * Sort the collected endpoints by name.
547 * Iterate over all config types and their respective objects
548 * Ignore !active objects
549 * Ignore objects which are !HARunOnce. This means, they can run multiple times in a zone and don't need an authority update.
550 * If this instance doesn't have a local zone, set authority to true. This is for non-clustered standalone environments where everything belongs to this instance.
551 * Calculate the object authority based on the connected endpoint names.
552 * Set the authority (true or false)
554 The object authority calculation works "offline" without any message exchange.
555 Each instance alculates the SDBM hash of the config object name, puts that in contrast
556 modulo the connected endpoints size.
557 This index is used to lookup the corresponding endpoint in the connected endpoints array,
558 including the local endpoint. Whether the local endpoint is equal to the selected endpoint,
559 or not, this sets the authority to `true` or `false`.
562 authority = endpoints[Utility::SDBM(object->GetName()) % endpoints.size()] == my_endpoint;
565 `ConfigObject::SetAuthority(bool authority)` triggers the following events:
567 * Authority is true and object now paused: Resume the object and set `paused` to `false`.
568 * Authority is false, object not paused: Pause the object and set `paused` to true.
570 **This results in activated but paused objects on one endpoint.** You can verify
571 that by querying the `paused` attribute for all objects via REST API
572 or debug console on both endpoints.
574 Endpoints inside a HA zone calculate the object authority independent from each other.
575 This object authority is important for selected features explained below.
577 Since features are configuration objects too, you must ensure that all nodes
578 inside the HA zone share the same enabled features. If configured otherwise,
579 one might have a checker feature on the left node, nothing on the right node.
580 This leads to late check results because one half is not executed by the right
581 node which holds half of the object authorities.
583 By default, features are enabled to "Run-Everywhere". Specific features which
584 support HA awareness, provide the `enable_ha` configuration attribute. When `enable_ha`
585 is set to `true` (usually the default), "Run-Once" is set and the feature pauses on one side.
588 vim /etc/icinga2/features-enabled/graphite.conf
590 object GraphiteWriter "graphite" {
596 Once such a feature is paused, there won't be any more event handling, e.g. the Elasticsearch
597 feature won't process any checkresults nor write to the Elasticsearch REST API.
599 When the cluster connection drops, the feature configuration object is updated with
600 the new object authority by the ApiListener timer and resumes its operation. You can see
601 that by grepping the log file for `resumed` and `paused`.
604 [2018-10-24 13:28:28 +0200] information/GraphiteWriter: 'g-ha' paused.
608 [2018-10-24 13:28:28 +0200] information/GraphiteWriter: 'g-ha' resumed.
611 Specific features with HA capabilities are explained below.
613 ### High Availability: Checker <a id="technical-concepts-cluster-ha-checker"></a>
615 The `checker` feature only executes checks for `Checkable` objects (Host, Service)
616 where it is authoritative.
618 That way each node only executes checks for a segment of the overall configuration objects.
620 The cluster message routing ensures that all check results are synchronized
621 to nodes which are not authoritative for this configuration object.
624 ### High Availability: Notifications <a id="technical-concepts-cluster-notifications"></a>
626 The `notification` feature only sends notifications for `Notification` objects
627 where it is authoritative.
629 That way each node only executes notifications for a segment of all notification objects.
631 Notified users and other event details are synchronized throughout the cluster.
632 This is required if for example the DB IDO feature is active on the other node.
634 ### High Availability: DB IDO <a id="technical-concepts-cluster-ha-ido"></a>
636 If you don't have HA enabled for the IDO feature, both nodes will
637 write their status and historical data to their own separate database
640 In order to avoid data separation and a split view (each node would require its
641 own Icinga Web 2 installation on top), the high availability option was added
642 to the DB IDO feature. This is enabled by default with the `enable_ha` setting.
644 This requires a central database backend. Best practice is to use a MySQL cluster
647 Both Icinga 2 nodes require the connection and credential details configured in
648 their DB IDO feature.
650 During startup Icinga 2 calculates whether the feature configuration object
651 is authoritative on this node or not. The order is an alpha-numeric
652 comparison, e.g. if you have `master1` and `master2`, Icinga 2 will enable
653 the DB IDO feature on `master2` by default.
655 If the connection between endpoints drops, the object authority is re-calculated.
657 In order to prevent data duplication in a split-brain scenario where both
658 nodes would write into the same database, there is another safety mechanism
661 The split-brain decision which node will write to the database is calculated
662 from a quorum inside the `programstatus` table. Each node
663 verifies whether the `endpoint_name` column is not itself on database connect.
664 In addition to that the DB IDO feature compares the `last_update_time` column
665 against the current timestamp plus the configured `failover_timeout` offset.
667 That way only one active DB IDO feature writes to the database, even if they
668 are not currently connected in a cluster zone. This prevents data duplication
669 in historical tables.
671 ### Health Checks <a id="technical-concepts-cluster-health-checks"></a>
673 #### cluster-zone <a id="technical-concepts-cluster-health-checks-cluster-zone"></a>
675 This built-in check provides the possibility to check for connectivity between
678 If you for example need to know whether the `master` zone is connected and processing
679 messages with the child zone called `satellite` in this example, you can configure
680 the [cluster-zone](10-icinga-template-library.md#itl-icinga-cluster-zone) check as new service on all `master` zone hosts.
683 vim /etc/zones.d/master/host1.conf
685 object Service "cluster-zone-satellite" {
686 check_command = "cluster-zone"
689 vars.cluster_zone = "satellite"
693 The check itself changes to NOT-OK if one or more child endpoints in the child zone
694 are not connected to parent zone endpoints.
696 In addition to the overall connectivity check, the log lag is calculated based
697 on the to-be-sent replay log. Each instance stores that for its configured endpoint
700 This health check iterates over the target zone (`cluster_zone`) and their endpoints.
702 The log lag is greater than zero if
704 * the replay log synchronization is in progress and not yet finished or
705 * the endpoint is not connected, and no replay log sync happened (obviously).
707 The final log lag value is the worst value detected. If satellite1 has a log lag of
708 `1.5` and satellite2 only has `0.5`, the computed value will be `1.5.`.
710 You can control the check state by using optional warning and critical thresholds
711 for the log lag value.
713 If this service exists multiple times, e.g. for each master host object, the log lag
714 may differ based on the execution time. This happens for example on restart of
715 an instance when the log replay is in progress and a health check is executed at different
717 If the endpoint is not connected, both master instances may have saved a different log replay
718 position from the last synchronisation.
720 The lag value is returned as performance metric key `slave_lag`.
722 Icinga 2 v2.9+ adds more performance metrics for these values:
724 * `last_messages_sent` and `last_messages_received` as UNIX timestamp
725 * `sum_messages_sent_per_second` and `sum_messages_received_per_second`
726 * `sum_bytes_sent_per_second` and `sum_bytes_received_per_second`
730 ## REST API <a id="technical-concepts-rest-api"></a>
732 Icinga 2 provides its own HTTP server which shares the port 5665 with
733 the JSON-RPC cluster protocol.
737 ## TLS Network IO <a id="technical-concepts-tls-network-io"></a>
739 ### TLS Connection Handling <a id="technical-concepts-tls-network-io-connection-handling"></a>
741 TLS-Handshake timeouts occur if the server is busy with reconnect handling and other tasks which run in isolated threads. Icinga 2 uses threads in many ways, e.g. for timers to wake them up, wait for check results, etc.
743 In terms of the cluster communication, the following flow applies.
745 #### Master Connects <a id="technical-concepts-tls-network-io-connection-handling-master"></a>
747 * The master initializes the connection in a loop through all known zones it should connect to, extracting the endpoints and their host/port attribute.
748 * This calls `AddConnection()` whereas a `Tcp::Connect()` is called to create a TCP socket.
749 * A new thread is spawned for future connection handling, this binds `ApiListener::NewClientHandler()`.
750 * On top of the TCP socket, a new TLS stream is created.
751 * The master performs a `TLS->Handshake()`
752 * Certificates are verified and the endpoint name is compared to the CN.
755 #### Clients Processes Connection <a id="technical-concepts-tls-network-io-connection-handling-client"></a>
757 * The client listens for new incoming connections as 'TCP server' pattern inside `ListenerThreadProc()` with an endless loop.
758 * Once a new connection is detected, `TCP->Accept()` performs the initial socket establishment.
759 * A new thread is spawned for future connection handling, this binds `ApiListener::NewClientHandler()`, Role being Server.
760 * On top of the TCP socket, a new TLS stream is created.
761 * The client performs a `TLS->Handshake()`.
764 #### Data Transmission between Server and Client Role <a id="technical-concepts-tls-network-io-connection-handling-data-transmission"></a>
766 Once the TLS handshake and certificate verification is completed, the role is either `Client` or `Server`.
768 * Client: Send "Hello" message.
769 * Server: `TLS->WaitForData()` waits for incoming messages from the remote client.
771 `Client` in this case is the instance which initiated the connection. If the master is doing this,
772 the Icinga 2 client/agent acts as "server" which accepts incoming connections.
775 ### Asynchronous Socket IO <a id="technical-concepts-tls-network-io-async-socket-io"></a>
777 Everything runs through TLS, we don't use any "raw" connections nor plain message handling.
779 The TLS handshake and further read/write operations are not performed in a synchronous fashion
780 in the new client's thread. Instead, all clients share an asynchronous "event pool".
782 The TlsStream constructor registers a new SocketEvent by calling its constructor. It binds the
783 previously created TCP socket and itself into the created SocketEvent object.
785 `SocketEvent::InitializeEngine()` takes care of whether to use **epoll** (Linux) or
786 **poll** (BSD, Unix, Windows) as preferred socket poll engine. epoll has proven to be
787 faster on Linux systems.
789 The selected engine is stored as `l_SocketIOEngine` and later `Start()` ensures to do the following:
791 * Use a fixed number for creating IO threads.
792 * Create a `dumb_socketpair` which basically is a pipe from `in->out` and multiplexes the TCP socket
793 into a local Unix socket. This removes the complexity and slowlyness of the kernel dealing with the TCP stack and new events.
794 * `InitializeThread()` prepares epoll with `epoll_create`, socket descriptors and event mapping for later wakeup.
795 * Each event FD has its own "worker event thread" which deals with incoming data, called `ThreadProc` as endless loop.
797 By default, there are 8 of these worker threads.
799 In the `ThreadProc` loop, the following happens:
801 * `epoll_wait` gets called and provides an event whether new data is `ready` (via socket IO from the Kernel).
802 * The event created with `epoll_event` holds the `.fd.data` attribute which references the multiplexed event FD (and therefore tcp socket FD).
803 * All events in this cycle are stored with their descriptors in a list.
804 * Once the epoll loop is finished, the collected events are processed and the socketevent descriptor (which is the TlsStream object) calls `OnEvent()`.
806 #### On Socket Event State Machine <a id="technical-concepts-tls-network-io-async-socket-io-on-event"></a>
808 `OnEvent` implements the "state machine" depending on the current desired action. By default, this is `TlsActionNone`.
810 Once `TlsStream->Handshake()` is called, this initializes the current action to
811 `TlsActionHandshake` and performs `SSL_do_handshake()`. This function returns > 0
812 when successful, anything below needs to be dealt separately.
814 If the handshake was successful, the registered condition variable `m_CV` gets signalled
815 and the thread waiting for the handshake in `TlsStream->Handshake()` wakes up and continues
816 within the `ApiListener::NewClientHandler()` function.
818 Once the handshake is completed, current action is changed to either `TlsActionRead` or `TlsActionWrite`.
819 This happens in the beginning of the state machine when there is no action selected yet.
821 * **Read**: Received events indicate POLLIN (or POLLERR/POLLHUP as error, but normally mean "read").
822 * **Write**: The send buffer of the TLS stream is greater 0 bytes, and the received events allow POLLOUT on the event socket.
823 * Nothing matched: Change the event sockets to POLLIN ("read"), and return, waiting for the next event.
825 This also depends on the returned error codes of the SSL interface functions. Whenever `SSL_WANT_READ` occurs,
826 the event polling needs be changed to use `POLLIN`, vice versa for `SSL_WANT_WRITE` and `POLLOUT`.
828 In the scenario where the master actively connects to the clients, the client will wait for data and
829 change the event sockets to `Read` once there's something coming on the sockets.
832 ---------------|---------------
833 Read | Calls `SSL_read()` with a fixed buffer size of 64 KB. If rc > 0, the receive buffer of the TLS stream is filled and success indicated. This endless loop continues until a) `SSL_pending()` says no more data from remote b) Maximum bytes are read. If `success` is true, the condition variable notifies the thread in `WaitForData` to wake up.
834 Write | The send buffer of the TLS stream `Peek()`s the first 64KB and calls `SSL_write()` to send them over the socket. The returned value is the number of bytes written, this is adjusted within the send buffer in the `Read()` call (it also optimizes the memory usage).
835 Handshake | Calls `SSL_do_handshake()` and if successful, the condition variable wakes up the thread waiting for it in `Handshake()`.
837 ##### TLS Error Handling
839 TLS error code | Description
840 -------------------------|-------------------------
841 `SSL_WANT_READ` | The next event should read again, change events to `POLLIN`.
842 `SSL_ERROR_WANT_WRITE` | The next event should write, change events to `POLLOUT`.
843 `SSL_ERROR_ZERO_RETURN` | Nothing was returned, close the TLS stream and immediately return.
844 default | Extract the error code and log a fancy error for the user. Close the connection.
846 From this [question](https://stackoverflow.com/questions/3952104/how-to-handle-openssl-ssl-error-want-read-want-write-on-non-blocking-sockets):
849 With non-blocking sockets, SSL_WANT_READ means "wait for the socket to be readable, then call this function again."; conversely, SSL_WANT_WRITE means "wait for the socket to be writeable, then call this function again.". You can get either SSL_WANT_WRITE or SSL_WANT_READ from both an SSL_read() or SSL_write() call.
853 ##### Successful TLS Actions
855 * Initialize the next TLS action to `none`. This re-evaluates the conditions upon next event call.
856 * If the stream still contains data, adjust the socket events.
857 * If the send buffer contains data, change events to `POLLIN|POLLOUT`.
858 * Otherwise `POLLIN` to wait for data.
859 * Process data when the receive buffer has them available and we are actively handling events.
860 * If the TLS stream is supposed to shutdown, close everything including the TLS connection.
862 #### Data Processing <a id="technical-concepts-tls-network-io-async-socket-io-data-processing"></a>
864 Once a stream has data available, it calls `SignalDataAvailable()`. This holds a condition
865 variable which wakes up another thread in a handled which was previously registered, e.g.
866 for JsonRpcConnection, HttpServerConnection or HttpClientConnection objects.
868 All of them read data from the stream and process the messages. At this point the string is available as JSON already and later decoded (e.g. Icinga data structures, as Dictionary).
872 ### General Design Patterns <a id="technical-concepts-tls-network-io-design-patterns"></a>
874 Taken from https://www.ibm.com/developerworks/aix/library/au-libev/index.html
877 One of the biggest problems facing many server deployments, particularly web server deployments, is the ability to handle a large number of connections. Whether you are building cloud-based services to handle network traffic, distributing your application over IBM Amazon EC instances, or providing a high-performance component for your web site, you need to be able to handle a large number of simultaneous connections.
879 A good example is the recent move to more dynamic web applications, especially those using AJAX techniques. If you are deploying a system that allows many thousands of clients to update information directly within a web page, such as a system providing live monitoring of an event or issue, then the speed at which you can effectively serve the information is vital. In a grid or cloud situation, you might have permanent open connections from thousands of clients simultaneously, and you need to be able to serve the requests and responses to each client.
881 Before looking at how libevent and libev are able to handle multiple network connections, let's take a brief look at some of the traditional solutions for handling this type of connectivity.
883 ### Handling multiple clients
885 There are a number of different traditional methods that handle multiple connections, but usually they result in an issue handling large quantities of connections, either because they use too much memory, too much CPU, or they reach an operating system limit of some kind.
887 The main solutions used are:
889 * Round-robin: The early systems use a simple solution of round-robin selection, simply iterating over a list of open network connections and determining whether there is any data to read. This is both slow (especially as the number of connections increases) and inefficient (since other connections may be sending requests and expecting responses while you are servicing the current one). The other connections have to wait while you iterate through each one. If you have 100 connections and only one has data, you still have to work through the other 99 to get to the one that needs servicing.
890 * poll, epoll, and variations: This uses a modification of the round-robin approach, using a structure to hold an array of each of the connections to be monitored, with a callback mechanism so that when data is identified on a network socket, the handling function is called. The problem with poll is that the size of the structure can be quite large, and modifying the structure as you add new network connections to the list can increase the load and affect performance.
891 * select: The select() function call uses a static structure, which had previously been hard-coded to a relatively small number (1024 connections), which makes it impractical for very large deployments.
892 There are other implementations on individual platforms (such as /dev/poll on Solaris, or kqueue on FreeBSD/NetBSD) that may perform better on their chosen OS, but they are not portable and don't necessarily resolve the upper level problems of handling requests.
894 All of the above solutions use a simple loop to wait and handle requests, before dispatching the request to a separate function to handle the actual network interaction. The key is that the loop and network sockets need a lot of management code to ensure that you are listening, updating, and controlling the different connections and interfaces.
896 An alternative method of handling many different connections is to make use of the multi-threading support in most modern kernels to listen and handle connections, opening a new thread for each connection. This shifts the responsibility back to the operating system directly but implies a relatively large overhead in terms of RAM and CPU, as each thread will need it's own execution space. And if each thread (ergo network connection) is busy, then the context switching to each thread can be significant. Finally, many kernels are not designed to handle such a large number of active threads.
900 ### Alternative Implementations and Libraries <a id="technical-concepts-tls-network-io-async-socket-io-alternatives"></a>
902 While analysing Icinga 2's socket IO event handling, the libraries and implementations
903 below have been collected too. [This thread](https://www.reddit.com/r/cpp/comments/5xxv61/a_modern_c_network_library_for_developing_high/)
904 also sheds more light in modern programming techniques.
906 Our main "problem" with Icinga 2 are modern compilers supporting the full C++11 feature set.
907 Recent analysis have proven that gcc on CentOS 6 or SLES11 are too old to use modern
908 programming techniques or anything which implemens C++14 at least.
910 Given the below projects, we are also not fans of wrapping C interfaces into
911 C++ code in case you want to look into possible patches.
913 One key thing for external code is [license compatibility](http://gplv3.fsf.org/wiki/index.php/Compatible_licenses#GPLv2-compatible_licenses) with GPLv2.
914 Modified BSD and Boost can be pulled into the `third-party/` directory, best header only and compiled
915 into the Icinga 2 binary.
919 * libevent: http://www.wangafu.net/~nickm/libevent-book/TOC.html
920 * libev: https://www.ibm.com/developerworks/aix/library/au-libev/index.html
921 * libuv: http://libuv.org
925 * Asio (standalone header only or as Boost library): http://think-async.com (the Boost Software license is compatible with GPLv2)
926 * Poco project: https://github.com/pocoproject/poco
927 * cpp-netlib: https://github.com/cpp-netlib/cpp-netlib
928 * evpp: https://github.com/Qihoo360/evpp