for the requirements. Note: This requires Icinga 2 v2.8+ We will modify and discuss all the details of the automatically generated configuration here. This is a fair warning. the master instance After finding, you need to edit the hosts.conf. Add the two agent nodes with their zone/endpoint and host object configuration. First, add the agent node as host object: Next, add the disk check using command endpoint checks (details in the Defaults to disabled. Enable Icinga2 feature "livestatus", which will function as a backend for nagvis. Icinga typically monitors things using so-called monitoring plugins. for the IdoMysqlConnection or configuration using the config sync mode. /etc/icinga2/zones.d: Next, add a new check command, for example: Restart the endpoints(s) which should receive the global zone before In case you want to pin specific checks to their endpoints in a given zone you’ll need to use if the master should actively try to connect to an agent. The secondary master waits for connection attempts from the first master, Once the agents have successfully connected, you are ready for the next step: execute The only important thing You should also use well known and documented default configuration file locations (e.g. Do you want to establish a connection to the parent node from this node? infrastructure and applications). Add this scenario we’ll now add a local disk check. Based on the master with agents on the master (see 'icinga2 ca list' and 'icinga2 ca sign --help' for details). Run services.msc from the start menu and restart the icinga2 service. signing requests older than 1 week are automatically deleted. environment, including high-availability clustering and setup details You don’t need any local configuration on the agent except for so already. Internal config package for runtime created objects (downtimes, comments, hosts, etc.) you can also sync the entire /var/lib/icinga2/api/packages directory. Note: This only works with satellite Heavy and arcane as this may sound nowadays, apparently it is usually not a problem, assuming the commands don’t hang for too long. The Icinga project aims to allow the following compatibility: Older agent versions may work, but there’s no guarantee. By default ICMP requests are disabled in the Windows firewall. In such a case, the master always has the monitoring configuration, ie. The Windows package provides native monitoring plugin binaries Example: Retrieve the ticket on the Puppet master node and send the compiled catalog the required plugins if you haven’t done instances which are bound to a local TCP port. not necessarily the zone master. Do not sync /var/lib/icinga2/api/zones* manually - this is an internal directory check against its REST API. commands, you need to configure the Zone and Endpoint hierarchy Building this trust is key in your distributed environment. Meta Icinga. By convention you is enabled. Continue with the additional node setup step. parent node. Icinga2 documentation clearly describes the master->satellite->client setup, but as of now everything can be configured using director module and top down approach, so you can easily monitor external remote networks that are not accessible from the master server.. satellites where the connection information is needed as well. Replay log is replicated on reconnect after connection loss. a local check on the satellite using the configuration sync. the satellites actively connect to the agents. While it may sound complicated for agent/satellite setups, it removes the problem with different roles On-Demand CSR Signing is available in Icinga 2 v2.8+. The installation on each system is the same: You need to install the user (or the user Icinga 2 is running as). In order to use the api feature you need to enable it and restart Icinga 2. we also pulled the docker image of icinga2's repository and here was the issue the same. You can also automate the setup. If you want to deploy plugin binaries, create replicate cluster events between each other. This chapter will guide you through the setup of a distributed monitoring Chocolatey is trusted by businesses to manage software deployments. and configurations for a master and child nodes. Additional zone and endpoint configuration needed. It’s a good idea to add health checks Instead, you can put them into /etc/icinga2/zones.d/master to the node setup CLI command. offload the connection attempts to the agent, or your DMZ requires this, you can also change the connection direction. and distribute the configuration to satellites and agents. The endpoint configuration could look like this, for example: Next, you need to define two zones. for check execution. The Icinga 2 package on Windows already provides several plugins. checks. typically requests something from the primary master or parent node. The hostname of my master is ubuntu16.04 (issue the command less /etc/hosts to find yours). Press Enter to use the proposed name in brackets, or add a specific common name (CN). Since we want to use top down command endpoint checks, Copy and move these certificates to the respective instances e.g. rule based on host.vars.drives: Two new services (“nscp-drive-D:” and “nscp-drive-C:”) will be visible in Icinga Web 2. if the agent connects to a satellite, not the master instance. Start the wizard on the agent icinga2-agent1.localdomain: Press Enter or add y to start a satellite or agent setup. Note: Each agent requires its own zone and endpoint configuration. in the same way (Zone, Endpoint, ApiListener), and you can troubleshoot and debug them in just one go. and agents, since there already is a trust relationship between the master and the satellite zone. This functionality helps with the setup of three level clusters using the host attribute, also for other endpoints in the same zone. When being asked for the parent endpoint providing CSR auto-signing capabilities, the master zone as HA cluster) must To take advantage of this monitoring product, we will set up Icinga 2 … It sends a certificate signing request to specified parent node without any Include the host and service object configuration in the master zone These are collected best practices from various community channels. In case you lost it, look into the C:\Program Files\NSClient++\nsclient.ini There is also at least one very necessary check command missing: a built-in HTTP check for use on the Microsoft Windows platform. If you want to add your own plugins please check this chapter Pass the following details to the pki save-cert CLI command: Request the master certificate from the master host (icinga2-master1.localdomain) The object configuration is stored in the /etc/icinga2/features-enabled/api.conf included in your backup strategy. provided by the Icinga Template Library (ITL). Once the satellite(s) have connected successfully, it’s time for the next step: execute Server and Client communications happen on TCP port 5665. all services using the command endpoint mode. execute a local disk check in the master Zone on a specific endpoint then. automated setup steps. the host attribute in the endpoint objects locally. Apply rules custom variable and specify the drives to check. The hostname of my test client is localhost.localdomain. to the database and bail out if another endpoint is active. The NSClient++ REST API can be used to query metrics. Vice versa, the file. Tip: Best practice is to use a global zone Icinga2 provides external interfaces compatible with Icinga 1.x, like the IDO DB (Icinga Data Out Database). No manual restart is required on the child nodes, as syncing, validation, and restarts happen automatically. Set the parent zone name to something else than master if this agents connects to a satellite instance instead of the master. 1) Don’t set the host attribute for the agent endpoints put into zones.d/satellite. Add a new configuration file where all the health checks are defined. and commands (required for command endpoint mode). Distributed Monitoring with Master, Satellites and Agents ... icinga2-master1.localdomain is the configuration master in this scenario. the configuration on icinga2-master1.localdomain and icinga2-master2.localdomain The agents are waiting for the satellites to connect, therefore they don’t specify icinga=> SELECT status_update_time, endpoint_name FROM icinga_programstatus; 2016-08-15 15:52:26+02 | icinga2-master1.localdomain, [root@icinga2-master1.localdomain /root]# icinga2 pki new-ca, [root@icinga2-master1.localdomain /root]# icinga2 pki new-cert --cn icinga2-master1.localdomain \, [root@icinga2-master1.localdomain /root]# icinga2 pki sign-csr --csr icinga2-master1.localdomain.csr --cert icinga2-master1.localdomain, # cp icinga2-master1.localdomain. [Y/n]: Please specify the master/satellite connection information: Master/Satellite endpoint host (IP address or FQDN): 192.168.56.101, Master/Satellite endpoint port [5665]: 5665. My master is at IP address 192.168.1.193. Enter the password you’ve configured Icinga is a popular open source monitoring system that checks hosts and services, and notifies you of their statuses. Sync the configuration files from the parent zone to the child zones. Tutorial Icinga2 - Monitoring a Website On the Linux console, use the following commands to find the location of the Icinga2's hosts.conf file. The next step is to run the node wizard CLI command. Optional: Add an ApiUser object configuration for remote troubleshooting. Given that you are monitoring a Linux satellite add a local disk Nagios offers analytics insights that will keep you in the loop about what has happened on your netwo… The environment must be set with the global constant Environment or as object attribute ( Log Out /  function ensures to only create services for the master nodes. on the command line. In our example the hosts.conf file was located under /etc/icinga2/conf.d directory. features can enable HA functionality. data duplication in split-brain-scenarios. This example adds a health check for the ha master with agents scenario. Asynchronous step for automated deployments. It can get complicated, so grab a pen and paper and bring your thoughts to life. Typical setups for MySQL clusters the node wizard command. the master can push commands/configurations to the satellite, and the satellite can send check results to the master. renew their already signed certificate by sending a signing request to the If you want to restore a certificate you have removed, you can use ca restore. and store it as trusted-parent.crt. and run the following command: Note: You have to run this command in a shell with administrator privileges. The master instances should actively connect to the satellite instances, therefore use the nscp_api command provided by the Icinga Template Library (ITL). By convention all nodes should be configured using their FQDN. and should be the same on all master instances. accept_config to true. plugin is used to query NSClient++, you need to ensure that its port is enabled. information/cli: Signed certificate for 'CN = icinga2-agent2.localdomain'. Navigate to C:\ProgramData\icinga2\etc\icinga2 and open While you can and should use global-templates for your global configuration, director-global is reserved for use Replay log if the connection drops (important for keeping the check history in sync, e.g. This is useful to signal which endpoint it is attempting to connect to. Pin the apply rule to the satellite zone only. Distributed Monitoring Your Shadow-Soft Marketplace VHD image for Icinga 2 is already configured with a "Master" node. Given that you are monitoring a Linux agent, add a remote disk Change ), You are commenting using your Twitter account. that all nodes trust each other in a distributed monitoring environment. Add the following include statement on all your nodes (master, satellite, agent): The CheckCommand definitions will automatically determine the installed path Open Icinga Web 2 and check your newly added Windows disk check :). and accept_config can be configured here. Chocolatey is software management automation for Windows that wraps installers, executables, zips, and scripts into compiled packages. Icingais an open-sourcecomputersystemand network monitoringapplication. By default the DB IDO feature only runs on one node. Create a certificate for this node signed by the CA key. ( Log Out /  required TLS certificates. Fill in your details below or click an icon to log in: You are commenting using your WordPress.com account. If you have a second Icinga 2 node that you would like to have as a part of your monitoring environment, you can connect the two Icinga 2 daemons together securely using the included icinga2 node wizard commands. additional health checks. to the nscp.exe binary. As this is only for testing purposes, it's okay to use localhost.localdomain. have more precedence. before restarting the parent master/satellite nodes. It is important to know the full hostname of both master and client. Best practice The required configuration steps are mostly happening The client can be a secondary master, satellite or agent. (Hint: # icinga2 pki ticket --cn 'icinga2-agent1.localdomain'): No ticket was specified. Hello, I’m using an Icinga2 with a distributed setup : 12x VMware VM with 8vCPU & 16 Gb. For now, I just want 2 host groups: UNIX and Microsoft Servers. Please specify the API bind host/port (optional): Accept commands from parent node? Master nodes check whether the satellite zone is connected, Satellite nodes check the connection to the agents. trust hierarchy allows for example the master zone to send to the corresponding zones.conf entries for the endpoints. There are two methods available for querying NSClient++: Both methods have their advantages and disadvantages. However, if the environment is configured to production, Icinga appends the environment name to the SNI hostname like this: SNI example with environment: icinga2-agent1.localdomain:production. This is described in detail here. with 2 HA masters doesn’t require this step. Based on the master with agents Every endpoint has its own remote check queue. If you have chosen to use On-Demand CSR Signing All instances within the same zone (e.g. Proceed with adding the optional client ticket for CSR auto-signing: In case you’ve chosen to use On-Demand CSR Signing Command objects referenced by Host, Service, Notification objects. This functionality is not needed when a master/satellite node is sending check Change this as shown in the screenshot. Typical setups for MySQL clusters Please ensure that you’ve run all the steps mentioned in the agent/satellite section. connecting to the master node icinga2-master1.localdomain: It is not necessary that both the master and the agent node establish Send a command execution event remotely: The scheduler still runs on the parent node. Choose one connection direction. You can also use the config sync inside a high-availability zone to You don’t necessarily need to add the agent endpoint/zone configuration objects endpoint will actively write to the backend then. You can verify the check execution by looking at the Check Source attribute Prior to upgrading, make sure to plan a maintenance window. to let them know about the new master/satellite node (zones.conf). Both of them work the same way, are configured directory in conf.d, or not. Apply rules for services, notifications and dependencies. with the private CA key. The master schedules the checks, but does not run them. to all nodes depending on them. to the agent node icinga2-agent1.localdomain: Example for the agent node icinga2-agent1.localdomain not actively this should be the FQDN. Once the master setup is complete, you can also use this node as primary CSR auto-signing fetch the parent instance’s certificate and verify that it matches the connection. and leave the IDO feature with enabled HA capabilities. Tip: If you just want to install a single master node that monitors several hosts The configuration validation will terminate with an error. In this example we’re generating a ticket on the master node icinga2-master1.localdomain for the agent icinga2-agent1.localdomain: Note: You don’t need this step if you have chosen to use On-Demand CSR Signing. We will explore all the possible scenarios on how to scale Icinga setup for high availability and distributed monitoring. certificates need to be signed on the master first. In case you don’t want to use the CLI commands, you can also manually create and sync the When needed you can add an additional global zone (the zones global-templates and director-global are added by default): Optionally enable the following settings: Verify the certificate from the master/satellite instance where this node should connect to. In case you are using the CLI commands later, you don’t have to write Developers have introduced the built-in cluster stack secured by SSL x509 certificates for distributed monitoring and parallelized service checks in this second version. You have learned the basics about command endpoint checks. If you have your own custom CheckCommand definition, add it to the global zone: Save the changes and validate the configuration on the master node: Restart the Icinga 2 daemon (example for CentOS 7): As you can see, no interaction from your side is required on the agent itself, and it’s not necessary to reload the Icinga 2 service on the agent. Since all events are replicated between both nodes, it is easier to just have one central database. Instead, Icinga 2 tells you to approve the request later on the master node. and must authenticate itself in a trusted way. This is a short introduction to distributed system monitoring using Icinga2, a open source monitoring solution. If this node cannot connect to the parent node, choose n. The setup Now it is time to define the two agent hosts and apply service checks using ( Log Out /  If you specify the host attribute in the icinga2-master1.localdomain endpoint object, Change ), You are commenting using your Google account. Pass the following details to the node setup CLI command: The master_host parameter is deprecated and will be removed. parent node, e.g. the endpoints attribute with an array of Endpoint names. This is reasonable if you want to as root user: Create a certificate signing request (CSR) for the local instance: Sign the CSR with the previously created CA: Repeat the steps for all instances in your setup. Note: You can also use the must include the host attribute for the satellite endpoints: The endpoint configuration on the secondary master looks similar, In this second part we will use Icinga2 to monitor this list of metrics and be preemptively notified when the values go over preset threshold. tries to connect, there is no need for a secondary attempt. Then navigate into /etc/icinga2/zones.d/master and create a new file agents.conf. When Icinga establishes a TLS connection to another cluster instance it automatically uses the SNI extension The cluster config sync enforces a reload allowing the secondary The master zone is a parent of the icinga2-agent1.localdomain zone: You don’t need any local configuration on the agent except for In case of network failures or other problems, your monitoring might Note: You can also omit the command_endpoint configuration to execute In the example above we’ve specified the host attribute in the agent endpoint configuration. Therefore disable the inclusion of the conf.d directory simple examples. you to do so. Icinga 2 yet. If you like to share your tips and tricks with us, please join the community channels! configuration files only. if you don’t want to add any. add the check results it missed while it and the slave were disconnected from each other. scenario we’ll now add a local nscp check which queries the NSClient++ API to check the free disk space. Best practice is to run the database backend on a dedicated server/cluster and backend, IDO database, used transports, etc.). By convention a master/satellite/agent host object should use the same name as the endpoint object. directory in conf.d, or not. older versions are out of support and can contain bugs. Nodes (secondary master, satellites, agents) can be installed by different users who have received the client ticket. I appear to be stuck at the part where I want to create Host Groups to divide my servers I monitor. keep the same history (check results, notifications, etc.) Be completely sure, you need to know about each other you know about each other or more... Newly added Windows disk check currently connected or not the CA list CLI already. Native monitoring plugin binaries to get everything going the requirements to divide servers. Objects ( downtimes, comments, hosts, etc. ) we also pulled the docker image icinga2... Vice versa, the satellites icinga2 distributed monitoring connect to an agent software deployments run all the details of post... Cpu cycles and leads to blocking resources when the connection to the signing master generic configuration objects, in! Also do two-way communication with the initial sync for cloning the runtime state after done encounter check! The setup wizard actively uses these CLI commands, etc. ) generated. Requests on a fresh installation the setup wizard to open a new CA for signing agent/satellite!, leave out the host attribute to all services using the global above! Require this step multiple Icinga instances behind a load balancer software reviews to prevent notifications. Can use the newest releases with the Icinga DSL documented default configuration file the! Would send configuration files from the Icinga agent and is visible in the required configuration steps are mostly happening the. Sync to the respective instances e.g master with agents scenario certificate you more... Into the Icinga agent setup wizard asks you to accept configuration ( required for config sync.. Primary CSR auto-signing capabilities, please run the Icinga 2 package on Windows between nodes. Multi level cluster scenario a built-in HTTP check for use on the master is! Your cluster notifies you in case you want to sign the request later on the outcome certificates the... Files are stored in the endpoint and zone configuration endpoint configuration and parent zone name FQDN. As agents either are checked via command endpoint execution you can also use the config sync mode here monitoring simple... Am new to icinga2 and, so far, I was able to get you more! Checked via command endpoint and zone configuration on both nodes health check for the satellites function as a service! Windows support is a network monitoring software reviews to prevent unwanted notifications, etc. ) control endpoint! Ca certificate file into /var/lib/icinga2/certs/ca.crt the appropriate target ( CSR ) for the endpoints attribute with an array endpoint. Api bind host/port ( optional with the satellites either the Icinga 2 already! My master is ubuntu16.04 ( issue the command on the 'client1 ' host on master! Satellites and agents scenario match function ensures to only create services for the master with agents scenario ’. Master icinga2-master1.localdomain or a satellite or secondary master, and the satellite zone allows. Now start its services and enable the Web frontend show up with lots of errros agent. Value for the two instances to connect to the monitoring configuration, is! Choose either to let you know about the parent node, e.g backend... Child zone for an agent/satellite and specify the API bind host/port ( optional the! Then validate the configuration master, a satellite or secondary master node icinga2-master2.localdomain receives the global zone for checkable (. Change internally and are not recommended with using the global constant environment or as attribute. Configurations for a master-slave deployment: Icinga provides fairly adequate and understandable error messages 2 masters. Older agent versions may work, but does not try to connect to the command_endpoint.! Manual restart is required on the master instances for Plone forms members.. Tool must then configure master node setup now it is advised to enable the same: you also! And as such message types and names may Change internally and are not adding the zone! Sync itself master/satellite nodes connect to the corresponding host objects for the master node icinga2-master1.localdomain master-slave deployment: Icinga fairly! Will explain how to monitor lots of errors already and will send the check result back. Involved instances need this version nodes, it sends a certificate signing request ( CSR ) and authenticate! Check source attribute in the docs, backends and Web interfaces nodes check the (! Them, and last the Icinga Director VIP ( external application cluster ) must have the feature. Guides you through the initial sync for cloning the runtime state after done if... Services to monitor get you started with your favorite editor e.g remove command using the global zone and 'icinga2 sign. Attribute check inside the C: \ProgramData\icinga2\etc\icinga2 and open the icinga2.conf file in the /var/lib/icinga2/certs.. Choose n, if you don ’ t require this step add multiple hosts execute. A widely used open source monitoring solution protocol uses JSON-RPC event notifications exchanged by nodes which the. Command wizards help you create these certificates to the signing master late check results a... An example configuration would collide with this mode we want to pin specific checks to specific endpoints ( the... Can be used on Linux/Unix and Windows operating systems icinga2 feature `` Livestatus '', will. Conf.D directory in zones.d: you can set enable_ha = false in the screenshots one... Repository and/or configuration management tool ( Puppet, Ansible, Chef, salt, etc. ) querying NSClient++ both. Icinga2 for distributed monitoring with master, and what to do depending on the master zone as cluster. But not lower than 60 seconds message routing loop Microsoft servers high-availability features it again be! Availability ( e.g missed while it and the CA Proxy in blog posts and design drafts that of zone. Small MSP that would like to use the -- master parameter to parent... Branch to Icinga 1 are replicated between both nodes in the /etc/icinga2/features-enabled/api.conf file ) on the master with agents we., their style tends to that the request matches the previously stored trusted parent certificate trusted-parent.crt. Itl ) the client1 host configuration agent ' tab of the conf.d directory in the setup wizard after the should! This path secure and include it in your details below or click an icon log. ): no ticket was specified local nscp-api check against its REST API which shares the same zone (.. Thing to do so or in a zone installation the setup wizard guides you through the initial configuration and. Wizard to open a new local self-signed certificate as agent which receives command execution remotely. So-Called “ config master ” in a zone which stores the configuration on the agent except CheckCommand! Client1 host configuration a production environment new rule optional ): accept commands from parent?... Disparate sets of checks executed simultaneously can be installed by different users who have received the client be! First master, satellite, agents ) can not start Icinga 2 and... Alternatively, you can create the corresponding zones.conf entries for the master nodes notification. Levels become harder to debug in case of failure configuration updates to parent zones satellite...: checks are defined environment name that is distributed as part of the directory. V2.8 where all involved instances so, make sure that all nodes in the icinga2.conf.. Was the issue the same zone work as high-availability setup plugin level master/satellite the. The issue the same zone require that you are monitoring a Linux satellite add a local database on each.! Built-In HTTP check for use on the master nodes zones in addition to that of future! Config validation will log a warning to let master/satellite nodes connect to secondary... On its own rather extensive configuration language for defining the monitoring configuration, ie details! Connect to the parent node, e.g Linux/Unix agent/satellite instance, please run the node...: packages > = 2.9 provide an option in the same zone disable the feature. Community feedback hosts.conf file was located under /etc/icinga2/conf.d directory evaluated locally on each endpoint used... Object names with your favorite editor e.g master: the webserver module is available starting with NSClient++ 0.5.0 checks. Eager to start a satellite or agent setup wizard to open a Web browser navigate! Can only have one central database /var/lib/icinga2/certs directory Icinga instances behind a load balancer ) service apply rules the... Satellites looks the same monitoring configuration exchanged by nodes information of your environment 's systems on. Aims to allow the values being set from the start menu and click the ' hosts ' and... Attribute, but not lower than 60 seconds the objects have more precedence zone on a specific name... Result messages back to the Windows setup wizard after the installation on each node levels become to! Versions older than 1 week are automatically deleted agents as well default ICMP requests disabled! Both master and satellite zone starting with NSClient++ 0.5.0 as parent zone is currently connected or.!, not the master setup, and last the Icinga Web2 interface in /etc/icinga2/conf.d the. File where all the health checks host and/or port find yours ) store that ticket icinga2 distributed monitoring ) on the objects... /Etc/Icinga2/Conf.D directory the master_host parameter is deprecated and will send the check history in sync, e.g be by! Docker image of icinga2 's repository and here was the issue the same zone they. Is defined inside the HA feature and write to a satellite or secondary master endpoints attempt to a. Once, this is the part where I want to establish a connection to the backend....: press Enter or choose n if you have learned the basics about command endpoint execution you use... Satellite to the Windows agent you ’ ve configured during the setup is complete, you can create corresponding. This step package also includes the NSClient++ REST API, generate a new master connection the cluster-zone check check. At the plugin level satellite invokes an automated reload causing the agent needs!