Installing and Configuring Agents

This section describes the installation of Foglight for Cassandra. Follow the steps in the order specified below:

• Configuring Cassandra
  
  • Database User Setup
  • JMX Setup
  • Configuring an SSL Connection
• Creating and Configuring Agents
• Configuring Agent Properties
• Removing Monitored Databases
• Roles

Configuring Cassandra

To enable full monitoring of Cassandra, the agent requires a user account with sufficient privileges to execute system queries. For JMX monitoring, remote JMX connections must be enabled on each node.

Database User Setup

Using a superuser role or a role with permission to create users, connect to a node in the cluster using cqlsh, and create a new user for the Foglight Agent as follows:

Cassandra version 2.2 or later

For Cassandra version 2.2 or later, or DSE version 5.0 or later, create a user as follows:

CREATE ROLE IF NOT EXISTS <username> WITH PASSWORD = '<your-password>' AND LOGIN = true AND SUPERUSER = false;

There are then two ways to set permissions. The simpler way is to use the following single permission:

GRANT SELECT ON ALL KEYSPACES to <username>;

It is also possible to use more specific permissions:

GRANT SELECT ON TABLE system.local to <username>;
GRANT SELECT ON TABLE system.peers to <username>;
GRANT SELECT ON TABLE system_traces.events to <username>;
GRANT SELECT ON TABLE system_traces.sessions to <username>;
GRANT SELECT ON TABLE system_auth.roles to <username>;

Using specific permissions, there is one version-dependent permission. For Cassandra version 3.0 or later:

GRANT SELECT ON TABLE system_schema.tables to <username>;

For Cassandra versions 2.2 through 2.x (but earlier than 3.0):

GRANT SELECT ON TABLE system.schema_columnfamilies to <username>;

If you are using DataStax Enterprise (DSE) on version 2.2 or later, use the following permissions:

GRANT SELECT ON TABLE dse_perf.node_slow_log to <username>;

Cassandra versions earlier than 2.2

For Cassandra versions earlier than 2.2 or DSE version 4.8 and earlier:

CREATE USER IF NOT EXISTS <username> WITH PASSWORD '<your-password>' NOSUPERUSER;
GRANT SELECT ON ALL KEYSPACES TO <username>;

JMX Setup

Enable Remote JMX Connections

Enable remote JMX authentication as per the instructions at either of the following links:

• DataStax Cassandra JMX Configuration
• Apache Cassandra JMX Configuration

JMX Remote Files

Only a single user needs to be added to the jmxremote.password and jmxremote.access files for Foglight monitoring and administration. The following is the full (template) content of a sample jmxremote.password file:

<your-JMX-username> <your-JMX-password>

The following is the full (template) content of a sample jmxremote.access file:

<your-JMX-username> readwrite

You can select between two JMX access permission options:

• readonly – For monitoring only.
• readwrite – For monitoring as well as administration, which is currently the ability to set the per-node trace probability.

Configuring an SSL Connection

The following instructions outline common steps to configure a TLS/SSL connection from the Cassandra Agent client. A comprehensive discussion of TLS/SSL keys, certificates, and certificate authorities (CAs) is beyond the scope of this document. These instructions assume familiarity with TLS/SSL concepts and related tools. Ensure that the required client and CA certificates are available before proceeding.

To use SSL, your Cassandra cluster must support SSL and be configured to allow secure client connections. Multiple configuration options are available for client authentication. Refer to the official Cassandra documentation to confirm that your server configuration supports the intended authentication method.

As a database client, the Foglight Agent requires access to the following:

• A private key
• A signed client certificate
• The signing CA’s certificate

The client key and certificate must be imported into a Java KeyStore (JKS), and the CA certificate must be imported into a separate truststore.

To create a JKS keystore for use with Foglight, you can use tools such as openssl and keytool. When generating the keystore, specify the appropriate filenames for the key and certificate, the alias name, and the keystore password.

openssl pkcs12 \
    -export \
    -in ${CERT_NAME}.crt \
    -inkey ${CERT_NAME}.key \
    -name $CERT_NAME \
    -out temp-keystore.p12 \
    -passout pass:${KEYPASS}

keytool -importkeystore \
    -srckeystore temp-keystore.p12 \
    -srcstoretype PKCS12 \
    -srcstorepass $KEYPASS \
    -destkeystore keystore \
    -deststoretype JKS \
    -deststorepass $KEYPASS

Regardless of how the keystore is constructed, it must list the client certificate as a ‘PrivateKeyEntry’, indicating that it also contains the private key, not just the signed certificate.

Separately, import the CA certificate into a truststore:

keytool -importcert \
    -keystore truststore \
    -alias $CA_NAME \
    -file ${CA_NAME}.crt \
    -keypass $TRUSTPASS \
    -storepass $TRUSTPASS \
    -storetype JKS \
    -noprompt

Next, edit the baseline.jvmargs.config file in the <FglAM-install-root>/state/default/config directory and add the following parameters with file paths and passwords appropriate for your system.

vmparameter.0 = "-Djavax.net.ssl.keyStore=/path/to/keystore";
vmparameter.1 = "-Djavax.net.ssl.keyStorePassword=changeit";
vmparameter.2 = "-Djavax.net.ssl.trustStore=/path/to/truststore";
vmparameter.3 = "-Djavax.net.ssl.trustStorePassword=changeit";

Escape any quotes with a backslash (‘\’). On an Agent Manager installed on Windows, use forward slashes in the file paths, like so:

vmparameter.0 = "-Djavax.net.ssl.keyStore=\\"C:/path/to/keystore\\"";

Then, restart the FglAM and continue with the agent configuration, setting the Use TLS/SSL? option in the Agent Properties to true for the driver and/or JMX connections, depending on the Cassandra server configuration.

Creating and Configuring Agents

Agents can be created in one of two ways:

• Using the Agent Installer Wizard
• Using the Agent Status Dashboard

The Agent Installer Wizard simplifies the agent creation and configuration process and can be accessed from the Databases dashboard. For advanced configuration or modification of agent properties post-creation, use the Agent Status dashboard.

Using the Agent Installer Wizard

Foglight for Cassandra offers a graphical, intuitive interface for creating and configuring agents. This method serves as an alternative to Foglight’s default approach, which involves creating agents and editing their properties through the Agent Status dashboard.

With Foglight for Cassandra, you can use a wizard that provides a unified entry point for adding database instances and configuring them for monitoring.

To run the instance installation wizard:

1. On the Welcome page, click Monitor Databases or click Databases in the navigation pane. On a fresh installation, no instances are listed for Cassandra in the Databases dashboard.
2. On the Databases dashboard, click Monitor and select Cassandra. The Agent Installer wizard opens.
3. The first card - Agent Deployment – has two fields:
   1. Agent Name – Provide a name for the agent that will be created. This is not canonical and should be representative of the database instance that this agent will monitor.
   2. Agent Manager - Choose the agent manager on which the agent should run. Considerations for this may include physical or virtual locality to the monitored instance, allocated resources, or grouping with other agents of the same type or monitored environment. If the agent package has not been deployed to this Agent Manager yet, it will be installed when the first agent of this type is created.
4. The second card – Agent Properties – requires a basic set of parameters for connecting to and monitoring the database instance. A full explanation of these properties is available in the Configuring Agent Properties section of this document.
5. The third card – Agent Summary – displays a review of the configuration that will be created and an option allowing the agent to be activated after creation. If the configuration looks good, click Finish to start the process.
6. When the process completes, a results screen will appear showing the results of agent creation. If the agent was not created, follow the instructions on the results screen. If successful, the database instance should appear in the Databases table within a few minutes.

The database instance gets created and appears in the list of all instances.

Using the Agent Status Dashboard

The Agent Status page can be used to create new agents and configure and manage existing agents. To access the page from the navigation pane, click Administration > Agents > Agent Status.

To create a new agent instance:

Deploy the Cassandra agent package to the FglAM before creating the agent if it has not been deployed yet. Click Deploy Agent Package on the Agent Status or Agent Managers page to perform this.

1. Navigate to Administration > Agents > Agent Status.
2. Click Create Agent.
3. Select the hosts to which you want to deploy agent packages.
   

   Considerations for this may include physical or virtual locality to the monitored instance, allocated resources, or grouping with other agents of the same type or monitored environment.

4. Click Next.
5. Select the Cassandra type. Then, select the Specify Name radio button and enter agent instance name. Click Next.
6. Click Finish.
7. Once the agent has been created, click the checkbox next to the Cassandra agent.
8. Click Edit Properties.
9. Select Modify the default properties for this agent.
10. Edit the agent properties for the Cassandra agent instance. See Configuring Agent Properties.
11. Click Activate.

To modify the properties of an existing agent, follow steps 7-11.

Configuring Agent Properties

This is a full list and explanation of the configurable properties of the Foglight for Cassandra agent. The Agent Installer Wizard provides access to the essential subset of available properties. To modify other properties or modify the agent configuration after creation, use the Agent Status dashboard.

This section includes the following key areas:

• Setting DB Connection
• Setting JMX Connections
• Setting Collection Periods
• Setting Options

Setting DB Connection

The agent requires a connection to the cluster in order to gather information about the cluster and data structure. The agent may be referred to other nodes than the one specified.

• IP or Hostname – Host where Cassandra node is running. Default is “localhost”. (e.g.<hostname> or <IP address>)
• Port – The CQL native transport port for the Cassandra node. Default is 9042.
• Username – User that can connect to the Cassandra node.
• Password – Password of the user that can connect to the Cassandra node.
• Use TLS/SSL? – Whether the database connection should be made over SSL.

Setting JMX Connections

The agent requires JMX access to individual nodes in the cluster to gather most operational metrics.

• Node Connections – IP/Hostname for each node in the cluster to be monitored must be added to this list. Values for the columns Port, Username, and Password can be specified for each node if they differ. If no value is provided for these, values will be taken from the default fields under JMX Connections (see below). Check “Use as Host Alias?” to indicate that the value provided for “IP or Hostname” should be used as an alias for the discovered hostname for the respective node. Secondary Property Lists are global and can be shared between agents. To create a new property list for a different cluster, clone an existing list and then edit and save it as the selected list for that agent.
• Default Port – JMX listening port for the Cassandra node. This value will be used as the default value in the Node Connections list if no other value is provided. Default is 7199.
• Default Username – JMX User that can connect to the Cassandra node. This value will be used as the default value in the Node Connections list if no other value is provided.
• Default Password – Password of the JMX User that can connect to the Cassandra node. This value will be used as the default value in the Node Connections list if no other value is provided.
• Use TLS/SSL? – Whether the JMX connection should be made over SSL.

Setting Collection Periods

The Collection Periods field list in the agent properties is used to set the sample frequencies. You can disable a collection by setting its collection interval to 0. The defaults are set based on the type of data being collected for relevancy.

• Availability – Controls the query interval for node availability collections.
• Cluster and Node – Controls the query interval for cluster and node collections over the database driver.
• Keyspace and Table – Controls the query interval for keyspace and table collections over the database driver.
• JMX Cluster – Controls the query interval for cluster metrics over JMX.
• JMX Keyspace – Controls the query interval for keyspace metrics over JMX. If Max Tables is set to zero keyspace metrics are not collected.
• JMX Table – Controls the query interval for table metrics over JMX. If Max Tables is set to a positive value, metrics controlled by this period are collected for the largest tables only. If Max Tables is equal to zero this period is not used. If Max Tables is set to -1 this period is used to collect metrics from all tables.
• JMX Table Size – Controls the query interval for table size metrics over JMX. If Max Tables is set to a positive value, metrics controlled by this period are collected for all tables for the purpose of determining the largest tables on the cluster. If Max Tables is less than or equal to zero this period is not used.
• Slow Query – Controls the query interval for collection of the DSE Slow Query log.
• Trace – Controls the query interval for collection of the trace log. Note that tracing must be enabled on the Cassandra instance(s) separately for data to be collected.
• JVM – Controls the query interval for the JVM collections over JMX.

Setting Options

• Cluster Alias - The name under which the Cassandra cluster will be known in Foglight. Normally this field should be left blank, in which case the cluster’s actual name is used. In environments where two or more clusters have the same name, setting aliases for at least all but one of the clusters will prevent their monitoring data from merging and/or overwriting each other in Foglight.
• Host Aliases - Aliases can optionally be added for Cassandra nodes in case hostnames as discovered by the agent differ from those used for the Hosts monitor. This Secondary Property List is a pairing of the uniquely identifying Host ID (a UUID string) and the desired hostname alias. There are two ways to determine a node’s Host ID for use in the Aliases list. For a node that has already been monitored for some time, the Host ID can be found on the Nodes Table of the Cassandra Clusters dashboard (the Host ID column is hidden by default, so it will need to be made visible via the table’s “Show columns” dialog). Host IDs can also be queried directly from a specific Cassandra instance with the following CQL:

SELECT host_id FROM system.local WHERE key='local';

• Contact Points - Additional contact points for the database driver to assist with making an initial connection if the node specified by the main contact point is not available.
• Max Traces/Queries - The maximum number of traces or slow query entries to collect per their respective collection periods. Limiting the number here prevents the FMS from being overwhelmed by data (particularly from traces) on very active instances.
• Max Tables - The number of tables for which to collect full metrics. If set to 0, no keyspace or table metrics are collected. Must be a non-negative value.

Removing Monitored Databases

1. Go to the Databases dashboard.
2. Select the databases you wish to remove.
3. Click Settings, and then OK.

Roles

Two roles, Cassandra User and Cassandra Administrator, are installed with the cartridge. Viewing Cassandra dashboards requires that a user be assigned one of these or have the core Administrator role. The Set Trace Probability function on the Traces Sample dashboard requires the Cassandra Administrator role.