Installing and Configuring Agents

Installation instructions for Foglight for Snowflake are detailed in the following sections and should be performed in the following order:

  1. Configuring Snowflake
  2. Creating and Configuring Agents

Configuring Snowflake

Monitoring requires the agent to have a Snowflake user with sufficient privileges to access specific data and functions in system tables within the Snowflake and user-created databases. To facilitate this, it is recommended to create a dedicated role and assign it to the user. Below is a brief list of steps to set this up. Skip any configuration steps that have already been completed. For more information, refer to the Snowflake User Management🔗 documentation.

Snowflake Agent User Permissions

Some Snowflake collections require the user to be assigned the ACCOUNTADMIN role. If you do not want to grant ACCOUNTADMIN to the user, the following collections will not be available:

  • Current Locks
  • Resource Monitors

Also, the following roles will be required:

  1. In the Snowflake console, using a user with the ACCOUNTADMIN role, navigate to Admin > Users & Roles in the navigation pane on the left.
  2. Select Roles at the top of the page, then select Table below it. At the top-right of the page, click + Role to create a new role and name it MONITORADMIN or a name of your choice.
  3. Select the new role in the list of roles and then on the details page click the … icon at the top-right. Click Manage global privileges in the menu.
  4. In the Manage Global Privileges for Role popup, use the Global privilege to grant dropdown to select Monitor Execution and Monitor Usage, then click Update Privileges.
  5. If the user already exists, click Grant to User on the details page to grant the MONITORADMIN role to the user. Otherwise, the role can be attached during user creation.
  6. Assign the role, the ability to execute queries on a warehouse. In the left navigation pane, navigate to Admin > Warehouses and select or create the warehouse that will be assigned to the monitoring user. On the details page, grant the USAGE privilege to MONITORADMIN. Refer to Selecting a Compute Warehouse for important information to consider when designating a warehouse.
  7. Assign the role privileges on data objects. In the left navigation pane, navigate to Data > Databases, and select the Snowflake database. In the Privileges section, click + Privilege, assign the IMPORTED PRIVILEGES to the MONITORADMIN role, and then click Grant Privileges. 8.For each additional database you want the agent to monitor, assign the USAGE role for the database to MONITORADMIN. For each existing pipe (found in _database > schema > Pipes), assign the MONITOR privilege. You can also assign the MONITOR - FUTURE PIPE privilege at the database level so that any new pipes created will automatically grant monitoring privileges to MONITORADMIN.

Snowflake User Setup

  1. In the Snowflake console, with a user that has the ACCOUNTADMIN role selected, navigate to Admin > Users & Roles in the left navigation pane.
  2. At the top-right of the page, click + User to create a new user and provide the relevant details.
  3. Expand Advanced User Options, and assign the default role as MONITORADMIN or the role you created with the appropriate privileges. Also, assign a default warehouse for the user to use when performing queries. Refer to Selecting a Compute Warehouse below for important information to consider when designating a warehouse.
  4. Click Create User.

Selecting a Compute Warehouse

While the Snowflake agent requires minimal resources to execute monitoring queries, there are important technical and billing implications to consider when designating a warehouse for the Snowflake user to employ. The warehouse is assigned to the user either through the default warehouse setting for the Snowflake user or through the Agent Properties.

First, the Snowflake user must have the privilege to execute queries on a warehouse. This requires the USAGE privilege. It is preferred that the warehouse be either constantly running or set to auto-resume. Otherwise, an admin with OPERATE privilege on the warehouse will be required to start it from a suspended state. If auto-suspend is enabled, the warehouse will suspend itself when it has not received a query in the last x minutes. If the warehouse is set to auto-resume, then it will automatically resume to execute a query when one is submitted. There is generally a delay of several seconds in query execution when a warehouse resumes while it provisions resources, but this will have no impact on the agent’s data collection.

Secondly, it is important to understand how Snowflake charges for compute resources. Based on the size of a warehouse, Snowflake will charge x credits per hour of usage. While charges are calculated per second (minimum 60 seconds) of usage, usage simply means that the warehouse is in an active (not suspended or provisioning) state. This also means that if a warehouse were to resume from a suspended state to run queries for 5 seconds, it will still be charged for 60 seconds of usage. If it were to do this once a minute, the cost would be the same as if the warehouse was constantly active and running queries.

Therefore, there are two recommended configurations for maximum benefit at low cost:

  1. Assign the monitoring user to a warehouse handling other workloads that has compute resources to spare that is either constantly running or not expected to receive a significant spike in query submissions while auto-suspended. Running agent queries on this type of warehouse will not result in any additional costs and very low impact on existing workload.
  2. Run the agent on a dedicated warehouse. This will have no impact on other workloads, but will increase costs. However, we have several recommendations on how to reduce these costs:
    1. Use a warehouse of X-Small size. Agent queries do not require many compute resources.
    2. Enable auto-resume and auto-suspend with a very low suspension delay, like 15 seconds. Queuing while waiting for a suspended warehouse to resume will not affect data integrity.
    3. Set collection intervals in the Agent Properties to similar intervals and none shorter than 5 or 10 minutes. If different collections are run at the same time and at a longer interval, this will maximize the amount of time that a warehouse can remain suspended and thus not using credits.

Creating and Configuring Agents

The Agent Status page can be used to create new agents and configure and manage existing agents. To access the page navigate to Dashboards > Administration > Agents > Agent Status in the left pane.

To create a new agent instance:

Deploy the Snowflake agent package to the FglAM before creating the agent if it has not been deployed yet. Click Deploy Agent Package on the Agent Status or Agent Managers page to perform this.

  1. Click Create Agent.
  2. Select the hosts to which you want to deploy agent packages.
    Considerations for this may include physical or virtual locality to the monitored instance, allocated resources, or grouping with other agents of the same type or monitored environment.
  3. Click Next.
  4. Select the SnowflakeAgent type. Then, select the Specify Name radio button and enter agent instance name. Click Next.
  5. Click Finish.
  6. Once the agent has been created, select the checkbox next to the Snowflake agent.
  7. Click Edit Properties.
  8. Select Modify the default properties for this agent.
  9. Edit the agent properties for the Snowflake agent instance:
  10. Click Activate.

To modify the properties of an existing agent, go to step 3, deactivate the agent, update the configuration, and then reactivate the agent.

Configuring Agent Properties

When an agent connects to the Foglight Management Server, it receives a set of properties used to configure its running state.

The Foglight Cartridge for Snowflake includes default agent properties. Agent properties can be specific to a single agent instance or apply to multiple agents. To configure the agent properties, navigate to Administration > Agent > Agent Status. On the Agent Status page, select the checkbox for the required agent, and click Edit Properties. Click Modify the private properties for this agent to edit the properties of the selected agent or click Modify the properties for all MySQLAgent agents to edit the properties of all Snowflake agents.

This section includes the following key areas:

Snowflake Agent properties

Setting Account Details (mandatory)

  • Organization – The organization that this account belongs to. The Foglight topology created for this account will be under this organization, so it is very important that any accounts belonging to this organization are all provided the exact same string in the agent properties. It is not required for this to match the organization name used by Snowflake.
  • Account – The name this account will be given in Foglight. This is not used for connection purposes, so it is not required for this to match the account name used by Snowflake.

Setting Connection Details (mandatory)

  • Account Identifier – This is generally in the format of organization-account and used in the URL for accessing an account like account_identifier.snowflakecomputing.com. You can also find the organization and account ID in the Snowflake console Admin>Accounts page. Hovering over the account name will make visible a link icon with the URL for the account containing the account ID.
  • Warehouse – Sets a default compute warehouse for the user provided to the agent to use when executing monitoring queries. If left blank, it will use the default warehouse assigned to the Snowflake user, if this exists. Read Selecting a Compute Warehouse earlier in this document for important information to consider when designating a warehouse.
  • Username – The Snowflake user that will be running monitoring queries.
  • Authentication Method – Options include Password, Key Pair, or OAuth.

Password Authentication (optional)

  • OAuth Token – Specifies the OAuth token to use for authentication when the OAuth authentication method is selected.
  • Password – The password for the Snowflake user when the Password authentication method is selected.

Key Pair Authentication (optional)

  • Private Key Path – File path to an optional private key file for user authentication.
  • Private Key Password – Password for the private key file.

Proxy Server (optional)

  • Use Proxy Server – Agent will connect to the specified proxy server when this is enabled.
  • Proxy Host – Proxy server hostname.
  • Proxy Port – Proxy server port number.
  • Proxy User – Proxy server user.
  • Proxy Password – Proxy server user password.
  • Proxy Protocol – HTTP or HTTPS.

Setting DB Overrides (optional)

  • Database Name – The name of a database to override default collection properties for.
  • Exclude DB-level Stats – If true, data will not be collected for schema, pipes, and tables belonging to this database.

Setting Top Tables Collection (optional)

  • # of Top Tables – Maximum number of tables that will be included for each sample collection.
  • Order By – Dimension by which to order the tables collected before the cutoff limit. For example, if Active Bytes is selected, then the x tables with the largest Active Bytes will be collected.

Setting Top SQL Collection (optional)

Collects individual top SQL statements run during the sample period according to the configured order by and limit properties.

  • Enable Statement Tracking – If true, top statements will be collected based on the limit and order by values.
  • # of Top Statements – Maximum number of statements that will be included in each sample collection.
  • Statement Text Length – Cutoff limit for characters collected in a single SQL statement text string. If set to 0 or lower, the entire query text will be collected.
  • Order by – Dimension by which to order the tables collected before the cutoff limit. For example, if Execution Time is selected, then the x statements with the longest Execution Time will be collected.

Setting Collection Intervals (optional)

The Collection Interval fields in the agent properties are used to set sample frequencies for collections. The values provided are the recommended default intervals. Certain collections are not included in this list due to technical requirements that they take place at specific intervals. Any collections that are removed from this list will be scheduled at their default interval, while providing an interval of 0 for a collection will disable it entirely.

  • Collection Name – Name of the collection to override.
  • Interval (sec) – Number of seconds between collection samples.

Foglight for Snowflake FMS Roles

The following descriptions are for roles installed with the Foglight for Snowflake cartridge. They can be assigned to any Foglight user.

  • Snowflake Administrator – Allows access to all Snowflake modules and pages. When available, any action that include user-initiated interaction with the Snowflake service (i.e. any commands send to the Snowflake service or on-demand data collection) will be restricted to users that have this role. No actions of this kind are currently available.
  • Snowflake User – Allows access to all Snowflake modules and pages.

The core Foglight Administrator role provides access to all Snowflake modules and pages, just like the Snowflake User role. However, since the Administrator role includes additional privileges in Foglight, users who need access to Snowflake pages but do not require Administrator privileges should only be assigned the Snowflake User role.