New Relic integrations with the Google Cloud Platform (GCP) include one that reports Google Cloud Dataproc data to New Relic. This document explains how to activate the GCP Dataproc integration and describes the data it reports.
Features
Cloud Dataproc is Google Cloud's managed service for running Apache Hadoop, Apache Spark, and other open-source data-processing workloads. New Relic Dataproc integration collects HDFS, YARN, job, and operation metrics for each Dataproc cluster.
Activate integration
To enable the integration, follow standard procedures to connect your GCP service to New Relic:
Polling frequency
New Relic integrations query your GCP services according to a polling interval that varies by integration. The polling frequency for Google Cloud Dataproc is 5 minutes. The resolution is 1 data point every minute.
Important
Cloud Dataproc supports 1-minute polling intervals for WIF integrations in limited preview. To request access, contact New Relic support.
Workload Identity Federation
Find and use data
After you enable the integration, your Dataproc resources appear as entities in the New Relic entity explorer. To see dashboards and manage services, go to one.newrelic.com > All capabilities > Infrastructure > GCP.
All Dataproc metrics available in GCP Cloud Monitoring are collected as dimensional metrics in the Metric event type. Additional metrics beyond this table are collected automatically. See Google's Dataproc metrics documentation for the complete list.
Entities
Metric data
Key metrics — Cluster
Metric name | Unit | Description |
|---|---|---|
| Count | Number of HDFS DataNodes running on the cluster. |
| Gibibytes | HDFS storage capacity available on the cluster. |
| Percent | Fraction of HDFS storage currently in use on the cluster. |
| Count | Number of unhealthy HDFS blocks on the cluster. |
| Count | Number of jobs submitted to the cluster. |
| Count | Number of jobs currently running on the cluster. |
| Count | Number of jobs that have failed on the cluster. |
| Seconds | Time from job submission to Dataproc reporting completion. |
| Seconds | Time jobs have spent in a given state, faceted by state. |
| Count | Number of cluster operations submitted. |
| Count | Number of cluster operations currently running. |
| Count | Number of cluster operations that have failed. |
| Seconds | Time from operation submission to Dataproc reporting completion. |
| Percent | Fraction of YARN memory currently allocated. |
| Gibibytes | Total YARN memory available on the cluster. |
| Gibibytes | YARN memory currently requested but not yet allocated by the scheduler. |
| Count | Number of YARN containers on the cluster. |
| Count | Number of active YARN applications on the cluster. |
| Count | Number of YARN NodeManagers running on the cluster. |
| Count | Number of YARN virtual cores on the cluster. |
For the complete list of Dataproc metrics, see Google's Dataproc metrics documentation.
Service account or user account
Find and use data
After activating the integration and waiting a few minutes (based on the polling frequency), data will appear in the New Relic UI. To find and use your data, including links to your and alert settings, go to one.newrelic.com > All capabilities > Infrastructure > GCP > (select an integration).
Data is attached to the following event type:
Entity | Event Type | Provider |
|---|---|---|
Cluster |
|
|
For more on how to use your data, see Understand and use integration data.
Dataproc Cluster data
Metric | Unit | Description |
|---|---|---|
| Count | Indicates the number of HDFS DataNodes that are running inside a cluster. |
| Gibibytes | Indicates capacity of HDFS system running on cluster in GB. |
| Percent | The percentage of HDFS storage currently used. |
| Count | Indicates the number of unhealthy blocks inside the cluster. |
| Seconds | The time jobs took to complete from the time the user submits a job to the time Dataproc reports it is completed. |
| Seconds | The time jobs have spent in a given state. |
| Count | Indicates the number of jobs that have failed on a cluster. |
| Count | Indicates the number of jobs that are running on a cluster. |
| Count | Indicates the number of jobs that have been submitted to a cluster. |
| Seconds | The time operations took to complete from the time the user submits a operation to the time Dataproc reports it is completed. |
| Seconds | The time operations have spent in a given state. |
| Count | Indicates the number of operations that have failed on a cluster. |
| Count | Indicates the number of operations that are running on a cluster. |
| Count | Indicates the number of operations that have been submitted to a cluster. |
| Percent | The percentage of YARN memory is allocated. |
| Count | Indicates the number of active YARN applications. |
| Count | Indicates the number of YARN containers. |
| Gibibytes | Indicates the YARN memory size in GB. |
| Count | Indicates the number of YARN NodeManagers running inside cluster. |
| Gibibytes | The current memory request, in GB, that is pending to be fulfilled by the scheduler. |
| Count | Indicates the number of virtual cores in YARN. |