hadoop_monitor
Hadoop Monitoring Probe

Hadoop is a software framework that facilitates distributed processing of large data sets. The Hadoop Monitoring probe handles all the common monitoring and data collection tasks (collecting QoS measurements and topology information) on a Hadoop cluster through a namenode, and gathers reports about all nodes in the cluster. The probe collects and stores data and information from the monitored system at customizable intervals. The probe generates alarms when the specified thresholds are breached.
Revision history
Date Description State Version
28.07.2020

What's New:

  • Added support for Cloudera 6.3.x.
  • Added support for Hadoop 3.0.x.
  • Added POBC templates support.
MD5 Checksum: 354dfd94a55cc3157cd3653bf1b88cc1
SHA-1 Checksum: 65a5f12de228b3b0d2e7246138467d04e7a9a736
GA 1.21
20.07.2020

What's New:

  • Added support for Cloudera 6.3.x.
  • Added support for Hadoop 3.0.x.
  • Added POBC templates support.

This is Beta build.

MD5 Checksum: 354dfd94a55cc3157cd3653bf1b88cc1
SHA-1 Checksum: 65a5f12de228b3b0d2e7246138467d04e7a9a736
1.21
23.06.2020

What's New:

  • Added support for multiple Apache Spark instances. Both Apache Spark and Spark 2 can be detected simultaneously.
  • Added support for HDFS roles FAILOVER and HTTPFS.
  • Fixed detection of HDFS role JOURNALNODE in Cloudera cluster environment.
  • Fixed detection of High Availability status for HDFS role NAMENODE and YARN role RESOURCEMANAGER in Cloudera cluster with Kerberos support enabled.
  • Fixed Impala and Zookeeper detection in Cloudera environment with Kerberos support enabled.
  • Added support for Zookeeper role Leader. 
  • Updated metric types for Impala and Zookeeper.
  • Changed listen IP for Hue from localhost to external IP. 
  • Default profile name changed from HADOOP to local IP address.

Fixed Defects:

  • Fixed an issue in which the probe failed to update the Hadoop inventory. Support case 31798577.
  • Fixed an issue with enabling alarms in Admin Console. Support case 31814444.
MD5 Checksum: 069f48cc52cb52d787d72d115ce7c4b2
SHA-1 Checksum: ab97f9221a9c15c014573ad1b7ae5775f50a2932
1.20
24.09.2019

For the hadoop_monitor 1.11 Release Notes, see https://docops.ca.com/rest/ca/product/latest/topic?hid=hadoop_monitor_RN&space=UIMPGA&language=&format=rendered

MD5 Checksum: ca7c789ab5bcb2a7d0deaf55edc840a7
SHA-1 Checksum: 6fd91e639facc9764598c5abfa605e6ee9c6e950
1.11
21.08.2019

For the hadoop_monitor 1.11-Beta Release Notes, see https://docops.ca.com/rest/ca/product/latest/topic?hid=hadoop_monitor_RN&space=UIMPGA&language=&format=rendered

This is Beta build.

MD5 Checksum: ca7c789ab5bcb2a7d0deaf55edc840a7
SHA-1 Checksum: 6fd91e639facc9764598c5abfa605e6ee9c6e950
1.11
17.07.2019

What's New:

  • Updated the probe framework to the latest probe API.
  • Added support for Apache Hive. The roles HiveServer2 and HiveMetastore have now been added.
  • Added support for Apache Impala. The roles Impala Daemon, StateStore Daemon, and Catalog Daemon have been added.
  • The following metrics are supported for the Hive and Impala roles:
    • QOS_HADOOP_PROC_CPU_SYS
    • QOS_HADOOP_PROC_CPU_SYS_PERCENT
    • QOS_HADOOP_PROC_CPU_TOTAL
    • QOS_HADOOP_PROC_CPU_PERCENT
    • QOS_HADOOP_PROC_CPU_USER
    • QOS_HADOOP_PROC_CPU_USER_PERCENT
    • QOS_HADOOP_PROC_IO_FD
    • QOS_HADOOP_JVM_GC_COUNT
    • QOS_HADOOP_JVM_GC_TIME
    • QOS_HADOOP_JVM_HEAP_CMTD
    • QOS_HADOOP_JVM_MEMMAX_M
    • QOS_HADOOP_JVM_MEMHEAP_USED
    • QOS_HADOOP_JVM_MEMHEAP_USED_PERCENT
    • QOS_HADOOP_JVM_THREADS_BLCKD
    • QOS_HADOOP_JVM_THREADSTIME_WTNG
    • QOS_HADOOP_JVM_THREADS_WTNG
    • QOS_HADOOP_JVM_THREADS_NEW
    • QOS_HADOOP_JVM_THREADS_RNBLE
    • QOS_HADOOP_JVM_THREADS_TERMINATED
    • QOS_HADOOP_PROC_SYS_MEM_MAJOR_FAULTS
    • QOS_HADOOP_PROC_MEM_MINOR_FAULTS
    • QOS_HADOOP_PROC_MEM_PAGE_FAULTS
    • QOS_HADOOP_PROC_MEM_SIZE**
    • QOS_HADOOP_JVM_MEMNONHEAP_CMTD
    • QOS_HADOOP_JVM_MEMNONHEAP_USED
    • QOS_HADOOP_PROC_ID
    • QOS_HADOOP_PROC_MEM_RESIDENT
    • QOS_HADOOP_PROC_MEM_SHARE

Fixed Defects:

  • Fixed an issue in which the probe failed to update the Hadoop inventory. Support case 00856698

 

For more information, refer https://docops.ca.com/rest/ca/product/latest/topic?hid=hadoop_monitor_RN&space=UIMPGA&language=&format=rendered

MD5 Checksum: 4e85e3ab27b3ba50acf8713314778ec1
SHA-1 Checksum: d0c3df972082d3ed83215791b6c0b4ce6126fdeb
1.10
29.05.2019

1. Updated old probe framework to the newest probe api
2. Fixed issue with changed robot metadata after probe deploy
3. Added roles for Apache Hive: HiveServer2 and HiveMetastore
4. Added roles for Apache Impala: Impala Daemon, StateStore Daemon and Catalog Daemon
5. Supported metrics for Hive and Impala roles:

  • QOS_HADOOP_PROC_CPU_SYS
  • QOS_HADOOP_PROC_CPU_SYS_PERCENT
  • QOS_HADOOP_PROC_CPU_TOTAL
  • QOS_HADOOP_PROC_CPU_PERCENT
  • QOS_HADOOP_PROC_CPU_USER
  • QOS_HADOOP_PROC_CPU_USER_PERCENT
  • QOS_HADOOP_PROC_IO_FD
  • QOS_HADOOP_JVM_GC_COUNT
  • QOS_HADOOP_JVM_GC_TIME
  • QOS_HADOOP_JVM_HEAP_CMTD
  • QOS_HADOOP_JVM_MEMMAX_M
  • QOS_HADOOP_JVM_MEMHEAP_USED
  • QOS_HADOOP_JVM_MEMHEAP_USED_PERCENT
  • QOS_HADOOP_JVM_THREADS_BLCKD
  • QOS_HADOOP_JVM_THREADSTIME_WTNG
  • QOS_HADOOP_JVM_THREADS_WTNG
  • QOS_HADOOP_JVM_THREADS_NEW
  • QOS_HADOOP_JVM_THREADS_RNBLE
  • QOS_HADOOP_JVM_THREADS_TERMINATED
  • QOS_HADOOP_PROC_SYS_MEM_MAJOR_FAULTS
  • QOS_HADOOP_PROC_MEM_MINOR_FAULTS
  • QOS_HADOOP_PROC_MEM_PAGE_FAULTS
  • QOS_HADOOP_PROC_MEM_SIZE
  • QOS_HADOOP_JVM_MEMNONHEAP_CMTD
  • QOS_HADOOP_JVM_MEMNONHEAP_USED
  • QOS_HADOOP_PROC_ID
  • QOS_HADOOP_PROC_MEM_RESIDENT
  • QOS_HADOOP_PROC_MEM_SHARE

This is Beta build.

MD5 Checksum: 4bc78141aaeb0e19fd52dfcb2fbb52ae
SHA-1 Checksum: bab21116504086f2057cab9f4fe04b1d07f30fd9
1.09
08.09.2018

This is a Beta build.

1.08
22.12.2014 First release of the probe for Admin Console GUI.
The probe is now available only through the web-based GUI and not through the Infrastructure Manager (IM) GUI.
For Detailed Release Notes- Please refer
https://wiki.ca.com/rest/ca/product/latest/topic?format=rendered&language=&space=UIMPGA&hid=hadoop_monitor_RN
1.01
Requirements
Platform: Please refer to the Nimsoft Compatibility Support Matrix for the latest information on supported platforms.
Software: CA Unified Infrastructure Management Server v7.6 or later,Robot version 5.23 or later,Java Virtual Machine (JVM) version 1.6 or later (deployed as part of the probe package)
Hardware: None