Add exception handling for MBean disappearing between listing and evaluation

Closes instaclustr#27.

Squashed commit of the following:

commit c8eafefc7732d84114c990f81d6ad70489dafcec
Author: Adam Zegelin <adam@zegelin.com>
Date:   Fri Feb 21 15:20:55 2020 -0800

    Code cleanup.

commit 425df4699f963213c5498b3f1eedec5fdb38a609
Merge: 0b5e2c0 21307ab
Author: Adam Zegelin <adam@zegelin.com>
Date:   Fri Feb 21 15:11:05 2020 -0800

    Merge branch 'fix-mbean-jmx-harvester' of https://github.com/suhailpatel/cassandra-exporter into suhailpatel-fix-mbean-jmx-harvester

commit 21307ab
Author: Suhail Patel <me@suhailpatel.com>
Date:   Tue Jul 9 16:46:24 2019 +0100

    Dynamically update the set of current MBeans as we go along

    Again, if exceptions occur in other parts of the pipeline, MBeans are
    registered but are not removed which causes other bugs.

    Note that we use an immutable copy of the sets because the currentMBeans
    set is being updated constantly.

commit c4b46dc
Author: Suhail Patel <me@suhailpatel.com>
Date:   Sat Jul 6 20:51:06 2019 +0100

    Add exception handling for MBean disappearing between listing and evaluation

    There was a condition where evaluating the set of added MBean can cause
    an exception. This would occur if the MBean had disappeared between
    listing the MBean and extracting the MBean info. If this happened in the
    middle of a set of added MBeans, it would mean our current MBeans set in
    the JMXHarvester isn't up to date with the current MBeans.

    This has cascading impact down the line because there are MBeans
    registered from previous runs that are not correctly associated with the
    set of current MBeans. This means they will never be cleaned up. This
    has impact on the collect stage which then results in other exceptions.

    We found this occur during Repair jobs when the repair durations are
    very small.

Author: zegelin

standalone/src/main/java/com/zegelin/cassandra/exporter/JMXHarvester.java

@@ -33,31 +33,43 @@ public class JMXHarvester extends Harvester {
         addCollectorFactory(RemoteGossiperMBeanMetricFamilyCollector.factory(metadataFactory));
     }
 
-    private Set<ObjectInstance> currentMBeans = ImmutableSet.of();
+    private Set<ObjectInstance> currentMBeans = Sets.newHashSet();
 
     void reconcileMBeans() {
         try {
-            final Set<ObjectInstance> mBeans = mBeanServerConnection.queryMBeans(null, null);
+            final Set<ObjectInstance> newMBeans = mBeanServerConnection.queryMBeans(null, null);
 
             // unregister
             {
-                final Set<ObjectInstance> removedMBeans = Sets.difference(currentMBeans, mBeans);
+                final Set<ObjectInstance> removedMBeans = Sets.difference(currentMBeans, newMBeans);
 
                 logger.debug("Removing {} old MBeans.", removedMBeans.size());
 
                 for (final ObjectInstance instance : removedMBeans) {
                     unregisterMBean(instance.getObjectName());
+                    currentMBeans.remove(instance);
                 }
             }
 
             // register
             {
-                final Set<ObjectInstance> addedMBeans = Sets.difference(mBeans, currentMBeans);
+                final Set<ObjectInstance> addedMBeans = Sets.difference(newMBeans, currentMBeans);
 
                 logger.debug("Found {} new MBeans.", addedMBeans.size());
 
                 for (final ObjectInstance instance : addedMBeans) {
-                    final MBeanInfo mBeanInfo = mBeanServerConnection.getMBeanInfo(instance.getObjectName());
+                    final MBeanInfo mBeanInfo;
+                    try {
+                        mBeanInfo = mBeanServerConnection.getMBeanInfo(instance.getObjectName());
+
+                    } catch (final InstanceNotFoundException e) {
+                        // We may get an InstanceNotFoundException if the mBean was
+                        // really short lived (such as for a repair job or similar)
+                        // and Cassandra unregistered it between the probe and this
+                        // instance being evaluated
+                        continue;
+                    }
+
                     final Descriptor mBeanDescriptor = mBeanInfo.getDescriptor();
 
                     final String interfaceClassName = (String) mBeanDescriptor.getFieldValue(JMX.INTERFACE_CLASS_NAME_FIELD);
@@ -79,7 +91,7 @@ void reconcileMBeans() {
 
                     logger.debug("Registering MBean/MXBean {}.", instance);
 
-                    final boolean isMXBean = Boolean.valueOf((String) mBeanDescriptor.getFieldValue(JMX.MXBEAN_FIELD));
+                    final boolean isMXBean = Boolean.parseBoolean((String) mBeanDescriptor.getFieldValue(JMX.MXBEAN_FIELD));
 
                     final ObjectName objectName = instance.getObjectName();
                     final Object mBeanProxy;
@@ -90,11 +102,10 @@ void reconcileMBeans() {
                     }
 
                     registerMBean(mBeanProxy, objectName);
+                    currentMBeans.add(instance);
                 }
             }
 
-            currentMBeans = mBeans;
-
         } catch (final Throwable e) {
             logger.error("Failed to reconcile MBeans.", e);
         }