Add JMX MBean for getting tracer flare files

[deejgregor]

What Does This Do

Allow flare dumps and individual files from flares to be downloaded with JMX.

Motivation

This work was originally paired with adding a way to get the long-running traces queue in #10309. I initially started by adding JMX MBeans to retrieve just the pending and long running traces and counters. Once I added the long running traces to the flare report to parity with pending traces, I realized that a more generic mechanism to allow getting flare details over JMX might be useful. After adding a TracerFlare MBean, this seemed like a far more valuable route and I removed the code I had added for pending/long running trace MBeans.

Additional Notes

This was originally part of #9874, which is being broken out into a few individual PRs.

Commit notes:

• Add JMX MBean for getting tracer flare files -- Note: see if the JMX MBean ObjectName and operation names sound good. I kept the existing add* methods as-is, but this could be simplified by refactoring the add* methods into Reporter instances (with a new signature that passes a few more parameters to addReportToFlare). I think this refactoring would be a good change to make--let me know and I'll happily do that. I also considered not making the zip file an intermediary, and if you like, I could look at what that change might be, as well.

The JMX telemetry feature is controlled by dd.telemetry.jmx.enabled and is disabled by default. This option enables both JMXFetch telemetry (when JMXFetch is enabled, which it is by default) and the new tracer flare MBean at datadog.flare:type=TracerFlare. This new MBean exposes three operations:

java.lang.String listFlareFiles()

• Returns a list of sources and files available from each source.

java.lang.String getFlareFile(java.lang.String p1, java.lang.String p2)

• Returns a single file from a specific reporter (or flare source).
• If the file ends in ".txt", it is returned as-is, otherwise it is base64 encoded.

java.lang.String generateFullFlareZip()

• Returns a full flare dump, base64 encoded.

An easy way to enable this for testing is to add these arguments:

    -Ddd.telemetry.jmx.enabled=true
    -Dcom.sun.management.jmxremote
    -Dcom.sun.management.jmxremote.host=127.0.0.1
    -Dcom.sun.management.jmxremote.port=9010
    -Dcom.sun.management.jmxremote.authenticate=false
    -Dcom.sun.management.jmxremote.ssl=false

To test, you can use jmxterm (https://github.com/jiaqi/jmxterm) like this:

echo "run -b datadog.flare:type=TracerFlare listFlareFiles" | \
    java --add-exports jdk.jconsole/sun.tools.jconsole=ALL-UNNAMED \
        -jar jmxterm-1.0.4-uber.jar -l localhost:9010 -n -v silent

echo "run -b datadog.flare:type=TracerFlare getFlareFile datadog.trace.agent.core.LongRunningTracesTracker long_running_traces.txt" | \
    java --add-exports jdk.jconsole/sun.tools.jconsole=ALL-UNNAMED \
        -jar jmxterm-1.0.4-uber.jar -l localhost:9010 -n -v silent | \
    jq .

echo "run -b datadog.flare:type=TracerFlare generateFullFlareZip" | \
    java --add-exports jdk.jconsole/sun.tools.jconsole=ALL-UNNAMED \
        -jar jmxterm-1.0.4-uber.jar -l localhost:9010 -n -v silent | \
    base64 -d > /tmp/flare.zip && \
    unzip -v /tmp/flare.zip

Contributor Checklist

• Format the title according the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any useful labels
• Don't use close, fix or any linking keywords when referencing an issue.
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, move, or deletion
• Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]