Filtering Metrics With the observIQ Distro for OpenTelemetry Collector

On this tutorial, we are going to handle the widespread monitoring use case of filtering metrics inside observIQ’s distribution of the OpenTelemetry (OTEL) collector. Whether or not the metrics are deemed pointless, or they’re filtered for safety issues, the method is pretty simple.

For our pattern surroundings, we are going to use MySQL on Purple Hat Enterprise Linux 8. The vacation spot exporter can be to Google Cloud Operations, however the course of is exporter agnostic. We’re utilizing this exporter to offer the visible charts exhibiting the metric earlier than and after filtering.

Atmosphere Conditions

• Appropriate working system
• observIQ Distro for OTEL Collector put in
• MySQL put in
• MySQL Least Privilege Consumer (LPU) setup
• OTEL configured to gather metrics from MySQL

Sources

Preliminary Metrics

As soon as configured utilizing the LPU I created, MySQL metrics must be flowing. For our functions, we are going to deal with the particular metric mysql.buffer_pool.restrict. At present, our config.yaml MySQL part appears to be like like this:

mysql:
    endpoint: localhost:3306
    username: otel
    password: otelPassword
    collection_interval: 60s

After ready for a minimum of 5 minutes to get a superb quantity of information, metrics will look one thing like this in Google’s Metrics Explorer:

Filtering

Now that metrics are flowing, we are able to filter them. First, allow us to focus on the explanations to filter this particular metric. The reply is straightforward: It isn’t actually all that helpful or vital. It is going to, barring a configuration change by the DBA, be a flat line. Even after a configuration change, it could merely step that flat line up or down.

To do the filtering, we first want to have a look at the metadata file for the MySQL receiver. On this file, we discover a itemizing of the attributes and metrics related to this receiver. If we go to the metrics part of the file, and discover our pool restrict metric, we study it appears to be like like this:

mysql.buffer_pool.restrict:
	enabled: true
	description: The configured measurement of the InnoDB buffer pool.
	unit: By
	sum:
  	value_type: int
  	input_type: string
  	monotonic: false
  	aggregation: cumulative

This lets us know that it’s enabled by default, provides an outline, and another vital knowledge concerning the metric. As these are the defaults, we are able to interpret from it that if we set the enabled parameter to false, then it ought to disable — aka filter — this metric. It won’t be collected, and because it isn’t collected, it additionally won’t be despatched to the exporter.

To attain this in our configuration file, we make the next adjustments:

 mysql:
    endpoint: localhost:3306
    username: otel
    password: otelPassword
    collection_interval: 60s
    metrics:
      mysql.buffer_pool.restrict:
        enabled: false

This replicates the construction from the metadata file, however with every part else trimmed apart from the naked minimal variety of traces wanted to realize our purpose.

As soon as this has been modified, and the collector restarted, I once more wait a minimum of 5 minutes and verify Google’s Metrics Explorer to see what has modified:

As proven within the screenshot, knowledge was final despatched to Google at 10:48, and it’s now 11:13.

Conclusion

Whereas the knowledge wanted is positioned in a couple of totally different locations, filtering could be very straightforward to do. Don’t neglect that the metadata we checked out additionally offers different data that may be helpful in understanding your knowledge.