Three Empty Grafana Panels, Three Different Causes: MySQL Auth Plugins, a Missing URL Scheme and an Exporter That Doesn't Exist

By Lars Nielsen ·

My search engine's monitoring had a Grafana dashboard with a panel per database: MySQL, PostgreSQL and OpenSearch. At different times over the summer, each of those panels was empty. An empty panel doesn't say why, and in all three cases the answer was different, and usually more than one thing at once. Here they are, because the causes are the kind that turn up again in anyone's monitoring setup.

The MySQL panel: a stale config, a wrong host, and an auth plugin

Every MySQL panel had been empty since the project split into a frontend server and a separate pipeline server that held MySQL. Three causes stacked up:

  • The pipeline server's monitoring agent config was a copy from before that server had MySQL at all. The intended setup was always that the frontend's own agent reached the database through an existing tunnel.
  • That agent's connection string pointed at mysql:3306, an alias from before the split that didn't exist on the frontend. It should have been the tunnel container.
  • With the host fixed, the exporter still reported mysql_up as 0. The monitoring user had been created with MySQL 8's default authentication plugin, caching_sha2_password.

MySQL's documentation explains the last one: to connect with an account that uses caching_sha2_password, you need either a secure connection or an unencrypted connection that supports password exchange using an RSA key pair. The tunnel was plain TCP, and the commit records that the exporter's client couldn't complete that key exchange over it. The fix switched the monitoring user to mysql_native_password, and mysql_up went from 0 to 1.

That fix has an expiry date worth knowing about. MySQL's documentation says mysql_native_password is deprecated as of 8.0.34, disabled by default in 8.4 and removed in 9.0. It worked on that server's MySQL 8.0, but the lasting fix is a TLS connection, or an exporter configured for the RSA key exchange, not the older plugin.

The PostgreSQL panel: four bugs in a config that had never run

The dedicated database server had a monitoring agent config, written when the server was set up. It had simply never been started, so every bug in it was untested, and each one hid the next:

  • A password variable the config needed was referenced but never defined in the encrypted secrets.
  • The Ansible task that starts the monitoring agents covered the two other servers but not this one, so even a correct config would never have run.
  • The config used a component called prometheus.exporter.opensearch, which doesn't exist in Grafana Alloy, the agent in use. It was disabled with a comment, to be fixed separately.
  • The PostgreSQL exporter was given data_source_name. Alloy's documentation lists the argument as data_source_names, a list. The config didn't parse.

Inside that last one was a fifth problem. The rendered connection string had no scheme: user:password@host:5432/db instead of postgresql://user:password@host:5432/db. Go's URL parser, which Alloy is built with, rejects that. I reproduced it: url.Parse fails with "first path segment in URL cannot contain colon", while the version with the scheme parses with the host where it should be.

The same test shows a less obvious problem. Go's error message quotes the whole string it couldn't parse, password included. The commit records that the password was rotated again for exactly that reason: during debugging, a malformed-connection-string error had printed it. If a secret is inside a URL, any error about the URL can leak it.

The OpenSearch panel: the component that didn't exist

That left OpenSearch, disabled during the PostgreSQL fix. Alloy has no OpenSearch exporter; Alloy's component list has no such page. It does have prometheus.exporter.elasticsearch, and OpenSearch is API-compatible with Elasticsearch for this purpose. The fix tested that exporter on its own against the live node before wiring it in, used the right argument name, address, and pointed it at localhost, because the agent runs on the host network and OpenSearch is only published on the loopback interface.

The panel then showed real data, including a cluster health of yellow. That was correct: a single node with replica shards that had nowhere to go.

What generalises

Three things. An empty panel means "no data reached Grafana", which can fail at the agent, the credentials, the network path or the config syntax; check whether the agent is even running before anything else. A config that has never run is not a config that works: every one of the PostgreSQL bugs was found the first time it was started. And treat connection strings with passwords in them as secrets everywhere they can appear, including in error messages.

Filed under

Leave a comment

Comments are reviewed before they appear.

Basic HTML allowed: links, emphasis, code and lists. Web addresses become links automatically.