Metrics

Metrics are an important part of observability, allowing us to understand what the Penumbra software is doing. Penumbra Labs runs Grafana instances for the public deployments:

  • https://grafana.testnet.penumbra.zone
  • https://grafana.testnet-preview.penumbra.zone

Viewing Metrics

TODO: add details on how to use Grafana:

  • link to https://grafana.testnet.penumbra.zone for dashboard on current testnet;
  • instructions on how to run Grafana + Prometheus for local dev setup (ideally this could work without requiring that pd itself is Dockerized, since local development is often more convenient outside of docker);
  • instructions on how to commit dashboards back to the repo.

Adding Metrics

We use a common structure for organizing metrics code throughout the penumbra workspace. Each crate that uses metrics has a top-level metrics module, which is private to the crate. That module contains:

  • a re-export of the entire metrics crate: pub use metrics::*;
  • &'static str constants for every metrics key used by the crate;
  • a pub fn register_metrics() that registers and describes all of the metrics used by the crate;

Finally, the register_metrics function is publicly re-exported from the crate root.

The only part of this structure visible outside the crate is the register_metrics function in the crate root, allowing users of the library to register and describe the metrics it uses on startup.

Internally to the crate, all metrics keys are in one place, rather than being scattered across the codebase, so it’s easy to see what metrics there are. Because the metrics module re-exports the contents of the metrics crate, doing use crate::metrics; is effectively a way to monkey-patch the crate-specific constants into the metrics crate, allowing code like:

#![allow(unused)]
fn main() {
metrics::increment_counter!(
    metrics::MEMPOOL_CHECKTX_TOTAL,
    "kind" => "new",
    "code" => "1"
);
}

The metrics keys themselves should:

  • follow the Prometheus metrics naming guidelines
  • have an initial prefix of the form penumbra_CRATE, e.g., penumbra_stake, penumbra_pd, etc;
  • have some following module prefix that makes sense relative to the other metrics in the crate.

For instance:

#![allow(unused)]
fn main() {
pub const MEMPOOL_CHECKTX_TOTAL: &str = "penumbra_pd_mempool_checktx_total";
}

Backing up Grafana

After being changed, Grafana dashboards should be backed up to the repository for posterity and redeployment.

Grafana has an import/export feature that we use for maintaining our dashboards.

  1. Export the dashboard as JSON with the default settings
  2. Rename the JSON file and copy into the repo (config/grafana/dashboards/)
  3. PR the changes into main, and confirm on preview post-deploy that it works as expected.

Editing metrics locally

To facilitate working with metrics locally, first run a pd node on your machine with the metrics endpoint exposed. Then, you can spin up a metrics sidecar deployment:

cd deployments/compose
just metrics

To add new Grafana visualizations, open http://localhost:3000 and edit the existing dashboards. When you’re happy with what you’ve got, follow the “Backing up Grafana” instructions above to save your work.