Metrics

Metrics are an important part of observability, allowing us to understand what the Penumbra software is doing.

Adding Metrics

We use a common structure for organizing metrics code throughout the penumbra workspace. Each crate that uses metrics has a top-level metrics module, which is private to the crate. That module contains:

a re-export of the entire metrics crate: pub use metrics::*;
&'static str constants for every metrics key used by the crate;
a pub fn register_metrics() that registers and describes all of the metrics used by the crate;

Finally, the register_metrics function is publicly re-exported from the crate root.

The only part of this structure visible outside the crate is the register_metrics function in the crate root, allowing users of the library to register and describe the metrics it uses on startup.

Internally to the crate, all metrics keys are in one place, rather than being scattered across the codebase, so it's easy to see what metrics there are. Because the metrics module re-exports the contents of the metrics crate, doing use crate::metrics; is effectively a way to monkey-patch the crate-specific constants into the metrics crate, allowing code like:

metrics::increment_counter!(
    metrics::MEMPOOL_CHECKTX_TOTAL,
    "kind" => "new",
    "code" => "1"
);

The metrics keys themselves should:

follow the Prometheus metrics naming guidelines (opens in a new tab)
have an initial prefix of the form penumbra_CRATE, e.g., penumbra_stake, penumbra_pd, etc;
have some following module prefix that makes sense relative to the other metrics in the crate.

For instance:

pub const MEMPOOL_CHECKTX_TOTAL: &str = "penumbra_pd_mempool_checktx_total";

Backing up Grafana

After being changed, Grafana dashboards should be backed up to the repository for posterity and redeployment.

Grafana has an import/export (opens in a new tab) feature that we use for maintaining our dashboards.

View the dashboard you want to export, and click the share icon in the top bar.
Choose Export, and enable Export for sharing externally, which will generalized the datasource.
Download the JSON file, renaming it as necessary, and copy into the repo (config/grafana/dashboards/)
PR the changes into main, and confirm on preview post-deploy that it works as expected.

Editing metrics locally

A minimal metrics setup will be automatically provisioned as part of the devnet quickstart.

To add new Grafana visualizations, open http://127.0.0.1:3000 (opens in a new tab) and edit the existing dashboards. When you're happy with what you've got, follow the Backing up Grafana instructions above to save your work.

Generating traffic

In order to confirm that the metrics are wired up correctly, it's useful to generate network traffic against a pd endpoint. There's a barebones measure tool in the protocol repo (opens in a new tab), which can be used for this purpose.

To stream compact-blocks and view stats on the fetching:

❯ cargo run -q --release --bin measure -- --node http://127.0.0.1:8080 stream-blocks
[14s] ██████████████████████████████████████████████████ 4873332/4873332 325097/s ETA: 0s
Fetched at least 229.7 MB
Fetched 4873333 compact blocks, containing:
        481965 nullifiers
        649353 state payloads, containing:
                521087 note payloads
                42804 swap payloads
                85462 rolled up payloads

To perform a full sync across multiple client connections concurrently:

cargo run --release --bin measure -- --node http://127.0.0.1:8080 open-connections --num-connections 50 --full-sync

If you want to maintain open connections, rather than exiting on completion, use open-connections-active.

Please be respectful of community-run RPCs; you should only run this tool against endpoints you control.

Building protobuf Zero-knowledge proofs