Metrics integration for fly-hosted Prometheus

There's a killer feature that was kind of hidden and burrowed within fly.io and I'll try to introduce you to it in just a second.

Ok, it wasn't hidden, it was right there in the docs.

In your DevOps or operations job, you most likely came across Prometheus and Grafana before. Whenever people need something graphed and have a data source with data to display, people reach for Grafana. Prometheus is a popular data source because there are streamlined metric providers for a multitude of services. All you have to do is give Prometheus a config to pull the data from the endpoint, a so-called scrape config.

The killer feature here is that with a certain configuration in your fly.toml, you can get those metrics pulled automatically to the fly-hosted Prometheus automagically.

Fly-hosted Prometheus? When you create a fly.io account you get access to the fly-hosted Prometheus and Grafana, which is nifty because then you don't need to host them yourself. Fly.io provides useful metrics in your Grafana/Prometheus for you to explore and build dashboards with. The docs have a full list of the metrics that come automatically, but those are more generic metrics, what we want are actual metrics from an application that we control.

For the following example, I've chosen caddy, which is easy and fast to deploy on fly.io. We only need a Dockerfile, a Caddyfile, an index.html and a fly.toml .
At the end of this article, you will have caddy metrics inside the hosted Prometheus and thus available in Grafana.

I'll paste the files and follow up with an explanation.

FROM caddy:2

ADD ./index.html /usr/share/caddy/
ADD Caddyfile /etc/caddy/Caddyfile

EXPOSE 80

Pretty straightforward, we use caddy:2 as our base and add an index.html and a Caddyfile. No surprises here.

{
    servers {
        metrics
    }
}

:80 {
    root * /usr/share/caddy
    file_server
}

:8091 {
    metrics
}

If you are new to Caddy, this syntax might seem alien. There are some things to note here. The { } at the top are the global settings for caddy. We have to explicitly enable metrics for caddy. It's not turned on by default because it has an impact on performance. Secondly, we define that for port 80 we have a file_server which has it's root at /usr/share/caddy. That's the folder we put our index.html in. Below, we specify our metrics endpoint at :8081. We won't provide the metrics on our public port but rather on an extra port.

You can put anything you want in the index.html. It's just for demoing purposes.

<html>
<head>
<title>Hello</title>
</head>
<body>
<h1>Let's go metrics</h1>
</body>
</html>

Lastly, we will take a look at our fly.toml that we need to have our metrics pulled automatically by fly.io. If you want to do this in your account, you would need to have installed the flyctl cli, created an account and created an app. When you are ready we can continue.

You can create a fly app with the following command.

$ flyctl apps create

Pick a fun name but remember it.
Here's the fly.toml for this little example.

app = "YOUR_APP_NAME"

kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]

[metrics]
port = 8091
path = "/metrics"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 80
  processes = ["app"]
  protocol = "tcp"

  [[services.ports]]
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

The [metrics] configuration directive instructs fly to scrape the caddy metrics provided at :8091 at /metrics. Just three lines but super powerful, especially when you plan to leverage Prometheus metrics within your apps.

If you want to deploy a fly.io application, as we specified in the fly.toml, you can issue the following command.

$ fly deploy

After it's deployed and healthy, make sure to make a few HTTP requests (your app is most likely available at $APP_NAME.fly.dev), and then explore those new metrics over at https://fly-metrics.net/explore

What would be the next steps? There's no alert manager on the fly-hosted Grafana. But we can easily use the grafana cloud in the free tier and add access to the fly-Prometheus instance over there. This way, you would get a cost-effective (read: free) way of getting alerted on your custom metrics.

Thanks for reading and let me know how you'll leverage this cool new trick.
And don't forget to run fly apps destroy $APP_NAME after you are done playing around.

sebbel on infrastructure

sebbel on infrastructure

Getting metrics into the fly-hosted prometheus instance

Leveraging automatic metric scraping on fly.io