Prometheus Blog

Enriching metrics with metadata labels can be surprisingly tricky in Prometheus, even if you're a PromQL wiz! The PromQL join query traditionally used for this is inherently quite complex because it has to specify the labels to join on, the info metric to join with, and the labels to enrich with. The new, still experimental info() function, promises a simpler way, making label enrichment as simple as wrapping your query in a single function call.

In Prometheus 3.0, we introduced the info() function, a powerful new way to enrich your time series with labels from info metrics. What's special about info() versus the traditional join query technique is that it relieves you from having to specify identifying labels, which info metric(s) to join with, and the ("data" or "non-identifying") labels to enrich with. Note that "identifying labels" in this particular context refers to the set of labels that identify the info metrics in question, and are shared with associated non-info metrics. They are the labels you would join on in a Prometheus join query. Conceptually, they can be compared to foreign keys in relational databases.

Beyond the main functionality, info() also solves a subtle yet critical problem that has plagued join queries for years: The "churn problem" that causes queries to fail when non-identifying info metric labels change, combined with missing staleness marking (as is the case with OTLP ingestion).

Whether you're working with OpenTelemetry resource attributes, Kubernetes labels, or any other metadata, the info() function makes your PromQL queries cleaner, more reliable, and easier to understand.

The Problem: Complex Joins and The Churn Problem

Let us start by looking at what we have had to do until now. Imagine you're monitoring HTTP request durations via OpenTelemetry and want to break them down by Kubernetes cluster. You push your metrics to Prometheus' OTLP endpoint. Your metrics have job and instance labels, but the cluster name lives in a separate target_info metric, as the k8s_cluster_name label. Here's what the traditional approach looks like:

sum by (http_status_code, k8s_cluster_name) (
    rate(http_server_request_duration_seconds_count[2m])
  * on (job, instance) group_left (k8s_cluster_name)
    target_info
)

While this works, there are several issues:

1. Complexity: You need to know:

• Which info metric contains your labels (target_info)
• Which labels are the "identifying" labels to join on (job, instance)
• Which data labels you want to add (k8s_cluster_name)
• The proper PromQL join syntax (on, group_left)

This requires expert-level PromQL knowledge and makes queries harder to read and maintain.

2. The Churn Problem (The Critical Issue):

Here's the subtle but serious problem: What happens when an OTel resource attribute changes in a Kubernetes container, while the identifying resource attributes stay the same? An example could be the resource attribute k8s.pod.labels.app.kubernetes.io/version. Then the corresponding target_info label k8s_pod_labels_app_kubernetes_io_version changes, and Prometheus sees a completely new target_info time series.

As the OTLP endpoint doesn't mark the old target_info series as stale, both the old and new series can exist simultaneously for up to 5 minutes (the default lookback delta). During this overlap period, your join query finds two distinct matching target_info time series and fails with a "many-to-many matching" error.

This could in practice mean your dashboards break and your alerts stop firing when infrastructure changes are happening, perhaps precisely when you would need visibility the most.

The Info Function Presents a Solution

The previous join query can be converted to use the info function as follows:

sum by (http_status_code, k8s_cluster_name) (
  info(rate(http_server_request_duration_seconds_count[2m]))
)

Much more comprehensible, isn't it? As regards solving the churn problem, the real magic happens under the hood: info() automatically selects the time series with the latest sample, eliminating churn-related join failures entirely. Note that this call to info() returns all data labels from target_info, but it doesn't matter because we aggregate them away with sum.

Basic Syntax

info(v instant-vector, [data-label-selector instant-vector])

• v: The instant vector to enrich with metadata labels
• data-label-selector (optional): Label matchers in curly braces to filter which labels to include

In its most basic form, omitting the second parameter, info() adds all data labels from target_info:

info(rate(http_server_request_duration_seconds_count[2m]))

Through the second parameter on the other hand, you can control which data labels to include from target_info:

info(
  rate(http_server_request_duration_seconds_count[2m]),
  {k8s_cluster_name=~".+"}
)

In the example above, info() includes the k8s_cluster_name data label from target_info. Because the selector matches any non-empty string, it will include any k8s_cluster_name label value.

It's also possible to filter which k8s_cluster_name label values to include:

info(
  rate(http_server_request_duration_seconds_count[2m]),
  {k8s_cluster_name="us-east-0"}
)

Selecting Different Info Metrics

By default, info() uses the target_info metric. However, you can select different info metrics (like build_info or node_uname_info) by including a __name__ matcher in the data-label-selector:

# Use build_info instead of target_info
info(up, {__name__="build_info"})

# Use multiple info metrics (combines labels from both)
info(up, {__name__=~"(target|build)_info"})

# Select build_info and only include the version label
info(up, {__name__="build_info", version=~".+"})

Note: The current implementation always uses job and instance as the identifying labels for joining, regardless of which info metric you select. This works well for most standard info metrics but may have limitations with custom info metrics that use different identifying labels. An example of an info metric that has different identifying labels than job and instance is kube_pod_labels, its identifying labels are instead: namespace and pod. The intention is that info() in the future knows which metrics in the TSDB are info metrics and automatically uses all of them, unless the selection is explicitly restricted by a name matcher like the above, and which are the identifying labels for each info metric.

Real-World Use Cases

OpenTelemetry Integration

The primary driver for the info() function is OpenTelemetry (OTel) integration. When using Prometheus as an OTel backend, resource attributes (metadata about the metrics producer) are automatically converted to the target_info metric:

• service.instance.id → instance label
• service.name → job label
• service.namespace → prefixed to job (i.e., <namespace>/<service.name>)
• All other resource attributes → data labels on target_info

This means that, so long as at least either the service.instance.id or the service.name resource attribute is included, every OTel metric you send to Prometheus over OTLP can be enriched with resource attributes using info():

# Add all OTel resource attributes
info(rate(http_server_request_duration_seconds_sum[5m]))

# Add only specific attributes
info(
  rate(http_server_request_duration_seconds_sum[5m]),
  {k8s_cluster_name=~".+", k8s_namespace_name=~".+", k8s_pod_name=~".+"}
)

Build Information

Enrich your metrics with build-time information:

# Add version and branch information to request rates
sum by (job, http_status_code, version, branch) (
  info(
    rate(http_server_request_duration_seconds_count[2m]),
    {__name__="build_info"}
  )
)

Filter on Producer Version

Pick only metrics from certain producer versions:

sum by (job, http_status_code, version) (
  info(
    rate(http_server_request_duration_seconds_count[2m]),
    {__name__="build_info", version=~"2\\..+"}
  )
)

Before and After: Side-by-Side Comparison

Let's see how the info() function simplifies real queries:

Example 1: OpenTelemetry Resource Attribute Enrichment

Traditional approach:

sum by (http_status_code, k8s_cluster_name, k8s_namespace_name, k8s_container_name) (
    rate(http_server_request_duration_seconds_count[2m])
  * on (job, instance) group_left (k8s_cluster_name, k8s_namespace_name, k8s_container_name)
    target_info
)

With info():

sum by (http_status_code, k8s_cluster_name, k8s_namespace_name, k8s_container_name) (
  info(rate(http_server_request_duration_seconds_count[2m]))
)

The intent is much clearer with info: We're enriching http_server_request_duration_seconds_count with Kubernetes related OpenTelemetry resource attributes.

Example 2: Filtering by Label Value

Traditional approach:

sum by (http_status_code, k8s_cluster_name) (
    rate(http_server_request_duration_seconds_count[2m])
  * on (job, instance) group_left (k8s_cluster_name)
    target_info{k8s_cluster_name=~"us-.*"}
)

With info():

sum by (http_status_code, k8s_cluster_name) (
  info(
    rate(http_server_request_duration_seconds_count[2m]),
    {k8s_cluster_name=~"us-.*"}
  )
)

Here we filter to only include metrics from clusters in the US (which names start with us-). The info() version integrates the filter naturally into the data-label-selector.

Technical Benefits

Beyond the fundamental UX benefits, the info() function provides several technical advantages:

1. Automatic Churn Handling

As previously mentioned, info() automatically picks the matching info time series with the latest sample when multiple versions exist. This eliminates the "many-to-many matching" errors that plague traditional join queries during churn.

How it works: When non-identifying info metric labels change (e.g., a pod is re-created), there's a brief period where both old and new series might exist. The info() function simply selects whichever has the most recent sample, ensuring your queries keep working.

2. Better Performance

The info() function is more efficient than traditional joins:

• Only selects matching info series
• Avoids unnecessary label matching operations
• Optimized query execution path

Getting Started

The info() function is experimental and must be enabled via a feature flag:

prometheus --enable-feature=promql-experimental-functions

Once enabled, you can start using it immediately.

Current Limitations and Future Plans

The current implementation is an MVP (Minimum Viable Product) designed to validate the approach and gather user feedback. The implementation has some intentional limitations:

Current Constraints

1. Default info metric: Only considers target_info by default

   • Workaround: You can use __name__ matchers like {__name__=~"(target|build)_info"} in the data-label-selector, though this still assumes job and instance as identifying labels

2. Fixed identifying labels: Always assumes job and instance are the identifying labels for joining

   • This unfortunately makes info() unsuitable for certain scenarios, e.g. including data labels from kube_pod_labels, but it's a problem we want to solve in the future

Future Development

These limitations are meant to be temporary. The experimental status allows us to:

• Gather real-world usage feedback
• Understand which use cases matter the most
• Iterate on the design before committing to a final API

A future version of the info() function should:

• Consider all info metrics by default (not just target_info)
• Automatically understand identifying labels based on info metric metadata

Important: Because this is an experimental feature, the behavior may change in future Prometheus versions, or the function could potentially be removed from PromQL entirely based on user feedback.

Giving Feedback

Your feedback will directly shape the future of this feature and help us determine whether it should become a permanent part of PromQL. Feedback may be provided e.g. through our community connections or by opening a Prometheus issue.

We encourage you to try the info() function and share your feedback:

• What use cases does it solve for you?
• What additional functionality would you like to see?
• How could the API be improved?
• Do you see improved performance?

Conclusion

The experimental info() function represents a significant step forward in making PromQL more accessible and reliable. By simplifying metadata label enrichment and automatically handling the churn problem, it removes two major pain points for Prometheus users, especially those adopting OpenTelemetry.

To learn more:

• PromQL functions documentation
• OpenTelemetry guide (includes detailed info() usage)
• Feature proposal

Please feel welcome to share your thoughts with the Prometheus community on GitHub Discussions or get in touch with us on the CNCF Slack #prometheus channel.

Happy querying!

Prometheus' target relabeling feature allows you to adjust the labels of a discovered target or even drop the target entirely. Relabeling rules, while powerful, can be hard to understand and debug. Your rules have to match the expected labels that your service discovery mechanism returns, and getting any step wrong could label your target incorrectly or accidentally drop it.

To help you figure out where things go wrong (or right), Prometheus 3.8.0 just added a relabeling visualizer to the Prometheus server's web UI that allows you to inspect how each relabeling rule is applied to a discovered target's labels. Let's take a look at how it works!

Using the relabeling visualizer

If you head to any Prometheus server's "Service discovery" page (for example: https://demo.promlabs.com/service-discovery), you will now see a new "show relabeling" button for each discovered target:

Clicking this button shows you how each relabeling rule is applied to that particular target in sequence:

The visualizer shows you:

• The initial labels of the target as discovered by the service discovery mechanism.
• The details of each relabeling rule, including its action type and other parameters.
• How the labels change after each relabeling rule is applied, with changes, additions, and deletions highlighted in color.
• Whether the target is ultimately kept or dropped after all relabeling rules have been applied.
• The final output labels of the target if it is kept.

To debug your relabeling rules, you can now read this diagram from top to bottom and find the exact step where the labels change in an unexpected way or where the target gets dropped. This should help you identify misconfigurations in your relabeling rules more easily.

Conclusion

The new relabeling visualizer in the Prometheus server's web UI is a powerful tool to help you understand and debug your target relabeling configurations. By providing a step-by-step view of how each relabeling rule affects a target's labels, it makes it easier to identify and fix issues in your setup. Update your Prometheus servers to 3.8.0 now to give it a try!

My first introduction to the Prometheus project was through the Linux Foundation mentorship program, where I conducted UX research. I remember the anxiety I felt when I was selected as a mentee. I was new not just to Prometheus, but to observability entirely. I worried I was in over my head, working in a heavily developer-focused domain with no development background.

That anxiety turned out to be unfounded. I went on to make meaningful contributions to the project, and I've learned that what I experienced is nearly universal among non-technical contributors to open source.

If you're feeling that same uncertainty, this post is for you. I'll share the challenges you're likely to face (or already face), why your contributions matter, and how to find your place in the Prometheus community.

The Challenges Non-Technical Contributors Face

As a non-technical contributor, I've had my share of obstacles in open source. And from conversations with others navigating these spaces, I've found the struggles are remarkably consistent. Here are the most common barriers:

1. The Technical Intimidation Factor

I've felt out of place in open source spaces, mostly because technical contributors vastly outnumber non-technical ones. Even the non-technical people often have technical backgrounds or have been around long enough to understand what's happening.

When every conversation references concepts you don't know, it's easy to feel intimidated. You join meetings and stay silent throughout. You respond in the chat instead of unmuting because you don't trust yourself to speak up in a recorded meeting where everyone else seems fluent in a technical language you're still learning.

2. Unclear Value Proposition

Open source projects rarely spell out their non-technical needs the way a job posting would. You would hardly find an issue titled "Need: someone to interview users and write case studies" or "Wanted: community manager to organize monthly meetups." Instead, you’re more likely to see a backlog of GitHub issues about bugs, feature requests, and code refactoring.

Even if you have valuable skills, you don't know where they're needed, how to articulate your value, or whether your contributions will be seen as mission-critical or just nice-to-have. Without a clear sense of how you fit in, it's difficult to reach out confidently. You end up sending vague messages like "I'd love to help! Let me know if there's anything I can do", which rarely leads anywhere because maintainers are busy and don't have time to figure out how to match your skills to their needs.

3. Lack of Visible Non-Technical Contributors

One of the things that draws me to an open source community or project is finding other people like me. I think it's the same way for most people. Representation matters. It's hard to be what you can't see.

It’s even more difficult to find non-technical contributors because their contributions are often invisible in the ways projects typically showcase work. GitHub contribution graphs count commits. Changelogs list code changes and bug fixes. You only get the "contributor" label when you've created a pull request that got merged. So, even when people are organizing events, supporting users, or conducting research, their work doesn't show up in the same prominent ways code does.

4. The Onboarding Gap

A typical "Contributing Guide" will walk you through setting up a development environment, creating a branch, running tests, and submitting a pull request. But it rarely explains how to contribute documentation improvements, where design feedback should go, or how community support is organized.

You see "Join our community" with a link to a Slack workspace. But between joining and making your first contribution, there's a significant gap. There are hundreds of people in dozens of channels. Who's a maintainer and who's just another community member? Which channel is appropriate for your questions? Who should you tag when you need guidance?

Why These Gaps Exist

It's worth acknowledging that most of the time, these gaps aren't intentional. Projects don't set out to exclude non-technical contributors or make it harder for them to participate.

In most cases, a small group of developers build something useful and decide to open-source it. They invite people they know who might need it (often other developers) to contribute. The project grows organically within those networks. It becomes a community of developers building tools for developers, and certain functions simply don't feel necessary yet. Marketing? The word spreads naturally through tech circles. Community management? The community is small and self-organizing. UX design? They're developers comfortable with command-line interfaces, so they may not fully consider the experience of using a graphical interface.

None of this is malicious. It's just that the project evolved in a context where those skills weren't obviously needed.

The shift happens when someone, often a non-technical contributor who sees the potential, steps in and says: "You've built something valuable and grown an impressive community. But here's what you might be missing. Here's how documentation could lower the barrier to entry. Here's how community management could retain contributors. Here's how user research could guide your roadmap."

Why Non-Technical Contributions Matter

Prometheus is a powerful monitoring system backed by a large community of developers. But like any open source project, it needs more than code to thrive.

It needs accessible documentation. From my experience working with engineers, most would rather focus on building than writing docs, and understandably so. Engineers who know the system inside out often write documentation that assumes knowledge newcomers don't have. What makes perfect sense to someone who built the feature can feel impenetrable to someone encountering it for the first time. A technical writer testing the product from an end user's perspective, not a builder's, can bridge that gap and lower the barrier to entry.

It needs organization. The GitHub issues backlog has hundreds of open items that haven't been triaged. Maintainers spend valuable time parsing what users actually need instead of building solutions. A project manager or someone with triage experience could turn that chaos into a clear roadmap, allowing maintainers to spend their time building solutions.

It needs community support. Imagine a user who joins the Slack workspace, excited to contribute. They don't know where to start. They ask a question that gets buried in the stream of messages. They quietly leave. The project just lost a potential contributor because no one was there to welcome them and point them in the right direction.

These are the situations non-technical contributions can help prevent. Good documentation lowers the barrier to entry, which means more adoption, more feedback, and better features. Active community management retains contributors who would otherwise drift away, which means distributed knowledge and less maintainer burnout. Organization and triage turn scattered input into actionable priorities.

The Prometheus maintainers are doing exceptional work building a robust, scalable monitoring system. But they can't do everything, and they shouldn't have to. The question now isn't whether non-technical contributions matter. It's whether we create the space for them to happen.

Practical Ways You Can Contribute to Prometheus

If you're ready to contribute to Prometheus but aren't sure where to start, here are some areas where non-technical skills are actively needed.

1. Join the UX Efforts

Prometheus is actively working to improve its user experience, and the community now has a UX Working Group dedicated to this effort.

If you're a UX researcher, designer, or someone with an eye for usability, this is an excellent time to get involved. The working group is still taking shape, with ongoing discussions about priorities and processes. Join the Slack channel to participate in these conversations and watch for upcoming announcements about specific ways to contribute.

I can tell you from experience that the community is receptive to UX contributions, and your work will have a real impact.

2. Write for the Prometheus Blog

If you're a technical writer or content creator, the Prometheus blog is a natural entry point. The blog publishes tutorials, case studies, best practices, community updates, and generally, content that helps users get more value from Prometheus.

Check out the blog content guide to understand what makes a strong blog proposal and how to publish a post on the blog. There's an audience eager to learn from your experience.

3. Improve and Maintain Documentation

Documentation is one of those perpetual needs in open source. There's always something that could be clearer, more complete, or better organized. The Prometheus docs repo is no exception.

You can contribute by fixing typos and broken links, expanding getting-started guides, creating tutorials for common monitoring scenarios, or triaging issues to help prioritize what needs attention. Even if you don't consider yourself a technical writer, if you've ever been confused by the docs and figured something out, you can help make it clearer for the next person.

4. Help Organize PromCon

PromCon is Prometheus's annual conference, and it takes significant coordination to pull off. The organizing team handles everything from speaker selection and scheduling to venue logistics and sponsor relationships.

If you have experience in event planning, sponsor outreach, marketing, or communications, the PromCon organizers would welcome your help. Reach out to the organizing team or watch for announcements in the Prometheus community channels.

5. Advocate and Amplify

Finally, one of the simplest but most impactful things you can do is talk about Prometheus. Write blog posts about how you're using Prometheus. Give talks at local meetups or conferences. Share tips and learnings on social media. Create video tutorials or live streams. Recommend Prometheus to teams evaluating monitoring solutions.

Every piece of content, every conference talk, every social media post expands Prometheus's reach and helps new users discover it.

How to Get Started

If you're ready to contribute to Prometheus, here's what I've learned from my own experience navigating the community as a non-technical contributor:

Start by introducing yourself. When you join the #prometheus-dev Slack channel, say hello. Slack doesn't always make it obvious when someone new joins, so if you stay silent, people simply won't know you're there. A simple introduction—your name, what you do, what brought you to Prometheus—is enough to make your presence known.

Attend community meetings. Check out the community calendar and sync the meetings that interest you. Even if you don't understand everything being discussed at first (and that's completely normal), stay. The more you sit in, the more you'll learn about the community's needs and find more opportunities to contribute.

Observe before you act. It's tempting to jump in with ideas immediately, but spending time as an observer first pays off. Read through Slack discussions and conversations in GitHub issues. Browse the documentation. Notice what kinds of contributions are being made. You'll start to see patterns: recurring questions, documentation gaps, areas where help is needed. That's where your opportunity lies.

Ask questions. Everyone was new once. If something isn't clear, ask. If you don't get a response right away, give it some time—people are busy—then follow up. The community is welcoming, but you have to make yourself visible.

The Prometheus community has room for you. Now you know exactly where to begin.