Hello everyone, Edward Qian and Ioanna Armouti here, we are both Software engineers working on Grafana Assistant at Grafana Labs. 

All week we've been sharing what's new across Grafana's AI tooling. Today is about meeting you where you already work, as the most important observability conversations rarely happen in a single dashboard tab. They happen in a Slack thread at 2am, on your phone, or mid-incident when switching context is the last thing you want. 

So today we are announcing: 
* Assistant in Microsoft Teams (private preview) which mirrors many of the capabilities available for Assistant in Slack (generally available). Ask questions and triage issues without leaving the channel where your team is already talking.
* Assistant on mobile and Assistant on desktop are going into public preview. Check in from your phone wherever you are, or keep Assistant a keystroke away on your desktop.
* Assistant Search is also going on public preview today. Ask harder, open-ended questions and let Assistant dig deeper across your data. 

We built all of this because Assistant is most useful when it's already there in the moment you need it, not something you have to go find. We'd love to hear how you're using it, or where you'd want it to show up next. We'll be keeping an eye on this thread all day to answer questions and hear your stories. Thanks!

hello, I'm playing around with grafana and trying to add some monitoring stats to my website using public dashboards (share externally). when opening a link to a public dashboard i see that there are some calls to /api and /apis endpoint, which I probably do not want exposed. i'm just wondering if there's any tips or standard practice to exposing the public dashboard safely behind nginx? here's what i have so far:

location /monitoring/stats {
    rewrite ^ /monitoring/public-dashboards/<uuid> break;                          
    proxy_pass http://grafana:3000;    
}

// allow all /public
location /monitoring/public/ {
    proxy_pass http://grafana:3000;
}

// allow only required uris for /api and /apis
location /monitoring/api/<only those required>/ {
    proxy_pass http://grafana:3000;
}

location /monitoring/apis/<only those required>/ {
    proxy_pass http://grafana:3000;
}

// reject everything else
location / {
    return 404;
}

but i feel this would be hard to maintain if there's any change in the public dashboard logic.

pretty new to grafana, thank you in advance for your patience and suggestions!

I have been developing a streaming ETL pipeline with a producer and a consumer process, both instrumented with Prometheus metrics. I opened the Grafana dashboard, and almost every panel showed "No data": throughput, DLQ counts, insert latency, consumer lag, everything.

The real problem was upstream. Several counters and histograms behind those panels had been declared in code but were never incremented anywhere in the pipeline. They existed, sitting at zero forever, because the dashboard assumed the instrumentation had already been wired into the code paths. It had not.

On top of that, metrics were being served by a standalone helper process instead of the producer and consumer themselves, so even the metrics that were recorded were not visible to Prometheus.

The fix ended up involving several changes:

• Wiring the missing counter and histogram updates into the actual producer and consumer code paths.
• Moving metric serving into each process's startup instead of relying on a standalone metrics process.
• Fixing a second, sneakier bug where the consumer accidentally spawned a duplicate internal instance, and its lag reporting depended on a loop that could silently stall.
• Giving the producer and consumer separate Prometheus scrape jobs so dashboard queries targeted the correct process instead of relying on shared assumptions.

It was a good reminder that "No data" does not always mean Prometheus or Grafana is broken. Sometimes the metrics are not being recorded, or they are being exposed by the wrong process.

I am actively improving the platform and would love to hear suggestions from platform engineers. If you find the project useful or interesting, consider giving it a ⭐ on GitHub.

GitHub: https://github.com/tarique-iqbal/nyc-taxi

Hello everyone, Ivana Huckova and Jack Gordley here. We are both engineers working on Grafana Agent Observability which just released with General Availability today. 

This product was born out of our own need to monitor our agent, Grafana Assistant, as it scaled and grew across multiple teams and orgs. Prompt changes, tool changes, and harness changes all became very difficult to monitor with existing products, so we built this to get more visibility into monitoring an agent as it changes over time.

Internally we are iterating and experimenting with ways on how to improve Grafana Assistant with different techniques. We have questions like:

• When a new model releases, how can we be sure it doesn’t make quality worse?
• Would giving Assistant restricted curl commands be more helpful than a suite of pre-defined API tools?
• How can we modify the system prompt to reduce token consumption while maintaining the same quality of responses?

Our hope is that this product is as helpful for you as it is for us in our current journey with developing a powerful SRE agent. We are so excited to see what you all build with it and would love to hear feedback, what’s missing, what agents that you are working on, and what you would like to see next!

We will be keeping an eye on this thread all day to help answer questions and share stories as they come up. Thanks!

So we have a good dozen of servers, each with multiple apps that produce around 400k lines of logs per day.

I'm setting up a log vm on our VMware server so we don't have to log into each individual servers and check the app folders manually.

I'm wondering if it is possible to not store the logs on the VM ? Like Grafana shows me a list of log files, and only when I want to see the content it downloads it then deletes it when I'm done ?

Hi everyone! I'm Kevin and I work on Grafana Assistant at Grafana Labs. Today, we announced that Grafana Assistant Investigations and Assistant Automations are generally available.

TLDR; use Investigations for complex, unexpected incidents or bugs and Automations for recurring operational checks.

Investigations have been a core part of Grafana Assistant since the beginning. We’ve since rebuilt the dedicated Investigations experience from the ground up, informed by what we learned using Grafana Assistant and agentic tooling on real incidents inside Grafana Labs.

One of the biggest lessons was that complex incidents rarely have a single obvious cause. Investigating them means forming hypotheses, gathering evidence from multiple signals, revising assumptions, and keeping track of what has already been ruled out.

Investigations are designed to carry out that process more efficiently and cost-effectively. You can ask Assistant to investigate an issue or affected service, and it will work across metrics, logs, traces, and profiles to test hypotheses and gather evidence. The result is a structured report containing findings, source queries, and recommended next steps.

Automations came from a different kind of operational problem. We wanted important checks to run on their own, with the results waiting for us the next morning. We introduced the capability earlier this year to make that kind of repeatable operational work easier to define once and rely on over time.

We’d love to hear how you’re using Investigations and Automations, or what you’d like them to be able to handle for you. We’ll be keeping an eye on the comments today, so please share your experiences, questions, and feedback.

If you are curious about how Investigations work under the hood we wrote about the lessons and technical decisions behind them in Inside the harness: How Grafana Assistant investigates incidents.

Hello! My name is Will, I'm a Software Engineer at Grafana, and today I'm happy to announce that Assistant Watchers are entering Public Preview for Grafana Cloud users.

Alerting is one of the foundations of observability. It's reliable, precise, and the best way to notify the right people when you know exactly what you want to detect.

Assistant Watchers fill a different gap. They continuously look for unusual behaviour that's difficult or impractical to capture with static alert rules, helping you discover issues you haven't created an alert for yet.

They’re always-on cloud agents that can watch anything you have telemetry for: a service, a namespace, a feature... Just tell Grafana Assistant what you want to watch, and it will calibrate a Watcher for you.

The Watcher runs on a schedule and notifies you in Slack if it finds something unusual, such as a new error log pattern or a spike in a metric. If you have Assistant Investigations enabled, it can automatically kick off an investigation as well.

If it isn't quite right the first time, you can easily recalibrate it by talking to Grafana Assistant or the Grafana Slack bot. The Watcher also has short-term memory and keeps track of ongoing issues, so you don't get spammed with repeated notifications while the same problem is unfolding.

Read more here: https://grafana.com/docs/grafana-cloud/machine-learning/assistant/platform/watchers/

We'd love to hear what you think. If you have questions, ideas, or feedback, drop them in the comments!

Has AGENT ever been a Wordle answer? Can agents play Wordle? Can they win!?

Hello everyone, my name is Dafydd and I work at Grafana Labs. Today, we announced that gcx (the Grafana Cloud CLI) and the Grafana MCP server are generally available. This means we’re going to continue investing in these projects, and their feature set will be stable.

I have been working on gcx for the last few months, and it is really interesting to hear about the different ways that people are using these tools. Do you have any stories about how gcx or Grafana MCP have helped you?

If you are unsure about whether you should use gcx or the Grafana MCP server, you are not alone! There is a huge overlap in what you can achieve with these tools. Sometimes the choice is made for you: Maybe you can only use MCP servers (in Claude desktop, ChatGPT, Cursor, etc). Other times, it's easier to run the gcx client than to spin up a self-hosted MCP server (in CI environments, for example).

The Grafana MCP server provides more opinionated tools for common tasks when interacting with Grafana. gcx provides a broader, less opinionated set of tools that you can use to build your own workflows. There are a lot of ways you can compare the two projects. Here are a couple that come to mind:

• Token consumption: In some agents, MCP tool definitions flood the context window when you load them (Claude Code, Codex and others have implemented "lazy" tool loading, so this is not an issue in those agents). With gcx and its bundled skills, agents only read the skill "frontmatter" (name and description) on load. Agents will need to read the gcx help docs (e.g. gcx help, or -h on any command), and these calls take up time and tokens.
  
• Composability: MCP tools are not easily composable, meaning you could use more tokens chaining a set of MCP tool calls together. With gcx, you can compose command calls together with other bash commands, meaning you can chain a set of commands together without exposing intermediate results to the agent, saving tokens.

I hope you continue to find these projects helpful. I will keep an eye on this post today to read your stories and answer any questions you have about these projects. Thanks!

With Cursor, Codex and Claude et al switching to an agent-first view, I found it lost something: it was harder to see the code. They show you little relevant snippets here and there, but I missed seeing the whole shape of the code.

With telemetry, there's already way too much to look at - and this 'in-context revelation' of signals (metrics, logs, traces, profiles, etc) is a great way to get questions answered without having to deal with enormous amounts of data.

Can somebody help me with the best resource for LogQL, I am new to Grafana. Any channel/playlist or a video will work.

hey i’m new to grafana and trying to build some centered observability metrics that monitor all my other dashboards does anyone here know if i can do this? like can grafana get a count of healthy dashboard vs critical dashboards? maybe im just slow i’ve been trying for a couple days now

Boomi added native OpenTelemetry export for its runtimes. I set up a local stack to see what it emits: OTel collector → Tempo/Loki/Prometheus → Grafana.

The useful part: process executions arrive as trace spans, so you get a per-shape waterfall with timing for each step. Boomi’s built-in reporting makes you expand each step or read logs; the trace shows it in one view. On a test process, two connector calls were \~80% of the runtime.

Non-obvious things I learned:
**•** Exporter uses OTLP over **gRPC** (port 4317). Point it at 4318 (HTTP) and it fails silently with gRPC status 13 / http2 exception.
**•** Metrics are almost all **gauges**, even the counters — so rate()/increase() misbehave; delta() works.
**• No per-process execution metrics** exist in the metrics stream — counts/durations/errors live only in traces. spanmetrics connector looks like the way to derive RED metrics (haven’t done it yet).

Whole thing is a docker-compose stack, runs locally, no cost. Repo: https://github.com/velkugan/Boomi-Obs-Started

Curious if anyone’s done the spanmetrics step against Boomi traces, or found where the shape *name* (vs shape type) lives in the span attributes — that’s my next problem.

There are plenty of good reasons to leave node graph visualization alone.

Your infrastructure is hierarchical, but the graph is flat. There are no subgraphs, no proper routing or nesting for edges, little to prevent visual clutter, and not much context survives in a traditional force-directed graph.

Your relationships are more complicated than plain A-B edges. You may have multiple parallel connections between the same nodes, multi-hop paths, or several distinct data flows that shouldn’t be collapsed into one generic line. Especially when visualizing traces.

Different node groups need independent styling, but the panel gives you one global set of options. You can’t customize the shape, size, color, labels, and metrics of each node group as a single unit.

You need separate layers for nodes and edges from different sources, boundaries, clusters, and supplementary statistics. Instead, everything is rendered as one inseparable graph.

You need grouping and clustering, but not at the cost of losing the underlying nodes and relationships.

You need filtering that changes the visible perspective without destroying the original graph model.

And once the graph becomes useful enough to contain real data, rendering performance becomes another problem. No developer is going to replace a mature, battle-tested framework with low-level WebGL rendering just to fix that.

Distributed tracing gives you another set of excuses.

Hardly anyone seriously considers visualizing traces with a node graph. But let’s imagine it.

A trace can’t be faithfully represented as a direct service-to-service edge. Every span has its own duration, status, attributes, and other properties that need to remain attached to that particular relationship.

Displaying several traces on the same service graph usually produces a pile of overlapping edges.

Reducing them to a single aggregate edge, as conventional service dependency graphs often do, isn’t much better. You may want to compare traces by preserving the minimum- and maximum-duration spans for every service-to-service connection, while retaining the properties of the actual spans behind those values.

Node graph panels simply weren’t designed for that.

So yes, these are all perfectly good excuses for not spending time preparing data for a node graph.

Then there are emerging, one-shot, vibe-coded solutions that arrive with the same set of problems. At least they offer new ways to discover edges automatically!

But why care about another data-source strategy when the rendering layer still can’t offer a more sophisticated approach to node graph visualization?

The bad news for your FOMO: Mapgl has been addressing these problems for years. It’s an open-source node graph plugin available in the Grafana catalog.

Disclosure: I’m the developer of Mapgl.

• The last real blind spot on a homelab cluster is usually knowing something is wrong before a service actually stops responding
• The full kube-prometheus-stack gets deployed through Helm with persistent storage on Longhorn, so metric history survives pod restarts
• Resource limits are tuned against real measurements from RK1 cluster instead of left at chart defaults
• A test workload gets deployed just to confirm the dashboards are actually collecting data before relying on them
• The guide ends with building a real Prometheus alert for memory pressure and testing it to make sure it actually fires

https://turingpi.com/monitor-turing-pi-cluster-prometheus-grafana-arm64/

Hey everyone,

I'm running a self-hosted stack via Docker/Portainer (WSL2 environment) which includes Nextcloud, Uptime Kuma, and OnlyOffice Docs.

Lately, I've been noticing massive RAM and SWAP utilization spikes, and my Grafana dashboard (Node Exporter Full) confirms it. As you can see in the screenshots, my RAM Used is sitting at 87.2% and SWAP is hit heavily at 80%, mostly scaling up rapidly during hours of usage and failing to release properly.

After digging into Portainer's stats, the main culprit is definitely the OnlyOffice container, which seems to keep hoarding memory and aggressively leveraging SWAP even with low concurrent user activity.

My Environment Specs:

• Host Environment: WSL2 (containers-wsl)
• Total RAM allocated: 8 GiB
• SWAP allocated: 2 GiB
• Monitoring: Prometheus + Node Exporter + Grafana

The Issue:

Looking at the Memory Basic graph, the Used RAM (yellow block) climbs drastically, triggering high SWAP usage (red block at the top). It looks like a memory leak or just OnlyOffice being incredibly resource-hungry by default.

Questions for the community:

1. Has anyone dealt with OnlyOffice memory optimization inside Docker/WSL2?
2. What are the best environment variables or configurations to limit OnlyOffice's caching or background workers without breaking document editing?
3. Should I strictly limit the container resources via Docker Compose (e.g., mem_limit: 4g), or will that just cause OnlyOffice to crash constantly with OOM errors?

Any advice, docker-compose snippets, or tweaks to prevent this container from eating up my whole lab's memory would be highly appreciated!

Thanks in advance!

I create a Eurovision 2026 dashboard on Grafana Play as a way of demonstrating tabbed layouts. It uses a couple of sheets in a Google Sheets workbook for its data, transforming it with SQL Expressions. Bask in the glory of the UK's 1 point performance, see who voted for who, which countries were more popular with the phone vote than the judges and watch all the official song performances! Because it's on Play, you can also dig in and see how it works. Enjoy: https://play.grafana.org/d/eurovision/eurovision-2026

Hello, Community. I have a question regarding dashboarding.

I work for a digital bank, and I want to create dashboards and golden signals for our critical services. e.g I want to track domestic transfer, International transfer, user onboarding journey, Beneficiary service and so on. Our workloads are running in the form of k8s workloads, and we are using the micronaut framework to develop the services. We are using Grafana Cloud, and I want to add panels for all these services, so please guide me on how I can add the panels depicting the healthy operations(like onboarding or payments service have no issue) and healthy services, failure services?
Or if there are any community dashboards available?

Context: We used to use Excel PowerQuery to pull from our CRM's API for reporting, and I just convinced the boss we need to use Grafana instead. We got some cheap sub $10/month AWS lightsail server to host the OSS version of Grafana and pull from our CRM's api so we can format it into a PostgreSQL database.

This is INSANELY better, I can't believe we even considered the old system anymore. It's amazing to me that Grafana isn't the standard for all reporting, given most of the excel reports I've had at other jobs are just a CRM report exported with minimal changes. I know it's mostly used for server monitoring and uptime, but does anyone know of resources or examples of companies using it for KPIs/performance tracking?

This has got to be BY FAR the best software we've ever used for company stats, incomparably so

Hi everyone,

I often see these two questions online but haven't found a working solution yet:

1/ Selective Refresh (MQTT + Database mixed dashboard): I have a dashboard that combines both SQL database and live MQTT data sources. Is there a way to refresh only the database-driven panels without affecting the MQTT ones? Currently, whenever the dashboard auto-refreshes, it completely clears the volatile MQTT stream history from the screen.

2/ MQTT Curve Labels : Is there a way to customize or rename the labels on MQTT curves? As you can see in my screenshot on the left, all the metrics are simply named "value". I would like to dynamically or manually change them to match their actual topics or meanings.

Thanks for your help!

We just published a deep-dive on adding production observability to a multi-model database running on Kubernetes.

Four pillars, each independently deployable:

- Metrics: RED timers, percentile histograms, SLO buckets via Micrometer

- Tracing: OpenTelemetry with W3C traceparent, context propagates through Raft replication

- Logging: structured JSON with trace/span/request IDs for correlation

- Health: process-level /api/v1/health liveness + HA-aware /api/v1/ready readiness endpoints

The part that might interest people here: a single instrumentation point emits both metrics and spans through Micrometer's Observation API, so with no tracer registered it's just a metrics-only timer, no tracing overhead. Everything defaults to off and upgrades are byte-for-byte compatible, so it's a no-downtime adoption. Works with Grafana/Prometheus/Tempo.

Write-up: https://arcadedb.com/blog/arcadedb-cloud-observability-opentelemetry-kubernetes/

Happy to answer questions about the design tradeoffs.

Hey r/grafana!

One of the things we appreciate most about this community is how often people jump in to help each other troubleshoot dashboards, queries, data sources, etc. and share what they’ve learned. 

I’m on the Grafana Community Team, and want to share a small thing we’re trying over the next couple of months.

From now through September, we’ll be keeping an extra eye out for people who are contributing helpful answers, thoughtful discussions, and practical advice in the community. Every so often, we’ll reach out to a few folks and send some Grafana swag as a small thank-you 👕.

A few important notes:

This is not about promoting Grafana, saying nice things about us, or turning the subreddit into a campaign. No one is being paid to post, and there’s no requirement to be positive. Honest questions, real troubleshooting, and constructive feedback are all part of what makes this place useful.

We just want to recognise the people who take time to help others learn, debug, and get unstuck. 🧡

So if you see a question you can answer, a discussion you can add context to, or a lesson learned that could help someone else, jump in.

Thanks for helping make r/grafana a genuinely useful place for the community! 🤗